CHUNK_TEXT — Document Chunking

CHUNK_TEXT is a built-in table-valued function that splits text into overlapping chunks suitable for embedding and retrieval-augmented generation (RAG) pipelines. It is markdown-aware — it prefers to break at paragraph boundaries, headings, and list items rather than mid-sentence.

Syntax

SQL

SELECT chunk_index, chunk_text, chunk_pos
FROM CHUNK_TEXT(
  content,
  [max_chars],
  [overlap_chars],
  [title]
);

All parameters except content are optional and can be passed positionally or as named arguments.

Parameter	Type	Default	Description
content	TEXT	—	The document text to chunk (max 1 MB).
max_chars	INT	3600	Target maximum chunk size (~900 tokens of English text). Zero or negative values fall back to the default.
overlap_chars	INT	540	How far each chunk reaches back into the previous one. The default is a fixed 540 — it does not scale with max_chars. Must be less than max_chars; negative values fall back to the default.
title	TEXT	NULL	Document title prepended to each chunk.

max_chars and overlap_chars count bytes, not characters

Both sizes, and the chunk_pos output column, are measured in UTF-8 bytes. For ASCII text that is the same as characters. For multi-byte text it is not: repeat('数', 100) (100 characters, 300 bytes) split with max_chars => 90, overlap_chars => 0 comes back as chunks of 30, 30, 30 and 10 characters, at chunk_pos 0, 90, 180 and 270.

Pass overlap_chars whenever max_chars is 540 or less

Because the default overlap stays at 540, any max_chars of 540 or less without an explicit overlap_chars fails with ERROR: internal error (XX000) for any non-NULL content, even CHUNK_TEXT('short text', max_chars => 200) or an empty string. The same error is raised, for non-NULL content, whenever overlap_chars is greater than or equal to max_chars.

Each chunk starts overlap_chars bytes before the previous chunk ends, and no chunk is longer than max_chars, so on text without break points every chunk advances only max_chars - overlap_chars bytes. Keep that gap wide: max_chars => 541 with the default overlap succeeds, but on repeat('word ', 2000) it advances one byte per chunk.

Output Columns

Column	Type	Description
chunk_index	INT	Zero-based chunk number.
chunk_text	TEXT	The chunk content (with title prefix if provided).
chunk_pos	INT	Byte offset of this chunk in the original document (equal to the character offset for ASCII text).

Basic Usage

SQL

SELECT chunk_index, length(chunk_text) AS len, chunk_pos
FROM CHUNK_TEXT(
  content => E'DB9 is a serverless database that speaks the PostgreSQL wire protocol. Every database ships with built-in extensions for HTTP calls, vector search and file storage.\n\n'
          || E'## Chunking\n\n'
          || E'CHUNK_TEXT splits a long document into overlapping pieces. Each piece is small enough to embed, and the overlap carries context across the boundary.\n\n'
          || E'## Retrieval\n\n'
          || 'Store each chunk with its embedding, then rank chunks by vector distance to answer a question.',
  max_chars => 200,
  overlap_chars => 30
);

▶ Run

Output

 chunk_index | len | chunk_pos
-------------+-----+-----------
           0 | 165 |         0
           1 | 193 |       135
           2 | 139 |       298

Chunks 0 and 1 end at the paragraph break before a heading rather than at the 200-byte limit, and each following chunk starts 30 bytes (overlap_chars) before that break — chunk 1 starts at 135, which is chunk 0’s length of 165 minus 30. The overlap is a raw byte offset, so a chunk can begin mid-word.

With Title

When title is provided, it is prepended to each chunk so that embedding models have document context:

SQL

SELECT chunk_index, chunk_text
FROM CHUNK_TEXT(
  'Hello world. This is the document body.',
  title => 'My Document'
);

▶ Run

Output

 chunk_index | chunk_text
-------------+--------------------------------------------------------------------
           0 | title: My Document | text: Hello world. This is the document body.

Smart Splitting

CHUNK_TEXT identifies natural break points in the text and assigns them a score based on proximity to the target chunk boundary:

• Paragraph breaks (double newlines) — strongest break point
• Markdown headings (## ...) — strong break point
• List items (- ..., 1. ...) — moderate break point
• Sentence endings — fallback break point

Code fences (triple backticks) are treated as no-break zones — the function will not split inside a fenced code block.

RAG Pipeline Example

Combine CHUNK_TEXT with embedding() and HNSW indexes for a complete RAG pipeline:

SQL

-- 1. Create the documents and chunks tables
CREATE TABLE rag_documents (
  id SERIAL PRIMARY KEY,
  title TEXT,
  content TEXT
);

CREATE TABLE rag_chunks (
  id SERIAL PRIMARY KEY,
  doc_id INT REFERENCES rag_documents(id),
  chunk_index INT,
  chunk_text TEXT,
  -- Must match the embedding model's output size: the default
  -- text-embedding-v4 model returns 1024 dimensions.
  embedding VECTOR(1024)
);

-- 2. Add a document to chunk
INSERT INTO rag_documents (title, content)
VALUES ('Branching', repeat('DB9 branching copies the full dataset asynchronously. ', 80));

-- 3. Chunk and embed that document
INSERT INTO rag_chunks (doc_id, chunk_index, chunk_text, embedding)
SELECT
  1,
  c.chunk_index,
  c.chunk_text,
  embedding(c.chunk_text)
FROM CHUNK_TEXT(
  content => (SELECT content FROM rag_documents WHERE id = 1),
  max_chars => 1500,
  overlap_chars => 200,
  title => (SELECT title FROM rag_documents WHERE id = 1)
) AS c;

-- 4. Create an HNSW index for fast search
CREATE INDEX ON rag_chunks USING hnsw (embedding vector_cosine_ops);

-- 5. Semantic search
SELECT chunk_text, embedding <=> embedding('How does branching work?') AS distance
FROM rag_chunks
ORDER BY distance
LIMIT 5;

▶ Run

Step 5 does not use the HNSW index

The probe is an inline embedding() call, which the planner does not treat as a constant, so this search runs as an exact scan. Results are correct; to use the index, swap the probe for VEC_EMBED_COSINE_DISTANCE(embedding, 'How does branching work?') — see When the index is used.

Use in CTEs

CHUNK_TEXT works in WITH clauses for multi-step processing:

SQL

WITH raw_chunks AS (
  SELECT chunk_index, chunk_text, chunk_pos
  FROM CHUNK_TEXT(
    content => 'Long document...',
    max_chars => 2000
  )
)
SELECT chunk_index, length(chunk_text) AS size
FROM raw_chunks
WHERE length(chunk_text) > 100;

▶ Run

Edge Cases

Input	Result
NULL content	Zero rows returned, whatever the other arguments.
Empty string ''	Zero rows returned — provided overlap_chars is less than max_chars.
Content no longer than max_chars bytes	Single chunk with the full content.
overlap_chars ≥ max_chars — including max_chars of 540 or less with the default overlap	ERROR: internal error (XX000), for any non-NULL content.
Content exceeding 1 MB	ERROR: internal error (XX000).

Limits

Limit	Value
Max content size	1 MB (1,048,576 bytes)
Default max_chars	3,600 bytes (~900 tokens of English text)
Default overlap_chars	540 bytes, fixed — it does not scale with max_chars
Min max_chars	1 (zero or negative falls back to the default)
Min overlap_chars	0 (negative falls back to the default); must be less than max_chars

See Limits and Quotas for the complete list.

Next Steps

• Vector Search — Embedding and HNSW indexes
• Built-in Functions — All SQL functions
• fs9 — File System — Ingest files to chunk from the filesystem