Extractors and chunking

Overview

file bytes → extract → chunk → embed → store

Extract

The extractor for the file’s extension turns bytes into text units. One unit per page for a PDF, so page numbers survive into citations.

Chunk

Units split along the document’s real structure, each chunk carrying a breadcrumb like Guide > Setup > Postgres.

Embed and store

Embedded in batches before the transaction opens, so the database write stays short and all-or-nothing.

Documents are mirrors of files, so the belief machinery does not apply: no dedup, no conflicts, no versions. Re-adding an unchanged file is a no-op; a changed file atomically replaces its chunks.

Built-in extractors

Kind	Extensions	Chunker	Notes
`md`	`.md`, `.markdown`	markdown	Title from the first heading
`txt`	`.txt`	outline
`pdf`	`.pdf`	outline	Text layer via unpdf, no OCR
`link`	any `http(s)` URL	markdown	Fetched and parsed locally
`audio`	`.mp3` `.m4a` `.wav` `.flac` `.ogg` `.opus` `.aac` `.wma`	markdown	Transcribed on this machine
`video`	`.mp4` `.mkv` `.mov` `.webm` `.avi` `.m4v`	markdown	The audio track, every track mixed in

Recordings have their own page: see Recordings. Web pages that build themselves in the browser are refused with advice rather than stored empty.

What the PDF extractor can and cannot do

Text is rebuilt from glyph geometry rather than stream order, so equation-heavy PDFs come out in reading order and 2-up print layouts are de-duplicated.Symbols a PDF’s fonts never map to Unicode (∞, ∈, ≠) cannot be recovered from the text layer. That needs OCR, which is deliberately out of scope.

Two chunking strategies

Both follow one rule: a chunk never starts mid-section, and always knows where it came from.

	Markdown	Outline
Splits at	ATX headings, ignoring ones inside code fences	ALL-CAPS titles and numbered points
Chunk is	One heading section, whole	One definition, theorem or clause
Size	Up to 16,000 chars, splitting past that toward 8,000	~1,600 chars, 2,048 cap, ~200 overlap
Overlap	None. Heading boundaries are exact	Yes, carried across boundaries

Every chunk gets its breadcrumb prepended to the text, so both the vector and keyword arms see the section context along with the paragraph. Text with no structure degrades to plain size-based splitting.

Why characters and not tokens

Both are sized in characters on purpose. A tokenizer dependency buys nothing when the embedding model has that much context headroom.

Adding a format

One object, no fork.

import { registerExtractor } from "@memloom/core";

registerExtractor({
  kind: "html",
  extensions: [".html", ".htm"],
  version: 1,
  chunker: "markdown",           // or "outline"
  async extract(bytes, path) {
    const text = htmlToText(new TextDecoder().decode(bytes));
    return { title: findTitle(text), units: [{ text, page: null }] };
  },
});

Registrations are keyed by extension and the last one wins, so this also overrides a built-in. You could replace the PDF extractor with an OCR-backed one.

Rule	Why
Lazy-import your parser, keep it pure JS	No native binaries, no cloud calls, light install
One unit per page or section	Citations depend on the unit’s `page` field
Bump `version` when output changes	It salts the content hash, so files re-ingest
Test end to end	Register, ingest, recall, assert the citation

Directories are scanned recursively. node_modules and dotfiles are skipped, and only files with a registered extractor are picked up. Version 1 is unsalted, so existing stores do not mass re-ingest on upgrade.

Getting Started

Concepts

Guides

Extractors and chunking

Overview

Built-in extractors

Two chunking strategies

Adding a format

​Overview

​Built-in extractors

​Two chunking strategies

​Adding a format

Overview

Built-in extractors

Two chunking strategies

Adding a format