Extractors & chunking

The ingestion pipeline

memloom context add <path> runs every file through four stages:

file bytes → extract (format-specific) → chunk (structure-aware) → embed → store

Extract. The extractor registered for the file’s extension turns bytes into text units: one unit for a text file, one unit per page for a PDF (so page numbers survive into citations).
Chunk. Units are split along the document’s real structure (below), each chunk carrying a breadcrumb like Guide > Setup > Postgres.
Embed. All chunks are embedded in batches before the store transaction: provider calls are slow and can fail, so the database write stays short and all-or-nothing.
Store. Chunks replace the document’s previous chunks in one transaction.

Documents are mirrors of files, so the belief machinery doesn’t apply to them: no dedup, no conflicts, no versions. Re-adding an unchanged file (same content hash) is a no-op; a changed file atomically replaces its chunks.

Built-in extractors

Kind	Extensions	Chunker	Notes
`md`	`.md`, `.markdown`	markdown	Title from the first `#` heading
`txt`	`.txt`	outline
`pdf`	`.pdf`	outline	Text layer via unpdf (pure JS); no OCR

PDF text is rebuilt from glyph geometry instead of stream order: equation-heavy PDFs (Word or LaTeX math) come out in reading order, and 2-up print layouts (the same content twice, side by side) are de-duplicated. Symbols that a PDF’s fonts never map to Unicode (∞, ∈, ≠) can’t be recovered from the text layer; that would need OCR, which is deliberately out of scope.

Two chunking strategies

Both aim at the same rule: a chunk never starts mid-section, and always knows where it came from. Markdown splits at ATX headings (headings inside code fences do not count as structure), tracking the heading stack. Each chunk gets its breadcrumb prepended to the text (Guide > Setup > Postgres\n\n<chunk>), so retrieval’s vector and keyword arms both see the section context along with the paragraph. Outline handles documents without markdown structure: lecture notes, contracts, exercise sheets. It recognizes ALL-CAPS title lines (Unicode-aware, so ŚREDNIA WAŻONA qualifies) and numbered points (2. DEFINITION. …) as boundaries. One point = one chunk, so a chunk is exactly one definition, theorem, or clause, with a breadcrumb like LIMITS OF FUNCTIONS > 2. DEFINITION. Text with no such structure degrades gracefully to plain size-based splitting. The two strategies size chunks differently. Markdown keeps a heading section whole: one section is one chunk up to 16,000 characters, and only a section past that cap splits, at paragraph boundaries toward an 8,000-character target with zero overlap. Heading boundaries are exact semantic boundaries, so nothing is cut mid-thought and nothing is duplicated across chunks. Outline chunks (plain text and PDF) target ~1,600 characters with a 2,048 hard cap and ~200 characters of overlap carried across boundaries. Both are sized in characters on purpose: a tokenizer dependency buys nothing when the embedding model has that much context headroom.

Adding a format

Adding a format means registering one object; no fork needed.

import { registerExtractor } from "@memloom/core";

registerExtractor({
  kind: "html",
  extensions: [".html", ".htm"],
  version: 1,
  chunker: "markdown",           // or "outline": the section strategy applied after extract
  async extract(bytes, path) {
    const text = htmlToText(new TextDecoder().decode(bytes));
    return { title: findTitle(text), units: [{ text, page: null }] };
  },
});

Registrations are keyed by extension (last one wins), so you can also override a built-in: for example, replace the PDF extractor with an OCR-backed one. Ground rules that keep memloom’s install light:

Lazy-import your parser (await import(...) inside extract) and keep it pure JS. No native binaries, no cloud calls.
Preserve locality: emit one unit per page/sheet/section when the format has them; citations depend on the page field of the unit.
Bump version whenever extraction output changes (see below).
Test end to end: register, ingest through contextAdd, recall, assert the citation. See the extractor registry test in packages/core/src/extract.test.ts; it runs against real Postgres (PGLite) with no setup.

Pipeline versioning

Each extractor declares a version. Bump it when the format’s extract/chunk logic changes: the version is salted into the document’s content hash (<sha256>#p2), so the next context add sees “changed” and re-ingests files whose bytes didn’t change, instead of no-op’ing forever on chunks produced by the old pipeline. Version 1 is unsalted so existing stores don’t mass-reingest on upgrade.

Directories passed to context add are scanned recursively; node_modules and dotfiles are skipped, and only files whose extension has a registered extractor are picked up. supportedExtensions() is what the CLI help and error messages print.

Getting Started

Concepts

Guides

Extractors & chunking

The ingestion pipeline

Built-in extractors

Two chunking strategies

Adding a format

Pipeline versioning

​The ingestion pipeline

​Built-in extractors

​Two chunking strategies

​Adding a format

​Pipeline versioning

The ingestion pipeline

Built-in extractors

Two chunking strategies

Adding a format

Pipeline versioning