Weave documents of any length
through LLM map/merge/repair

LongLoom splits oversized documents the way humans do — read in chunks, extract, combine, condense — with measured information preservation, over any LLM and any storage you already run.

9 LLM integrations + custom endpoints 9 storage backends — your schema, your tables Measured no-loss — 41/41 facts MIT licensed

$ pip install longloom

Built for real pipelines, not demos

One dependency seam each. Every claim below is backed by a runnable example or a test in the repository.

🔌

Any LLM, any storage

init_llm maps 9 provider integrations plus any OpenAI-compatible endpoint. Memory → SQLite → JSONL → Postgres → MySQL → MSSQL → Oracle → MongoDB → Supabase — or subclass one ABC with four methods.

🧵

Measured no-loss

Verbatim extraction preserves 100% of planted facts at scale (83,023-fact stress test at 6.7M chars) where naive summarization drops 2–10%.

⚡

Parallel map

max_workers=8 fans chunk calls across a thread pool with shared rate-limit pacing. Measured ≈4×.

🔁

Checkpoint / resume

Map results are checkpointed per chunk. resume=True after a crash re-maps only the missing chunks — nothing twice, nothing lost.

📡

Observability built in

on_event= emits structured events with per-call token usage and durations. Retry-After, jitter and pacing are built in.

🗄️

Lives in YOUR database

Embed into existing tables with table=, columns= and multi-tenant namespace=. MSSQL and Oracle dialects included.

🛡️

Enterprise hygiene

Explicit credentials (never read from the environment), pluggable PII scrubber, retention sweeps, advisory conflict detection.

How it works

The context window only ever sees chunks and compact intermediate results — the document never has to fit.

CHUNK

line-aware splitting — a fact on one line is never cut in half

MAP

one LLM call per chunk — parallel, checkpointed per chunk

MERGE

hierarchical combine in window-sized groups

REPAIR

bounded condensation, only if still too big

verified at the end of every run: chunk losslessness, fact preservation, token accounting

83,023

facts preserved —
100% at 6.7M chars / 2.16M tokens

8 × 2

storage targets × real LLM
providers — matrix verified

≈4×

parallel map speedup,
measured on real API

tests, incl. real-API and
real-database runs

60-second quickstart

One import. The context window only ever sees chunks and compact intermediate results — the document never has to fit.

# pip install longloom
import os
from longloom import MapReduceProcessor, SQLiteContextStore, init_llm, PromptTemplates

processor = MapReduceProcessor(
    llm=init_llm("nvidia", "nvidia/nemotron-3-ultra-550b-a55b",
                 api_key=os.environ["NVIDIA_API_KEY"], thinking=False),
    store=SQLiteContextStore("ctx.db"),
    max_chunk_tokens=3000, max_context_tokens=8192,
    prompts=PromptTemplates.for_extraction(),   # no-loss verbatim mode
    max_workers=4,                              # parallel map
)

result = processor.process(annual_report, task="Summarize the key findings")
print(result["final_result"])
print(result["sources"])                       # provenance per chunk

Weave documents of any lengththrough LLM map/merge/repair