klea-stores-create¶
Create stores from documents (vector stores and BM25 stores).
The build and store commands accept --bm25-store <path>,
which writes the combined chunked documents to a single pickle that can
be configured as a bm25_stores entry in the RAG config. When
--bm25-store is omitted, the corpus is written to
<collection>.pkl in the current directory.
The pre-check command classifies each PDF by whether it needs OCR
(based on whether it carries an embedded text layer) and, with
--organise, copies files into ocr/ and no-ocr/ subdirectories
so you can chunk each with the right --ocr / --no-ocr flag. See
Create and use a RAG system for the worked workflow.
The store-lint command reviews a stored corpus (the BM25 pickle) with
LLM-free checks and prints a summary, suspicious chunks, and
--samples evenly-spaced windows of contiguous chunks for human
review. It is printed automatically at the end of store when a BM25
corpus is written.
Three options deserve special attention:
--collection– the collection name inside the store. It must match thenameof the store’svector_stores/bm25_storesentry in the RAG config file (e.g.klea.json); retrieval looks stores up by name, so a mismatch silently returns no results.--store– for local Chroma stores this points at the store folder; the database file inside it is always namedchroma.sqlite3(the filename is not configurable), so passing the path of an existing file is rejected. A folder that does not exist yet is created. One Chroma store file can hold several collections, so--collectionselects which collection within the file is used.--bm25-store– path to the combined corpus pickle (see above).
Usage: klea-stores-create [OPTIONS] COMMAND [ARGS]...
Create stores from documents
╭─ Options ──────────────────────────────────────────────────────────╮
│ --install-completion Install completion for the current │
│ shell. │
│ --show-completion Show completion for the current │
│ shell, to copy it or customize the │
│ installation. │
│ --help Show this message and exit. │
╰────────────────────────────────────────────────────────────────────╯
╭─ Commands ─────────────────────────────────────────────────────────╮
│ pre-check Decide which PDFs need OCR before chunking. │
│ chunk Chunk and cache documents without writing to a vector │
│ store. │
│ map-lint Report issues in a metadata map so it can be reviewed │
│ efficiently. │
│ store Write cached document chunks to a vector store. │
│ store-lint Report issues in a stored corpus so it can be reviewed │
│ efficiently. │
│ build Full pipeline: chunk, embed, and write to a vector │
│ store. │
╰────────────────────────────────────────────────────────────────────╯
Usage: klea-stores-create pre-check [OPTIONS] SOURCE_DIR
Decide which PDFs need OCR before chunking.
Reads each PDF's embedded text layer with pypdfium2 and reports
whether it is image-based (scanned, needs OCR) or text-based (no OCR
needed). This lets you avoid the cost of OCR on born-digital PDFs
while keeping it for older scanned papers -- without guessing by
publication year.
Without ``--organise`` the command only reports. With
``--organise``
it copies files into ``ocr/`` and ``no-ocr/`` subdirectories (the
originals are never modified), then prints the recommended chunk and
store commands to build both into the same collection.
╭─ Arguments ────────────────────────────────────────────────────────╮
│ * source_dir TEXT Directory containing source documents │
│ (PDFs) │
│ [required] │
╰────────────────────────────────────────────────────────────────────╯
╭─ Options ──────────────────────────────────────────────────────────╮
│ --organise Copy classified files into 'ocr/' │
│ (scanned/image PDFs) and 'no-ocr/' (text PDFs │
│ plus all non-PDF files) subdirectories. │
│ Copies, never moves -- your original │
│ bibliography directory is left untouched. │
│ Recommended workflow: relocate the copies to a │
│ scratch directory, then chunk and store each │
│ subdirectory into the same collection │
│ --debug Enable debug logging │
│ --help Show this message and exit. │
╰────────────────────────────────────────────────────────────────────╯
Usage: klea-stores-create chunk [OPTIONS] SOURCE_DIR
Chunk and cache documents without writing to a vector store.
Converts all files in SOURCE_DIR with Docling, chunks them, and
caches the result in ``<source_dir>/.klea-cache/``. Also writes a
``metadata-map.template.json`` file organised by source file, with
empty ``{}`` placeholders for each heading chain. Fill in the
metadata values and pass the file to
``klea-stores-create store --metadata-map``.
Uncached files are converted in short-lived worker subprocesses
(bounded by ``--worker-mem-limit``), so a run over a very large
corpus stays memory-bounded instead of being killed by the kernel
when Docling's per-conversion memory growth is never released.
Already-cached files are handled in-process and never spawn workers.
╭─ Arguments ────────────────────────────────────────────────────────╮
│ * source_dir TEXT Directory containing source documents │
│ [required] │
╰────────────────────────────────────────────────────────────────────╯
╭─ Options ──────────────────────────────────────────────────────────╮
│ --max-tokens INTEGER Maximum tokens per │
│ chunk │
│ [default: 450] │
│ --ocr --no-ocr Whether to perform │
│ optical character │
│ recognition (OCR) │
│ during PDF │
│ conversion (default: │
│ on). Keep for │
│ scanned/image PDFs; │
│ disable for │
│ text-based PDFs to │
│ speed up conversion │
│ significantly │
│ [default: ocr] │
│ --force -f Re-process all files │
│ even if unchanged │
│ --worker-mem-limit FLOAT Maximum memory (in │
│ GiB) a conversion │
│ worker process may │
│ use before it is │
│ restarted. This │
│ includes the ~1-1.5 │
│ GiB Docling's models │
│ occupy at worker │
│ startup, so the │
│ headroom for │
│ Docling's │
│ per-conversion │
│ memory growth is │
│ roughly the limit │
│ minus that. │
│ Uncached files are │
│ converted in │
│ short-lived │
│ subprocesses so that │
│ growth is released │
│ when a worker exits; │
│ raise this on │
│ machines with more │
│ RAM to restart │
│ workers less often │
│ [default: 4.0] │
│ --worker-batch-size INTEGER Maximum files handed │
│ to any single │
│ conversion worker │
│ before it is │
│ restarted. A worker │
│ stops at whichever │
│ comes first: its │
│ memory cap or this │
│ batch size │
│ [default: 200] │
│ --debug Enable debug logging │
│ --help Show this message │
│ and exit. │
╰────────────────────────────────────────────────────────────────────╯
Usage: klea-stores-create map-lint [OPTIONS] SOURCE_DIR
Report issues in a metadata map so it can be reviewed efficiently.
Runs only deterministic checks (missing fields, suspicious titles or
DOIs, year/filename mismatches, stale 'venue' keys, excess url*
keys,
placeholder counts, and whether the top-level keys are the actual
source filenames) -- no LLM is needed. Useful after editing the
map by hand, and printed automatically at the end of 'chunk'.
A source file with no map entry is fatal (the store step raises), so
the full report is printed first and the command then exits non-zero
when any such file is found.
╭─ Arguments ────────────────────────────────────────────────────────╮
│ * source_dir TEXT Directory containing the metadata map │
│ (uses metadata-map.template.json in the │
│ .klea-cache folder unless │
│ --metadata-map is given) │
│ [required] │
╰────────────────────────────────────────────────────────────────────╯
╭─ Options ──────────────────────────────────────────────────────────╮
│ --metadata-map -M TEXT Explicit metadata-map JSON file to │
│ lint, instead of the template in │
│ SOURCE_DIR │
│ --debug Enable debug logging │
│ --help Show this message and exit. │
╰────────────────────────────────────────────────────────────────────╯
Usage: klea-stores-create store [OPTIONS] SOURCE_DIR
Write cached document chunks to a vector store.
Cache-only: every source file must already have a cache entry (run
``klea-stores-create chunk`` first). Reads the cached chunks from
``<source_dir>/.klea-cache/``, applies the metadata map, and writes
them to the vector store. Conversion settings (OCR, max tokens)
belong to ``chunk``; ``store`` never converts on the fly.
Incremental by default: a store manifest in
``<source_dir>/.klea-cache/`` records which files are in the
collection, so unchanged files are skipped and changed files are
updated in place. Pass ``--force`` to drop the whole collection
and rebuild it (documents within a collection cannot be updated in
place across all backends).
The ``--bm25-store`` option (default ``<collection>.pkl`` in the
current directory) writes the combined chunked documents to a single
pickle file that can be used as a BM25 store.
Run ``klea-stores-create chunk`` first to populate the cache and
generate a ``metadata-map.template.json``.
╭─ Arguments ────────────────────────────────────────────────────────╮
│ * source_dir TEXT Directory containing source documents │
│ [required] │
╰────────────────────────────────────────────────────────────────────╯
╭─ Options ──────────────────────────────────────────────────────────╮
│ * --collection -n TEXT Collection name for the │
│ vector store. Must match │
│ the 'name' of the │
│ corresponding │
│ vector_stores/bm25_stores │
│ entry in the RAG config │
│ file (e.g. klea.json); a │
│ different name on an │
│ existing store file │
│ creates a new collection │
│ [required] │
│ * --store -s TEXT Vector store URI (e.g. │
│ chroma:/path/to/store). │
│ For local Chroma stores, │
│ point at the store folder: │
│ the database file inside │
│ it is always named │
│ chroma.sqlite3 │
│ [required] │
│ --model -m TEXT Embedding model identifier │
│ [default: │
│ ollama:bge-m3:latest] │
│ --metadata-map -M TEXT JSON file keyed by source │
│ filename; each file entry │
│ maps heading chains to │
│ metadata dicts (with │
│ per-file DEFAULT │
│ fallback). Defaults to │
│ metadata-map.template.json │
│ in the source directory's │
│ cache folder; required │
│ when no template exists │
│ --bm25-store TEXT Write the combined │
│ document corpus to this │
│ path for BM25 retrieval (a │
│ pickle of all chunked │
│ documents). Defaults to │
│ <collection>.pkl in the │
│ current directory; the │
│ file can be moved after │
│ creation │
│ [default: │
│ (<collection>.pkl)] │
│ --embed-batch-size INTEGER Number of chunks embedded │
│ per store write call. │
│ Smaller values report │
│ progress more frequently; │
│ larger values reduce │
│ per-request overhead on │
│ very large corpora │
│ [default: 256] │
│ --force -f Drop the collection and │
│ re-store all files from │
│ scratch (the portable way │
│ to update a collection; │
│ documents within a │
│ collection cannot be │
│ updated in place). │
│ Without --force, `store` │
│ is incremental: unchanged │
│ files are skipped and │
│ changed files are updated │
│ in place. Does not │
│ reconvert: files must │
│ already be cached by │
│ 'chunk' │
│ --debug Enable debug logging │
│ --help Show this message and │
│ exit. │
╰────────────────────────────────────────────────────────────────────╯
Usage: klea-stores-create store-lint [OPTIONS] CORPUS_PATH
Report issues in a stored corpus so it can be reviewed efficiently.
Loads the pickled BM25 corpus (a list of chunked documents) and runs
only deterministic checks (no LLM): a corpus summary, suspicious
chunks (near-empty text from a conversion/OCR miss, or missing
bibliographic metadata), and structural problems (chunks without a
``file_name``, invalid ``page_content`` / ``year`` types). Also
prints ``--samples`` evenly-spaced windows of contiguous chunks so
you
can eyeball that chunking and metadata look right across the corpus
--
no need to unpickle the file yourself. Useful after ``store``, and
printed automatically at the end of ``store`` when a BM25 corpus was
written.
╭─ Arguments ────────────────────────────────────────────────────────╮
│ * corpus_path TEXT Path to the pickled BM25/vector corpus │
│ (e.g. <collection>.pkl) │
│ [required] │
╰────────────────────────────────────────────────────────────────────╯
╭─ Options ──────────────────────────────────────────────────────────╮
│ --samples INTEGER Number of evenly-spaced sampling │
│ locations across the corpus; each shows │
│ 3 contiguous chunks (truncated text + │
│ metadata) for human review. Pass 0 to │
│ suppress sampling │
│ [default: 3] │
│ --debug Enable debug logging │
│ --help Show this message and exit. │
╰────────────────────────────────────────────────────────────────────╯
Usage: klea-stores-create build [OPTIONS] SOURCE_DIR
Full pipeline: chunk, embed, and write to a vector store.
Processes all files in SOURCE_DIR: converts them with Docling,
chunks them, embeds them, and writes to the vector store. This is
memory-bounded end-to-end: conversion runs in short-lived worker
subprocesses (bounded by ``--worker-mem-limit``) and cached chunks
are streamed into the store one file at a time.
Processed chunks are cached in ``<source_dir>/.klea-cache/`` so
subsequent runs (e.g. with ``--metadata-map``) skip conversion.
The ``--bm25-store`` option (default ``<collection>.pkl`` in the
current directory) writes the combined chunked documents to a single
pickle file that can be used as a BM25 store.
The optional ``--metadata-map`` / ``-M`` flag accepts a JSON file
organised by source file. Within each file entry, the most specific
heading chain match wins; a ``DEFAULT`` entry provides fallback for
any heading not listed.
Example metadata-map.json::
{
"PrimerOnCElegans.md": {
"DEFAULT": {},
"C. elegans tissue morphology": {
"url": "https://example.com/worm"
}
},
"c302-paper.pdf": {
"DEFAULT": {
"url": "https://example.com/c302"
}
}
}
╭─ Arguments ────────────────────────────────────────────────────────╮
│ * source_dir TEXT Directory containing source documents │
│ [required] │
╰────────────────────────────────────────────────────────────────────╯
╭─ Options ──────────────────────────────────────────────────────────╮
│ * --collection -n TEXT Collection name for │
│ the vector store. │
│ Must match the │
│ 'name' of the │
│ corresponding │
│ vector_stores/bm25… │
│ entry in the RAG │
│ config file (e.g. │
│ klea.json); a │
│ different name on │
│ an existing store │
│ file creates a new │
│ collection │
│ [required] │
│ * --store -s TEXT Vector store URI │
│ (e.g. │
│ chroma:/path/to/st… │
│ For local Chroma │
│ stores, point at │
│ the store folder: │
│ the database file │
│ inside it is always │
│ named │
│ chroma.sqlite3 │
│ [required] │
│ --model -m TEXT Embedding model │
│ identifier │
│ [default: │
│ ollama:bge-m3:late… │
│ --max-tokens INTEGER Maximum tokens per │
│ chunk │
│ [default: 450] │
│ --ocr --no-ocr Whether to perform │
│ optical character │
│ recognition (OCR) │
│ during PDF │
│ conversion │
│ (default: on). Keep │
│ for scanned/image │
│ PDFs; disable for │
│ text-based PDFs to │
│ speed up conversion │
│ significantly │
│ [default: ocr] │
│ --metadata-map -M TEXT JSON file keyed by │
│ source filename; │
│ each file entry │
│ maps heading chains │
│ to metadata dicts │
│ (with per-file │
│ DEFAULT fallback). │
│ If not given, build │
│ generates │
│ metadata-map.templ… │
│ from the extraction │
│ and uses it without │
│ a review step │
│ --bm25-store TEXT Write the combined │
│ document corpus to │
│ this path for BM25 │
│ retrieval (a pickle │
│ of all chunked │
│ documents). │
│ Defaults to │
│ <collection>.pkl in │
│ the current │
│ directory; the file │
│ can be moved after │
│ creation │
│ [default: │
│ (<collection>.pkl)] │
│ --embed-batch-si… INTEGER Number of chunks │
│ embedded per store │
│ write call. Smaller │
│ values report │
│ progress more │
│ frequently; larger │
│ values reduce │
│ per-request │
│ overhead on very │
│ large corpora │
│ [default: 256] │
│ --force -f Re-process all │
│ files even if │
│ unchanged │
│ --worker-mem-lim… FLOAT Maximum memory (in │
│ GiB) a conversion │
│ worker process may │
│ use before it is │
│ restarted. This │
│ includes the ~1-1.5 │
│ GiB Docling's │
│ models occupy at │
│ worker startup, so │
│ the headroom for │
│ Docling's │
│ per-conversion │
│ memory growth is │
│ roughly the limit │
│ minus that │
│ [default: 4.0] │
│ --worker-batch-s… INTEGER Maximum files │
│ handed to any │
│ single conversion │
│ worker before it is │
│ restarted. A │
│ worker stops at │
│ whichever comes │
│ first: its memory │
│ cap or this batch │
│ size │
│ [default: 200] │
│ --debug Enable debug │
│ logging │
│ --help Show this message │
│ and exit. │
╰────────────────────────────────────────────────────────────────────╯