UI — store creation CLI¶

CLI for creating stores from documents

File: klea_utils/ui/stores_create.py

klea_utils.ui.stores_create.build(source_dir: str = <typer.models.ArgumentInfo object>, collection_name: str = <typer.models.OptionInfo object>, store_path: str = <typer.models.OptionInfo object>, embedding_model: str = <typer.models.OptionInfo object>, max_tokens: int = <typer.models.OptionInfo object>, ocr: bool = <typer.models.OptionInfo object>, metadata_map_path: str = <typer.models.OptionInfo object>, bm25_store: str = <typer.models.OptionInfo object>, embed_batch_size: int = <typer.models.OptionInfo object>, force: bool = <typer.models.OptionInfo object>, worker_mem_limit: float = <typer.models.OptionInfo object>, worker_batch_size: int = <typer.models.OptionInfo object>, debug: bool = <typer.models.OptionInfo object>)[source]¶

Full pipeline: chunk, embed, and write to a vector store.

Processes all files in SOURCE_DIR: converts them with Docling, chunks them, embeds them, and writes to the vector store. This is memory-bounded end-to-end: conversion runs in short-lived worker subprocesses (bounded by --worker-mem-limit) and cached chunks are streamed into the store one file at a time. Processed chunks are cached in <source_dir>/.klea-cache/ so subsequent runs (e.g. with --metadata-map) skip conversion.

The --bm25-store option (default <collection>.pkl in the current directory) writes the combined chunked documents to a single pickle file that can be used as a BM25 store.

The optional --metadata-map / -M flag accepts a JSON file organised by source file. Within each file entry, the most specific heading chain match wins; a DEFAULT entry provides fallback for any heading not listed.

Example metadata-map.json:

{
    "PrimerOnCElegans.md": {
        "DEFAULT": {},
        "C. elegans tissue morphology": {
            "url": "https://example.com/worm"
        }
    },
    "c302-paper.pdf": {
        "DEFAULT": {
            "url": "https://example.com/c302"
        }
    }
}

klea_utils.ui.stores_create.chunk(source_dir: str = <typer.models.ArgumentInfo object>, max_tokens: int = <typer.models.OptionInfo object>, ocr: bool = <typer.models.OptionInfo object>, force: bool = <typer.models.OptionInfo object>, worker_mem_limit: float = <typer.models.OptionInfo object>, worker_batch_size: int = <typer.models.OptionInfo object>, debug: bool = <typer.models.OptionInfo object>)[source]¶

Chunk and cache documents without writing to a vector store.

Converts all files in SOURCE_DIR with Docling, chunks them, and caches the result in <source_dir>/.klea-cache/. Also writes a metadata-map.template.json file organised by source file, with empty {} placeholders for each heading chain. Fill in the metadata values and pass the file to klea-stores-create store --metadata-map.

Uncached files are converted in short-lived worker subprocesses (bounded by --worker-mem-limit), so a run over a very large corpus stays memory-bounded instead of being killed by the kernel when Docling’s per-conversion memory growth is never released. Already-cached files are handled in-process and never spawn workers.

klea_utils.ui.stores_create.debug_option = <typer.models.OptionInfo object>¶: Shared --debug option attached to every command. When given, the console shows full DEBUG logging; otherwise the console stays at INFO (progress on stdout, warnings/errors on stderr). See klea_utils.plogging.resolve_log_level() for the flag/env precedence.

klea_utils.ui.stores_create.map_lint(source_dir: str = <typer.models.ArgumentInfo object>, metadata_map_path: str = <typer.models.OptionInfo object>, debug: bool = <typer.models.OptionInfo object>)[source]¶

Report issues in a metadata map so it can be reviewed efficiently.

Runs only deterministic checks (missing fields, suspicious titles or DOIs, year/filename mismatches, stale ‘venue’ keys, excess url* keys, placeholder counts, and whether the top-level keys are the actual source filenames) – no LLM is needed. Useful after editing the map by hand, and printed automatically at the end of ‘chunk’.

A source file with no map entry is fatal (the store step raises), so the full report is printed first and the command then exits non-zero when any such file is found.

klea_utils.ui.stores_create.pre_check(source_dir: str = <typer.models.ArgumentInfo object>, organise: bool = <typer.models.OptionInfo object>, debug: bool = <typer.models.OptionInfo object>)[source]¶

Decide which PDFs need OCR before chunking.

Reads each PDF’s embedded text layer with pypdfium2 and reports whether it is image-based (scanned, needs OCR) or text-based (no OCR needed). This lets you avoid the cost of OCR on born-digital PDFs while keeping it for older scanned papers – without guessing by publication year.

Without --organise the command only reports. With --organise it copies files into ocr/ and no-ocr/ subdirectories (the originals are never modified), then prints the recommended chunk and store commands to build both into the same collection.

klea_utils.ui.stores_create.store(source_dir: str = <typer.models.ArgumentInfo object>, collection_name: str = <typer.models.OptionInfo object>, store_path: str = <typer.models.OptionInfo object>, embedding_model: str = <typer.models.OptionInfo object>, metadata_map_path: str = <typer.models.OptionInfo object>, bm25_store: str = <typer.models.OptionInfo object>, embed_batch_size: int = <typer.models.OptionInfo object>, force: bool = <typer.models.OptionInfo object>, debug: bool = <typer.models.OptionInfo object>)[source]¶

Write cached document chunks to a vector store.

Cache-only: every source file must already have a cache entry (run klea-stores-create chunk first). Reads the cached chunks from <source_dir>/.klea-cache/, applies the metadata map, and writes them to the vector store. Conversion settings (OCR, max tokens) belong to chunk; store never converts on the fly.

Incremental by default: a store manifest in <source_dir>/.klea-cache/ records which files are in the collection, so unchanged files are skipped and changed files are updated in place. Pass --force to drop the whole collection and rebuild it (documents within a collection cannot be updated in place across all backends).

The --bm25-store option (default <collection>.pkl in the current directory) writes the combined chunked documents to a single pickle file that can be used as a BM25 store.

Run klea-stores-create chunk first to populate the cache and generate a metadata-map.template.json.

klea_utils.ui.stores_create.store_lint(corpus_path: str = <typer.models.ArgumentInfo object>, samples: int = <typer.models.OptionInfo object>, debug: bool = <typer.models.OptionInfo object>)[source]¶

Report issues in a stored corpus so it can be reviewed efficiently.

Loads the pickled BM25 corpus (a list of chunked documents) and runs only deterministic checks (no LLM): a corpus summary, suspicious chunks (near-empty text from a conversion/OCR miss, or missing bibliographic metadata), and structural problems (chunks without a file_name, invalid page_content / year types). Also prints --samples evenly-spaced windows of contiguous chunks so you can eyeball that chunking and metadata look right across the corpus – no need to unpickle the file yourself. Useful after store, and printed automatically at the end of store when a BM25 corpus was written.