LLM utilities

LLM related utils

See devdocs/system/structured-output-fallback.md; ADRs 0014 (runtime model switching), 0017 (invoke retry) and 0028 (prompt cache).

File: klea_rag/llm.py

Copyright 2026 Ankur Sinha Author: Ankur Sinha <sanjay DOT ankur AT gmail DOT com>

class klea_utils.llm.CredentialScope(provider: str, endpoint: str | None = None)

Bases: NamedTuple

Provider (and, for custom endpoints, endpoint) a credential belongs to.

API keys are credentials for a provider – and for an explicit or custom: endpoint, that endpoint – not for a role or a model. Two models sharing a provider (openai:gpt-4o and openai:gpt-4.1) share a scope; custom:m:<url> and any http(s) endpoint suffix scope to that URL.

endpoint: str | None

Alias for field number 1

provider: str

Alias for field number 0

class klea_utils.llm.CustomEndpoint(base_url: str, model_provider: str, use_responses_api: bool | None = None)

Bases: NamedTuple

Resolved API surface for a custom: model endpoint URL.

base_url is the URL to hand to the provider SDK (the endpoint resource path is stripped, because the SDK appends it again). model_provider is the LangChain provider to use, and use_responses_api selects the OpenAI Responses API when True (None leaves the SDK default).

base_url: str

Alias for field number 0

model_provider: str

Alias for field number 1

use_responses_api: bool | None

Alias for field number 2

klea_utils.llm.DEFAULT_MAX_OUTPUT_TOKENS = 32768

Fallback max output tokens used when no node/role default provides a value.

Deliberately generous: reasoning models spend most of the completion budget on hidden reasoning tokens, so a small cap truncates even tiny structured payloads and forces a slow grow-ladder retry. The value is a ceiling, not a target, and is clamped per model by resolve_output_token_limit() (catalog output limit and context headroom), so a large default is safe for small models too. HuggingFace is the exception: its total-budget accounting makes a large reservation trigger usage limits, so it keeps the smaller HUGGINGFACE_MAX_OUTPUT_TOKENS default and relies on retries.

klea_utils.llm.HUGGINGFACE_DEFAULT_INFERENCE_PROVIDER = 'auto'

Inference provider used for the hosted backend when none is given (the HuggingFace router’s default selection).

klea_utils.llm.HUGGINGFACE_LOCAL_SUFFIX = 'local'

Suffix selecting the local (pipeline) HuggingFace backend.

klea_utils.llm.HUGGINGFACE_MAX_OUTPUT_TOKENS = 16384

Conservative HuggingFace default. HuggingFace-style providers account the reserved output window against the context and rate-limit large reservations, so keep the first attempt small and let the grow-ladder handle a genuinely truncated completion.

klea_utils.llm.HUGGINGFACE_PROVIDER = 'huggingface'

HuggingFace provider id. Its backend is selected by the third model-string segment: huggingface:<model> (or an inference provider / routing policy) uses the hosted Inference Providers API, while huggingface:<model>:local runs the model locally with the pipeline backend.

klea_utils.llm.HUGGINGFACE_ROUTING_POLICIES: frozenset[str] = frozenset({'cheapest', 'fastest', 'preferred'})

HuggingFace routing policies. Unlike a concrete provider name these must be appended to the model id: the client’s provider argument accepts only auto or a concrete provider name, not a policy.

class klea_utils.llm.HuggingFaceBackend(backend: str, model_id: str, kwargs: dict[str, Any])

Bases: NamedTuple

Resolved HuggingFace backend for a model string suffix.

backend is the ChatHuggingFace backend ("pipeline" or "endpoint"), model_id is the repo id to pass (with a routing policy appended for the hosted backend), and kwargs are the from_model_id kwargs to inject (backend and, for the hosted backend, provider).

backend: str

Alias for field number 0

kwargs: dict[str, Any]

Alias for field number 2

model_id: str

Alias for field number 1

klea_utils.llm.INPUT_TOKEN_ESTIMATE_SAFETY_FACTOR = 1.05

Safety factor applied on top of the char-based token estimate so the reserved output window keeps headroom for tokenizer variance: the ~4 chars/token rule can undercount the real token count (e.g. 9736 chars estimate 2434 tokens while the server tokenizes 2435), which pushes input + output one token over the model’s context window and fails the request.

class klea_utils.llm.LLMModel(*, model_name: str = '', model_type: Literal['text_generation', 'embedding'] = 'text_generation', instance: Any, role_defaults: dict[str, Any] = {}, provider_defaults: dict[str, dict[str, Any]] = {}, modifiable: bool = True, required: bool = True, user_agent: str = '')

Bases: BaseModel

Container for a single LLM model instance and its runtime configuration.

instance holds the model object (typically a _ConfigurableModel returned by init_chat_model). role_defaults stores role-wide default parameters (e.g. max_tokens, temperature) that apply to every node sharing this role, unless overridden by node or user config.

build_config() performs a five-layer merge:

Layer 0 — role_defaults: role-wide parameters (e.g. {"max_tokens": 4096}).

Layer 1 — model_name: the default model identifier from the graph config.

Layer 2 — context_overrides: per-request fields from the API (model, api_key, etc.). Only applied when modifiable=True, and skipping any keys frozen by node defaults.

Layer 3 — node_defaults: frozen per-node defaults (always win).

Layer 4 — provider_defaults: per-provider defaults from the graph config (e.g. HuggingFace role budgets), applied after the model string is parsed so the resolved provider is known. Applied with setdefault so explicit role/context/node values always win.

modifiable controls whether the model can be changed at runtime (both the API and web UI reject modifications to locked roles). Set to False to lock a role (e.g. guard) against user overrides in managed deployments.

required marks roles that need a default model for the app to function (e.g. chat). At startup, required roles with an empty model trigger a warning (not a failure) listing the environment variables to set. Optional roles (e.g. guard) are skipped when their model is empty.

build_config(context_overrides: dict[str, Any] | None = None, node_defaults: dict[str, Any] | None = None, session_id: str | None = None) → RunnableConfig

Merge up to five layers of model configuration into a RunnableConfig.

Layer order (lowest -> highest priority):

0. self.role_defaults — role-wide parameters

1. self.model_name — role model from graph config

2. context_overrides — per-request user overrides

3. node_defaults — frozen per-node defaults

4. self.provider_defaults — per-provider defaults (setdefault)

Parameters:

• context_overrides – Per-request fields from the API (e.g. model, api_key). Only applied when self.modifiable is True, and skipping any keys present in node_defaults.

• node_defaults – Frozen per-node defaults (e.g. {"temperature": 0.3}). Always win.

• session_id – Per-conversation session id (LangGraph thread_id) used for the opencode x-opencode-session header.

Returns:

A RunnableConfig with the configurable key populated.

model_config = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_type: Literal['text_generation', 'embedding']

Inference kind the role’s model must provide. text_generation covers chat LLMs (chat/plan/tool_picker/guard), resolved through create_configurable_model; embedding covers vector embedders, resolved through setup_embedding(). Reserved future values: decision (System One / Jev-style typed-decisions), reranking and classification.

user_agent: str

Klea User-Agent sent to OpenAI-compatible endpoints (klea-agent/<v>).

klea_utils.llm.PROVIDERS_WITHOUT_API_KEY: frozenset[str] = frozenset({'ollama'})

Providers that authenticate locally/anonymously and therefore need no API key. Used so readiness checks do not flag e.g. a local Ollama deployment as missing a credential.

klea_utils.llm.PROVIDERS_WITHOUT_CUSTOM_URL: frozenset[str] = frozenset({'huggingface', 'ollama', 'openrouter'})

Providers that do not take a custom URL in the model string. For ollama / openrouter the third segment is a model tag; for huggingface it is the inference provider (huggingface:org/model:novita). The model UIs hide the URL field for these.

klea_utils.llm.PROVIDERS_WITH_MODEL_TAG: frozenset[str] = frozenset({'ollama', 'openrouter'})

Providers where model:tag is part of the model id itself (not a Klea suffix). Example: ollama:qwen3:0.6b, openrouter:org/model:free.

class klea_utils.llm.ParsedModelName(provider: str | None, model_name: str, suffix: str | None, variant: str | None = None)

Bases: NamedTuple

Parsed components of a model name string.

model_name: str

Alias for field number 1

provider: str | None

Alias for field number 0

suffix: str | None

Alias for field number 2

variant: str | None

Model variant. Not supported yet, so None for now.

klea_utils.llm.add_memory_to_prompt(context_summary: str) → str

Add the context summary to the system prompt.

Returns a text block framing the previous-context summary. Recent conversation messages are no longer flattened into this block: they are injected as real message objects by the node’s prompt assembly (see klea_utils.nodes.base).

The summary is dynamic text concatenated into the system prompt template, so its braces are escaped (escape_prompt_braces()); a summary that echoes a JSON/code fragment from the conversation would otherwise be parsed as a template variable.

Parameters:

context_summary – Summary of the past conversation.

Returns:

Prompt text block, or "" when there is no summary.

klea_utils.llm.apply_provider_overrides(overrides: dict[str, Any], *, session_id: str | None = None, user_agent: str = '') → None

Add provider-specific request headers to a configurable dict (in place).

Called once at the end of LLMModel.build_config(). Only providers with a registered builder are affected; any default_headers already in overrides win over ours.

Parameters:

• overrides – The merged config["configurable"] dict.

• session_id – Per-conversation session id (LangGraph thread_id).

• user_agent – Klea User-Agent (see resolve_user_agent()).

klea_utils.llm.check_model_works(model, timeout=30, retries=5)

Check if a model works since it is not tested when loaded

klea_utils.llm.check_ollama_model(logger, model, exit=False)

Check if ollama model is available

Parameters:

• logger (logging) – logger instance

• model (str) – ollama model name

• exit (bool) – if we should call sys.exit if check fails

Returns:

None

Throws ollama.ResponseError:

if model is not available

Throws ConnectionError:

if cannot connect to an Ollama server

klea_utils.llm.classify_llm_invocation_error(exc: BaseException) → LLMInvocationErrorCategory

Classify an LLM invocation exception into a category.

Providers report failures inconsistently, so this uses tolerant regex matching over the exception message and its cause chain. Categories are checked in order of specificity (context overflow first), so a message matching several heuristics lands in the most actionable bucket.

Parameters:

exc – The exception raised by an LLM invocation.

Returns:

The best-effort klea_utils.errors.LLMInvocationErrorCategory.

klea_utils.llm.clear_structured_output_cache() → None

Clear the process-local structured-output capability cache (tests).

klea_utils.llm.content_to_str(content: str | list[dict | str] | None) → str

Normalise an AIMessage.content value to a plain string.

AIMessage.content can be a plain string, a list of content blocks (when the LLM returns tool calls or structured output), or None. This helper always returns a string suitable for downstream text processing (regex, in checks, prompt interpolation, etc.).

Parameters:

content – The raw .content value from an AIMessage.

Returns:

A plain string.

klea_utils.llm.create_configurable_model(logger: Logger)

Set up a configurable chat model.

Creates a _ConfigurableModel with no default model. Model, provider, and all other parameters (base_url, api_key, temperature, etc.) are specified per-invoke via the config["configurable"] dict passed to ainvoke().

This enables runtime model switching — each ainvoke() call creates a fresh underlying model instance for the given provider, so there is no stale configuration leakage between calls.

The lookup function check_model_works is deliberately not called here — we prefer a “leap before you look” approach so that startup is fast and model availability is checked only at query time.

klea_utils.llm.credential_scope(model_name: str) → CredentialScope

Return the credential scope for model_name.

Parameters:

model_name – Provider-prefixed model string, e.g. openai:gpt-4o or custom:model:https://host/v1.

Returns:

CredentialScope – a lowercased provider and an optional endpoint for custom / explicit-URL models.

klea_utils.llm.escape_prompt_braces(text: str) → str

Escape { and } for safe embedding in a prompt template.

ChatPromptTemplate parses {...} in its template strings as input variables, so dynamic text concatenated into a template (a context summary, a generated JSON schema, configured domain text) must be escaped or a brace in it raises INVALID_PROMPT_INPUT. Text passed as a template variable value, and message objects placed in the prompt, are not templated and need no escaping.

Parameters:

text – Dynamic text to embed in a prompt template.

Returns:

The text with braces doubled ({ -> {{, } -> }}).

klea_utils.llm.estimate_input_tokens(input_chars: int) → int

Conservative token estimate for a prompt’s character count.

Used to keep the reserved output window within a model’s total budget (input + output <= context). ~4 characters per token is a reasonable average for mixed English/code text; an exact count would require provider-specific tokenizers. The estimate is deliberately inflated by INPUT_TOKEN_ESTIMATE_SAFETY_FACTOR (rounded up) so a tokenizer that counts more tokens than the average never pushes the request over the context window.

Parameters:

input_chars – Number of characters in the prompt.

Returns:

Conservative estimated token count.

klea_utils.llm.extract_llm_output_content(output: AIMessage | dict) → str

Extract plain-text content from an LLM output.

Handles both AIMessage (non-structured output) and dict (structured output with raw / parsed keys):

• AIMessage – returns content_to_str(message.content).

• dict – extracts the raw AIMessage from a structured output response and returns its content; falls back to output["parsed"] and finally str(output).

Parameters:

output – The raw output from llm.invoke().

Returns:

A plain-text string.

klea_utils.llm.format_alert(text: str, level: str = 'warning') → str

Wrap text as a GitHub-style markdown alert (e.g. > [!WARNING]).

Multi-line text is prefixed per line so the whole thing stays inside the blockquote. Renderers with the markdown2 alerts extra (the NiceGUI speech bubbles) show it as a styled callout; others fall back to a plain blockquote.

Parameters:

• text – Alert body text

• level – Alert level (note, tip, important, warning, caution)

Returns:

Markdown alert blockquote

klea_utils.llm.get_last_n_conversations(all_messages, start: int = 0, stop: int | None = None) → tuple[str, list[BaseMessage]]

Get recent conversations between start and stop indices.

Returns the conversation as a single text block (used as prompt/summary input) along with the ordered BaseMessage objects, preserving the interleaved user/assistant order of the original history.

Parameters:

• all_messages – all the messages

• start – start index

• stop – stop index

Returns:

(conversation, ordered list of human/ai messages)

klea_utils.llm.get_provider_allowed_fields(provider: str) → set[str]

Return the set of init-param names accepted by a given provider’s model class.

Uses LangChain’s internal provider registry to look up the Pydantic model class and introspect its fields (including aliases so that both api_key and openai_api_key pass through).

Falls back to an empty set if the provider is not registered in LangChain’s built-in providers. Raises ImportError if the provider’s integration package is not installed — callers should handle this at configuration time, not silently fall through.

The caller should always include {"model", "model_provider"} on top of the returned set since those are consumed by _ConfigurableModel before reaching the model constructor.

klea_utils.llm.get_recent_messages(messages, max_chars: int, keep_at_least: int = 1) → list[BaseMessage]

Return the most recent human/ai messages bounded by max_chars.

Walks backwards through messages (in their original interleaved order) accumulating pretty_repr() length until max_chars would be exceeded. The first keep_at_least messages are always included, so the latest exchange is never dropped even when it alone exceeds the budget.

Parameters:

• messages – All conversation messages.

• max_chars – Maximum total characters of the returned window.

• keep_at_least – Minimum number of messages to always include.

Returns:

Ordered list of recent human/ai messages.

klea_utils.llm.get_token_limit_param(provider: str) → str

Return the max-output token parameter name for a provider.

Providers disagree on the parameter name for the maximum number of output tokens: Ollama uses num_predict, while HuggingFace’s ChatHuggingFace (which internally maps it to max_new_tokens) and other OpenAI-compatible providers all use max_tokens.

Note

Known benign warning

When Klea resolves max_tokens for HuggingFace, the inner HuggingFaceEndpoint constructed by ChatHuggingFace.from_model_id (which declares max_new_tokens, not max_tokens) logs WARNING! max_tokens is not default parameter and shuffles it into model_kwargs. This is a false positive: the limit is still delivered correctly as max_tokens to InferenceClient.chat_completion via the outer ChatHuggingFace, which is the parameter the HuggingFace Inference API actually accepts. Do not “fix” it by switching to max_new_tokens here.

Parameters:

provider – Klea provider id (huggingface, ollama, …)

Returns:

The token parameter name to send in the invoke config.

klea_utils.llm.is_empty_structured_parse_error(exc: BaseException) → bool

Return True if exc is a structured-output parse error on blank text.

On the structured path, with_structured_output(...).ainvoke raises OutputParserException for a blank model response instead of returning a dict, so is_output_empty() never sees it and the empty-response retry in BaseLLMNode._invoke_with_retries is skipped. LangChain’s parsers phrase the failure as "Invalid json output: ..." followed by the (here empty) raw text, so match that shape rather than treating every parser error as retryable: a non-empty invalid payload is a real schema mismatch that should not be retried.

Parameters:

exc – The exception raised by the structured-output invoke.

Returns:

True when the parser failed because the response was blank.

klea_utils.llm.is_output_empty(output: AIMessage | dict[str, Any]) → bool

Return True if an LLM output carries no usable text content.

Some providers (notably HuggingFace) intermittently return a successful response with blank content. Handles both plain AIMessage outputs and structured-output dicts ({"raw": AIMessage, "parsed": ...}); for the structured form the raw message is inspected, so a blank raw response is flagged even when the parser produced an all-default instance.

Parameters:

output – The raw output from llm.invoke().

Returns:

True when the output has no non-whitespace text.

klea_utils.llm.is_output_truncated(output: AIMessage | dict[str, Any]) → bool

Return True if an LLM output was truncated by the max-token limit.

Providers signal truncation via finish_reason == "length" on the message metadata. Handles both plain AIMessage outputs and structured-output dicts ({"raw": AIMessage, ...}), and the list-form finish_reason some providers return.

Parameters:

output – The raw output from llm.invoke().

Returns:

True when the model stopped because it hit the output cap.

klea_utils.llm.is_structured_capability_rejection(exc: BaseException) → bool

Return True if exc is a stable capability rejection of structured output.

Stricter than the STRUCTURED_OUTPUT_REJECTED category returned by classify_llm_invocation_error(): only messages naming response_format / json_schema / structured output qualify. The category’s generic 400 invalid_request_error pattern is excluded so an unrelated bad request cannot poison the capability cache.

Parameters:

exc – The exception raised by the structured invoke.

Returns:

True when the endpoint will keep refusing the parameter.

klea_utils.llm.is_structured_output_failure(exc: BaseException) → bool

Return True if exc means the structured call produced no usable output.

The structured path (with_structured_output(...).ainvoke) can fail in ways the plain invoke cannot: the provider rejects the response_format parameter, or the model returns content that does not satisfy the schema. Some SDKs surface the latter as a pydantic ValidationError (the OpenAI client validates JSON inside its own parser) rather than a LangChain parser error, so both types are treated as a structured failure.

Detection is by exception type wherever possible, so a provider’s new error wording does not need a new pattern; the message-classified cases are STRUCTURED_OUTPUT_REJECTED (parameter refusal), which has no distinct exception type, and any HTTP 400 from the structured call. The latter covers OpenAI-compatible gateways that return a bare 400 (e.g. only {"model": ...}) when an upstream cannot process the parameter, leaving the SDK message as just "Error code: 400". It is a per-call fallback only: because the same status also covers unrelated bad requests, it must not populate the capability cache.

Empty responses are deliberately included: the caller decides whether to retry them on the structured path or fall back to plain.

Parameters:

exc – Exception raised by the structured invoke.

Returns:

True when the structured call should give way to a plain invoke.

klea_utils.llm.join_model_string(provider: str | None, model_name: str, suffix: str | None = None) → str

Join model name components back into a model string.

The inverse of parse_model_name(), used by the model selection UIs on save. Gives provider:model_name, or provider:model_name:suffix when a suffix (e.g. a custom endpoint URL) is set.

Without a provider the model name is returned as it is, so a full model string typed straight into the model field still works.

Parameters:

• provider – Klea provider id, e.g. "openai".

• model_name – Model id within the provider, e.g. "gpt-4o".

• suffix – Optional third segment.

Returns:

The model string, or "" when no model name is set.

klea_utils.llm.load_prompt(prompt_name: str, prompt_registry_location: str)

Load a prompt from file called prompt_name.md

Parameters:

• str – prompt file name

• prompt_registry_location – location of prompts folder/registry

Returns:

loaded prompt text

klea_utils.llm.mark_structured_output_unsupported(provider: str, model: str, base_url: str | None = None) → None

Record a stable capability rejection for the process lifetime.

Parameters:

• provider – Resolved model_provider.

• model – Resolved model identifier.

• base_url – Endpoint base URL, when one applies.

klea_utils.llm.missing_required_roles(models: Mapping[str, Any]) → list[str]

Return the roles that are required but have no model configured.

Single source of truth for the “is a required model missing?” predicate, shared by the graph’s startup warning (BaseLangGraph._check_required_models) and the frontends (which need it to decide whether a query may be sent and to point the user at the model selector).

Values may be either the resolved API config dict returned by the model endpoints (e.g. GET /chat/{user_id}/models/active, with model / required keys) or LLMModel instances (with model_name / required). A role is missing when required is true (the default) and its model is empty. Roles are returned in the input’s iteration order.

Parameters:

models – Mapping of role name to its config dict or object.

Returns:

Required role names with no model set.

klea_utils.llm.parse_model_name(raw: str) → ParsedModelName

Split a model name into provider, model identifier, and suffix.

Follows the provider:model_id convention. The provider is expected to be explicitly included; no provider inference is done.

With three segments the third is usually a suffix (provider hint, URL, and so on). For ollama and openrouter it is kept on the model name instead, because those providers use model:tag ids.

Examples:

• ollama:bge-m3:latest -> provider=ollama, model=bge-m3:latest, suffix=None

• openrouter:recursive/flint-v1.0:free -> provider=openrouter, model=recursive/flint-v1.0:free, suffix=None

• huggingface:org/model:auto -> provider=huggingface, model=org/model, suffix=auto

• custom:model:https://example.com/v1 -> provider=custom, model=model, suffix=https://example.com/v1

• openai:gpt-4o -> provider=openai, model=gpt-4o, suffix=None

• bge-m3 -> provider=None, model=bge-m3, suffix=None

Parameters:

raw – Model name with optional provider prefix

Returns:

Parsed model name components

klea_utils.llm.parse_output_with_thought(message: AIMessage, schema: type[TSchema]) → tuple[TSchema, str]

Parse AI message with thought to a dict based on given schema

klea_utils.llm.prompt_value_to_messages(prompt: PromptValue) → list[dict]

Convert a PromptValue to a clean list of message dicts.

Each dict has role and content keys, suitable for JSON serialisation in the inspector debug panel.

Parameters:

prompt – The LangChain PromptValue (filled, variables already substituted).

Returns:

A list of {"role": "...", "content": "..."} dicts.

klea_utils.llm.provider_api_key_env(provider: str) → str | None

Return the env var holding provider’s API key, or None.

Providers that need no key (see requires_api_key()) return None. Otherwise mirrors the SDK convention so the UI can report whether a credential comes from the environment.

klea_utils.llm.requires_api_key(model_name: str) → bool

Return whether model_name needs an API key.

Accepts either a bare provider id (openai, ollama) or a full model string (openai:gpt-4o, huggingface:org/model:local). Local providers (Ollama) and locally-run HuggingFace models (huggingface:<model>:local) authenticate locally and need no credential; every other provider does. A string that cannot be parsed is judged by its provider part; a string with no : is treated as a provider id.

klea_utils.llm.resolve_catalog_provider_endpoint(provider: str, model_name: str | None = None) → CustomEndpoint | None

Resolve a catalog provider to a wire surface via its models.dev entry.

Lets a user write provider:model (e.g. openrouter:...) without an explicit URL: the catalog’s api supplies the base URL and its npm package identifies the wire protocol. The wire surface is resolved from the npm:

• @ai-sdk/anthropic -> Anthropic Messages API (/v1/messages);

• @ai-sdk/openai -> OpenAI Responses API (/responses);

• @ai-sdk/openai-compatible (and anything else with an api) -> OpenAI Chat Completions, the catalog’s overwhelming default.

The npm is resolved per model when a model name is given, so gateway providers that serve different models on different surfaces (OpenCode, OpenRouter, …) pick the right one – the catalog records this as a per-model provider.npm override.

The returned base_url is what the SDK appends its resource path to. For the Anthropic surface the catalog api typically already ends in /v1 (e.g. .../anthropic/v1) while ChatAnthropic re-appends /v1/messages, so a trailing /v1 is stripped here; a full .../v1/messages is handled by resolve_custom_endpoint().

Returns None when the provider is not in the catalog, has no api (native SDK providers resolve their own endpoint), uses a surface Klea does not implement (e.g. @ai-sdk/google), or the catalog is unavailable – callers then leave the model string to LangChain.

Parameters:

• provider – Klea provider id (e.g. "openrouter").

• model_name – Model identifier within the provider, or None.

Returns:

The resolved CustomEndpoint, or None.

klea_utils.llm.resolve_custom_endpoint(url: str) → CustomEndpoint

Detect the wire API surface from a custom: model endpoint URL.

The suffix of url selects the API surface so that one custom: model string works for any OpenAI-compatible endpoint, the OpenAI Responses API, or the Anthropic Messages API (e.g. opencode Go serves different models on different surfaces). Because each provider SDK appends its own resource path, the matched suffix is stripped from the returned base_url:

• .../chat/completions -> OpenAI Chat Completions (openai)

• .../responses -> OpenAI Responses API (openai, responses)

• .../v1/messages -> Anthropic Messages API (anthropic; the SDK re-appends /v1/messages, so .../zen/go is returned)

Any other URL is treated as a plain base URL and defaults to the standard OpenAI Chat Completions surface (openai, flag unset), preserving the behaviour for self-hosted OpenAI-compatible endpoints.

Parameters:

url – The custom: model string suffix (a base URL or a full endpoint URL).

Returns:

The resolved CustomEndpoint.

Raises:

ValueError – If url is empty.

klea_utils.llm.resolve_huggingface_backend(model_name: str, suffix: str | None) → HuggingFaceBackend

Resolve the HuggingFace backend from a model name and its suffix.

:local selects the local pipeline backend; any other suffix is an inference provider (auto when absent) for the hosted Inference Providers API. The routing policies (HUGGINGFACE_ROUTING_POLICIES) are appended to the model id instead of being passed as provider, because the HuggingFace client’s provider argument accepts only auto or a concrete provider name.

Parameters:

• model_name – HuggingFace repo id, e.g. "openai/gpt-oss-20b".

• suffix – Third model-string segment, or None.

Returns:

The resolved HuggingFaceBackend.

klea_utils.llm.resolve_langchain_endpoint(instance: Any, config: RunnableConfig) → str | None

Resolve the base URL/endpoint of a configurable chat model.

create_configurable_model returns a generic _ConfigurableModel whose concrete provider instance is only built at invoke time (via its private _model(config) method, which is already called on every ainvoke). Native providers (mistral:, anthropic:, deepseek:, …) do not carry a base_url in the configurable dict – the provider resolves its own default endpoint internally – so to probe such an endpoint we materialise the concrete instance and read its resolved endpoint attribute.

Cheap by construction: _model(config) only parses the model string and constructs the provider object (module import is cached per provider); no network or API calls happen at construction.

Parameters:

• instance – The configurable model (_ConfigurableModel) whose concrete instance to materialise.

• config – Per-invoke RunnableConfig carrying the merged configurable dict.

Returns:

The resolved base URL, or None when the concrete model exposes no known endpoint attribute (e.g. plain openai: models, whose default URL lives inside the OpenAI SDK client, not on the model object).

klea_utils.llm.resolve_output_token_limit(overrides: dict[str, Any], provider: str, role: str | None = None, input_chars: int | None = None, *, use_endpoint: bool = False) → None

Ensure a bounded max-output token param is set in overrides.

HuggingFace-style providers apply a total budget: the reserved output window (max_new_tokens) is accounted against the model’s context window alongside the input, and an unset value makes them reserve the entire remaining window (causing spurious usage limits and rate limiting). This helper guarantees a finite, clamped value.

Resolution precedence:

1. An explicit provider token param (max_tokens / max_new_tokens / num_predict) already present in overrides (user/node/role value).

2. The generic max_output_tokens key (provider-agnostic count).

3. The built-in per-role fallback for role and provider: non-HF providers get the generous DEFAULT_MAX_OUTPUT_TOKENS; HuggingFace gets the conservative HUGGINGFACE_MAX_OUTPUT_TOKENS because large reservations count against its total budget and trigger usage limits.

The resolved value is clamped to min(value, catalog limit.output) and, when the catalog exposes a context window and input_chars is given, to the remaining budget (context - estimated input tokens) so HuggingFace’s total-budget check is never exceeded.

The context source depends on use_endpoint: for endpoints with a known base_url (custom OpenAI-compatible endpoints, or a native provider endpoint resolved via resolve_langchain_endpoint()), max_model_len from the live /models probe is authoritative (models.dev’s values are per-deployment and may under-report). The normal (non-retry) path passes use_endpoint=False so it stays on the fast, offline models.dev value – its purpose is only to produce a finite budget (mainly for the HuggingFace whole-window reservation), not an exact cap. The retry path passes use_endpoint=True to clamp against the real server context, falling back to models.dev when the endpoint probe fails. The output clamp always uses models.dev (the endpoint exposes no separate output cap).

Parameters:

• overrides – The merged configurable dict to update in place.

• provider – Klea provider id (huggingface, ollama, …).

• role – Model role (e.g. "chat"), used for the built-in per-role, per-provider fallback.

• input_chars – Character count of the prompt, to bound the output within the total budget.

• use_endpoint – When True, prefer the live endpoint’s max_model_len as the context source, falling back to models.dev. When False (default), use models.dev only and never query the endpoint.

klea_utils.llm.resolve_user_agent(prefix: str) → str

Return <prefix>/<version> for the running application.

Mirrors web_fetch.honest_user_agent: the version is read from the installed distribution named prefix (importlib.metadata normalises separators, so the graph name doubles as the distribution name – klea-agent -> klea_agent) and falls back to dev when metadata is unavailable (e.g. an uninstalled checkout).

Parameters:

prefix – Application name, e.g. "klea-agent".

Returns:

The User-Agent string, e.g. "klea-agent/0.0.1".

klea_utils.llm.split_output_by_section(text: str, section_start_marker: str, section_end_marker: str | None = None)

Split out thoughts and actual responses from AI responses

klea_utils.llm.structured_output_known_unsupported(provider: str, model: str, base_url: str | None = None) → bool

Return True if this endpoint was seen refusing structured output.

Consulted before building with_structured_output so a rejecting endpoint is not re-probed on every node call. A miss (False) may still reject – the first attempt for a key always probes.

Parameters:

• provider – Resolved model_provider (e.g. "openai").

• model – Resolved model identifier.

• base_url – Endpoint base URL, when one applies.

Returns:

True when the key is known to reject response_format.