Model catalog

Client for the models.dev catalog, providing per-model context and output token limits with an in-memory + on-disk cache and graceful fallback when the catalog is unavailable.

Client for the models.dev model catalog.

Fetches https://models.dev/api.json (a ~3MB JSON mapping of provider -> model -> properties) and exposes the per-model token limits used to bound LLM output token reservations. This is needed because some providers (e.g. HuggingFace) reserve the whole context window as output when no max-token parameter is set, which leads to spurious usage limits and rate limiting. The provider and model lists are also exposed, so the model selection UIs can offer them as suggestions.

The catalog is fetched lazily on first use, kept in memory for the process lifetime (lru_cache), and mirrored to an on-disk cache ({user_cache_dir}/klea/models-dev.json) with a one-day TTL so that restarts do not need to re-download it.

Providers without a catalog entry (local ollama, unknown custom endpoints) and models missing from the catalog resolve to None so callers can fall back gracefully instead of failing.

File: klea_utils/models_catalog.py

Copyright 2026 Ankur Sinha Author: Ankur Sinha <sanjay DOT ankur AT gmail DOT com>

klea_utils.models_catalog.DEFAULT_MODELS_DEV_URL = 'https://models.dev/api.json'

Default models.dev catalog URL. Overridable via the KLEA_MODELS_DEV_URL environment variable (e.g. for offline mirrors or enterprise proxies).

klea_utils.models_catalog.DISK_CACHE_FILE = 'models-dev.json'

File name of the on-disk catalog cache, under the platformdirs cache dir.

klea_utils.models_catalog.DISK_CACHE_TTL_SECONDS = 86400

Time-to-live for the on-disk catalog cache, in seconds (1 day).

klea_utils.models_catalog.ENDPOINT_LIMITS_CACHE_TTL_SECONDS = 43200

Time-to-live for the in-memory live-endpoint model-limits cache, in seconds. A model’s max_model_len is a stable property of its deployment and rarely changes, so we cache it for the process lifetime; this TTL is just a safety valve against very long-running servers.

klea_utils.models_catalog.ENDPOINT_LIMITS_FETCH_TIMEOUT_SECONDS = 5.0

it is best-effort and must never stall a query.

Type:

HTTP timeout for the live-endpoint /models probe. Short

klea_utils.models_catalog.FETCH_TIMEOUT_SECONDS = 15.0

this is only hit on the first use after the disk cache expires.

Type:

HTTP timeout for fetching the catalog. Short

klea_utils.models_catalog.MODELS_DEV_PROVIDERS_IGNORED = {'huggingface'}

Provider ids whose endpoint must NOT be taken from the catalog, because Klea has dedicated native handling for them (HuggingFace backend/provider selection and token mapping). Checked before any catalog lookup. Others are covered by the per-provider api guard: anthropic and openai carry no api in the catalog, so they already resolve to None.

class klea_utils.models_catalog.ModelLimits(context: int | None = None, input: int | None = None, output: int | None = None)[source]

Bases: NamedTuple

Token limits for a single model from the catalog.

All fields are optional: the catalog always carries context and output, while input is only defined for a subset of models.

context: int | None

Alias for field number 0

input: int | None

Alias for field number 1

output: int | None

Alias for field number 2

klea_utils.models_catalog.NON_CATALOG_PROVIDERS: tuple[str, ...] = ('ollama', 'custom')

Providers offered by the model selection UIs that have no catalog entry: local ollama models and custom: endpoints. Their models are typed in as free text.

class klea_utils.models_catalog.ProviderEndpoint(api: str | None = None, npm: str | None = None)[source]

Bases: NamedTuple

Endpoint metadata for one models.dev provider (optionally per model).

api is the base URL an OpenAI/Anthropic-compatible client appends its resource path to (/chat/completions, /v1/messages, …), and npm is the AI-SDK package name, which identifies the wire protocol (@ai-sdk/anthropic vs the OpenAI-shaped rest). The catalog’s env field is deliberately not exposed: it names variables for the npm package, not for LangChain.

Some providers (gateways such as OpenCode, OpenRouter, Cloudflare) serve different models on different wire surfaces. The catalog records this as a per-model provider.npm override which, when present, wins over the provider default; get_provider_endpoint() resolves it when given a model name.

api: str | None

Alias for field number 0

npm: str | None

Alias for field number 1

klea_utils.models_catalog.get_catalog_model_limits(provider: str, model_name: str) → ModelLimits | None[source]

Return the token limits for a provider + model, or None.

Returns None when the provider has no catalog entry (local ollama, custom endpoints), the model is missing from the catalog, or the catalog could not be fetched. Callers should treat None as “no information available”.

Parameters:

  • provider – Klea provider id (e.g. "huggingface").

  • model_name – Model identifier, e.g. "gpt-4o" or "Qwen/Qwen3-Coder-30B-A3B-Instruct".

Returns:

ModelLimits with whatever fields the catalog defines.

klea_utils.models_catalog.get_provider_api_key_env(provider: str) → str[source]

Return the API key env var name a LangChain SDK reads for provider.

The provider id here is a LangChain _BUILTIN_PROVIDERS key, and the whole point is to mirror how the LangChain SDKs resolve credentials so the endpoint probe authenticates with the same key the model invoke would use. Follows the {PROVIDER}_API_KEY convention unless an explicit alias is registered in _LANGCHAIN_PROVIDER_API_KEY_ENV.

Also used by the UI to report whether a model’s credential comes from the environment; pass a lowercase Klea model prefix.

klea_utils.models_catalog.get_provider_endpoint(provider: str, model_name: str | None = None) → ProviderEndpoint | None[source]

Return the endpoint metadata for a models.dev provider, or None.

Used to resolve provider:model model strings to an endpoint without the user supplying a URL (the catalog api), mirroring how token limits are read from the same catalog. Returns None when the provider has no catalog entry, the catalog is unavailable, or the entry carries no api (native SDK providers such as groq/mistral resolve their own endpoint).

When model_name is given and the catalog carries a per-model provider.npm override (as gateways do for models served on a different surface, e.g. muse-spark-1.2-contributor on OpenCode Go), that npm is returned instead of the provider default.

Parameters:

  • provider – Klea provider id (e.g. "openrouter").

  • model_name – Model identifier within the provider, or None.

Returns:

ProviderEndpoint, or None when unavailable.

klea_utils.models_catalog.list_catalog_models(provider: str) → list[str][source]

Return the catalog model ids for one provider, sorted.

The ids are what goes after provider: in a model string (e.g. gpt-4o for openai). Ids with a : are normally skipped, because parse_model_name would treat the part after : as a suffix. OpenRouter and Ollama are the exception: their model ids can include a tag, so those catalog entries are kept.

Returns an empty list for NON_CATALOG_PROVIDERS (their models are free text), unknown providers, or when the catalog could not be fetched.

Parameters:

provider – Klea provider id (e.g. "openai").

Returns:

Sorted list of model ids.

klea_utils.models_catalog.list_catalog_providers() → list[str][source]

Return the providers for the model selection UIs, sorted.

Every models.dev provider is listed under its Klea provider id (e.g. google -> google_genai), plus NON_CATALOG_PROVIDERS. Those two are listed even when the catalog is unavailable, so a local ollama model or a custom endpoint can still be picked offline.

Returns:

Sorted list of Klea provider ids.

klea_utils.models_catalog.probe_endpoint_model_limits(provider: str, model_name: str, base_url: str | None, api_key: str | None = None) → ModelLimits | None[source]

Return token limits for a model served by a live OpenAI-compatible endpoint.

models.dev’s limit values are per-deployment configuration, not a property of the model, and private/custom endpoints (e.g. an internal vLLM server) are not in the catalog at all. For those we ask the endpoint directly: GET {base_url}/models returns each model’s max_model_len (the total context window, e.g. 262144 for a large vLLM deployment), which is the authoritative per-deployment value.

Any provider with a known base_url is probed – not just openai custom endpoints. Native providers (mistral:, anthropic:, deepseek:, …) normally carry no base_url in the configurable dict (their SDK resolves a default endpoint internally); callers resolve it via klea_utils.llm.resolve_langchain_endpoint and pass it here. The probe is best-effort: providers whose /models response does not advertise max_model_len (e.g. Anthropic’s non-OpenAI-shaped payload) simply return None and the catalog fallback is used. Results are cached in memory keyed by (base_url, model_name) for ENDPOINT_LIMITS_CACHE_TTL_SECONDS; a user switching to a different model simply misses and triggers a fresh probe for the new key.

Never raises and never blocks a query: any error, missing model, or inapplicable input returns None.

Parameters:

  • provider – Klea provider id (openai for custom endpoints).

  • model_name – Model identifier as served by the endpoint.

  • base_url – Base URL of the OpenAI-compatible endpoint, or None.

  • api_key – Optional bearer token for the endpoint.

Returns:

ModelLimits with context set from max_model_len, or None when the endpoint does not expose it or is not usable.