Model catalog

Client for the models.dev catalog, providing per-model context and output token limits with an in-memory + on-disk cache and graceful fallback when the catalog is unavailable.

Client for the models.dev model catalog.

Fetches https://models.dev/api.json (a ~3MB JSON mapping of provider -> model -> properties) and exposes the per-model token limits used to bound LLM output token reservations. This is needed because some providers (e.g. HuggingFace) reserve the whole context window as output when no max-token parameter is set, which leads to spurious usage limits and rate limiting.

The catalog is fetched lazily on first use, kept in memory for the process lifetime (lru_cache), and mirrored to an on-disk cache ({user_cache_dir}/klea/models-dev.json) with a one-day TTL so that restarts do not need to re-download it.

Providers without a catalog entry (local ollama, unknown custom endpoints) and models missing from the catalog resolve to None so callers can fall back gracefully instead of failing.

File: klea_utils/models_catalog.py

Copyright 2026 Ankur Sinha Author: Ankur Sinha <sanjay DOT ankur AT gmail DOT com>

klea_utils.models_catalog.DEFAULT_MODELS_DEV_URL = 'https://models.dev/api.json'

Default models.dev catalog URL. Overridable via the KLEA_MODELS_DEV_URL environment variable (e.g. for offline mirrors or enterprise proxies).

klea_utils.models_catalog.DISK_CACHE_FILE = 'models-dev.json'

File name of the on-disk catalog cache, under the platformdirs cache dir.

klea_utils.models_catalog.DISK_CACHE_TTL_SECONDS = 86400

Time-to-live for the on-disk catalog cache, in seconds (1 day).

klea_utils.models_catalog.ENDPOINT_LIMITS_CACHE_TTL_SECONDS = 43200

Time-to-live for the in-memory live-endpoint model-limits cache, in seconds. A model’s max_model_len is a stable property of its deployment and rarely changes, so we cache it for the process lifetime; this TTL is just a safety valve against very long-running servers.

klea_utils.models_catalog.ENDPOINT_LIMITS_FETCH_TIMEOUT_SECONDS = 5.0

it is best-effort and must never stall a query.

Type:

HTTP timeout for the live-endpoint /models probe. Short

klea_utils.models_catalog.FETCH_TIMEOUT_SECONDS = 15.0

this is only hit on the first use after the disk cache expires.

Type:

HTTP timeout for fetching the catalog. Short

class klea_utils.models_catalog.ModelLimits(context: int | None = None, input: int | None = None, output: int | None = None)[source]

Bases: NamedTuple

Token limits for a single model from the catalog.

All fields are optional: the catalog always carries context and output, while input is only defined for a subset of models.

context: int | None

Alias for field number 0

input: int | None

Alias for field number 1

output: int | None

Alias for field number 2

klea_utils.models_catalog.get_catalog_model_limits(provider: str, model_name: str) → ModelLimits | None[source]

Return the token limits for a provider + model, or None.

Returns None when the provider has no catalog entry (local ollama, custom endpoints), the model is missing from the catalog, or the catalog could not be fetched. Callers should treat None as “no information available”.

Parameters:

  • provider – Klea provider id (e.g. "huggingface").

  • model_name – Model identifier, e.g. "gpt-4o" or "Qwen/Qwen3-Coder-30B-A3B-Instruct".

Returns:

ModelLimits with whatever fields the catalog defines.

klea_utils.models_catalog.probe_endpoint_model_limits(provider: str, model_name: str, base_url: str | None, api_key: str | None = None) → ModelLimits | None[source]

Return token limits for a model served by a live OpenAI-compatible endpoint.

models.dev’s limit values are per-deployment configuration, not a property of the model, and private/custom endpoints (e.g. an internal vLLM server) are not in the catalog at all. For those we ask the endpoint directly: GET {base_url}/models returns each model’s max_model_len (the total context window, e.g. 262144 for a large vLLM deployment), which is the authoritative per-deployment value.

Any provider with a known base_url is probed – not just openai custom endpoints. Native providers (mistral:, anthropic:, deepseek:, …) normally carry no base_url in the configurable dict (their SDK resolves a default endpoint internally); callers resolve it via klea_utils.llm.resolve_langchain_endpoint and pass it here. The probe is best-effort: providers whose /models response does not advertise max_model_len (e.g. Anthropic’s non-OpenAI-shaped payload) simply return None and the catalog fallback is used. Results are cached in memory keyed by (base_url, model_name) for ENDPOINT_LIMITS_CACHE_TTL_SECONDS; a user switching to a different model simply misses and triggers a fresh probe for the new key.

Never raises and never blocks a query: any error, missing model, or inapplicable input returns None.

Parameters:

  • provider – Klea provider id (openai for custom endpoints).

  • model_name – Model identifier as served by the endpoint.

  • base_url – Base URL of the OpenAI-compatible endpoint, or None.

  • api_key – Optional bearer token for the endpoint.

Returns:

ModelLimits with context set from max_model_len, or None when the endpoint does not expose it or is not usable.