fix(ai): BUG-044 capacités des modèles lues chez le fournisseur (Mistral vision)
Le panneau de modèle par défaut et la bulle ⓘ n'affichaient aucun modèle Mistral « Vision capable » alors que GET api.mistral.ai/v1/models en déclare 28 : la table de capacités était entièrement statique et aucun de ses motifs ne correspondait aux familles Mistral actuelles (seul `pixtral`, retiré de l'API, les matchait). - backend/provider_capabilities.py (nouveau) : capacités déclarées par le fournisseur (Mistral `capabilities`, OpenRouter `architecture`), détectées par la forme du payload, snapshot en cache process-wide (TTL 30 min, surchargeable par AI_CAPABILITIES_TTL_SECONDS) rempli par GET /api/config/ai-models. - backend/model_capabilities.py : une déclaration prime sur la table statique pour chaque drapeau mentionné ; la table ne comble que le reste (Mistral ne déclare jamais `embedding`). Table corrigée pour le repli hors ligne : familles vision Mistral (ministral, magistral, mistral-small, mistral-medium, mistral-vibe-cli, labs-leanstral), mistral-ocr = vision sans chat, et défaut du fournisseur Mistral sans `embeddings` (mistral-large / codestral n'étaient plus des « embedders »). - backend/ai_routes.py : GET /api/ai/model-capabilities reste sans appel réseau (cache froid → table statique). - Tests : tests/test_provider_capabilities.py (nouveau), TestMistralFamilies et TestDeclaredCapabilities (bout en bout via l'API). - Docs : CHANGELOG [Unreleased], registre + journal ISSUES_TODOLIST, fiche docs/features/ai-provider-picker.md (§L). Vérifié : 28/28 modèles vision déclarés par Mistral détectés (0 avant), 0 écart dans les deux sens ; pytest 1007 passed / 6 skipped ; ruff 0 (backend) ; mypy 0 ; tests frontend unit 9/9 + IA 66/66 + validate-imports 37 modules ; instance de test reconstruite et vérifiée sur http://localhost:2020.
This commit is contained in:
@@ -77,6 +77,24 @@ et [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
configuré, donc le **premier** fournisseur ajouté s'y monte aussi. Une
|
||||
sélection dont le fournisseur n'est plus configuré est purgée de
|
||||
`obsigate_ai_picker` (retour au défaut au lieu d'un modèle fantôme).
|
||||
- **BUG-044 — Capacités des modèles erronées (aucun modèle Mistral vision)** : la
|
||||
bulle ⓘ et le sélecteur de modèle par défaut n'affichaient **aucun** modèle Mistral
|
||||
« Vision capable », alors que `GET https://api.mistral.ai/v1/models` en déclare 28
|
||||
(`mistral-medium`, `mistral-small`, `ministral-*`, `magistral-*`, `mistral-ocr-*`,
|
||||
`mistral-vibe-cli-*`). Cause : la table de capacités était **entièrement statique** et
|
||||
aucun de ses motifs ne correspondait aux familles Mistral actuelles (seul `pixtral`,
|
||||
retiré de l'API, les matchait). ObsiGate lit désormais les capacités **déclarées par le
|
||||
fournisseur** (nouveau `backend/provider_capabilities.py`, snapshot mis en cache par
|
||||
`GET /api/config/ai-models`) : Mistral (`capabilities.completion_chat` / `vision` /
|
||||
`audio_transcription` / `audio_speech`) et OpenRouter (`architecture.input_modalities` /
|
||||
`output_modalities`) sont pris en charge ; la table statique ne sert plus qu'à combler
|
||||
les drapeaux non déclarés et de repli hors ligne. La table est également corrigée
|
||||
(familles vision Mistral, `mistral-ocr` = vision sans chat) et le défaut du fournisseur
|
||||
Mistral ne prétend plus qu'un modèle non reconnu sait produire des embeddings
|
||||
(BUG-044bis : `mistral-large-latest` / `codestral-latest` étaient annoncés
|
||||
« Embeddings »). Conséquence : la porte vision de l'assistant
|
||||
(`frontend/js/bookslm.js`, `backend/bookslm_routes.py`) accepte enfin les images avec un
|
||||
modèle Mistral vision.
|
||||
|
||||
### Sécurité
|
||||
|
||||
|
||||
@@ -253,10 +253,14 @@ async def api_model_capabilities(
|
||||
model: str = Query("", description="Model identifier (optional)"),
|
||||
current_user=Depends(require_auth),
|
||||
):
|
||||
"""Return the curated capability flags for a provider/model pair.
|
||||
"""Return the capability flags for a provider/model pair.
|
||||
|
||||
The table is static (see ``backend.model_capabilities``) so the answer is
|
||||
available offline and never depends on a provider API call.
|
||||
Two layers (BUG-044): flags the provider itself declares in its models
|
||||
endpoint (Mistral ``capabilities``, OpenRouter ``architecture``) win, the
|
||||
curated table in ``backend.model_capabilities`` fills the rest. The
|
||||
declaration snapshot is populated by ``GET /api/config/ai-models``; when it
|
||||
is cold (or the provider declares nothing) the curated table answers alone,
|
||||
so this endpoint never performs a blocking provider call.
|
||||
"""
|
||||
return {
|
||||
"provider": provider,
|
||||
|
||||
@@ -3738,6 +3738,7 @@ async def api_list_ai_models(provider: str = Query(...), current_user=Depends(re
|
||||
provider = provider.lower()
|
||||
|
||||
from backend.model_capabilities import get_capabilities_for_models
|
||||
from backend.provider_capabilities import remember_declared_capabilities
|
||||
|
||||
all_providers = ("deepseek", "openrouter", "gemini", "nvidia", "qwencloud", "xiaomi", "mistral")
|
||||
if provider not in all_providers:
|
||||
@@ -3791,6 +3792,12 @@ async def api_list_ai_models(provider: str = Query(...), current_user=Depends(re
|
||||
else:
|
||||
models = [m.get("id", "") for m in data.get("data", []) if m.get("id")]
|
||||
|
||||
# Cache the capabilities the provider declares for these models
|
||||
# (BUG-044) — get_capabilities_for_models() below then returns the
|
||||
# provider's own truth for the flags it declares, the curated table
|
||||
# for the rest. Providers that declare nothing are left untouched.
|
||||
remember_declared_capabilities(provider, data)
|
||||
|
||||
if models:
|
||||
# Prepend the configured default if not already present
|
||||
default = PROVIDERS.get(provider, {}).get("model")
|
||||
|
||||
@@ -1,21 +1,34 @@
|
||||
"""Curated model-capability metadata for the AI assistant.
|
||||
"""Model-capability metadata for the AI assistant.
|
||||
|
||||
ObsiGate does not query every provider for the modalities a model supports
|
||||
(not all of them expose that information, and the network call is not always
|
||||
reliable). Instead a static, curated table maps known model-name patterns to
|
||||
capability flags, with per-provider defaults. The UI uses this to show, when a
|
||||
model is selected, which features it supports (Chat, Embeddings, Rerank,
|
||||
Images, Video, Audio Speech, Audio Transcriptions, Vision).
|
||||
Two layers, in order of trust:
|
||||
|
||||
The table is intentionally conservative: an unknown model falls back to the
|
||||
provider default (usually ``chat`` only), so we never claim a capability the
|
||||
model may not have.
|
||||
1. **Provider-declared** (:mod:`backend.provider_capabilities`) — when the
|
||||
provider publishes per-model capabilities in its models endpoint (Mistral
|
||||
``capabilities``, OpenRouter ``architecture``), that declaration *wins* for
|
||||
every flag it mentions. The snapshot is cached by the model-list endpoint.
|
||||
2. **Curated table** (this module) — a static, hand-maintained map of known
|
||||
model-name patterns to capability flags with per-provider defaults. It fills
|
||||
the flags the provider stays silent about, and is the only source for
|
||||
providers and models that declare nothing (offline, no API key, DeepSeek,
|
||||
NVIDIA, QwenCloud, Xiaomi…).
|
||||
|
||||
The UI uses the result to show, when a model is selected, which features it
|
||||
supports (Chat, Embeddings, Rerank, Images, Video, Audio Speech, Audio
|
||||
Transcriptions, Vision).
|
||||
|
||||
The curated layer is intentionally conservative: an unknown model falls back to
|
||||
the provider default (usually ``chat`` only), so we never claim a capability the
|
||||
model may not have. A provider declaration, on the other hand, is authoritative
|
||||
in both directions — it can also *revoke* a flag the curated table guessed
|
||||
wrongly (e.g. ``mistral-embed`` declares no chat).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import Any
|
||||
|
||||
from backend.provider_capabilities import get_declared_capabilities
|
||||
|
||||
# Ordered list of capability keys exposed to the UI. Keep in sync with the
|
||||
# frontend ``AI_CAPABILITY_KEYS`` and the i18n ``ai.cap_*`` labels.
|
||||
CAPABILITY_KEYS: tuple[str, ...] = (
|
||||
@@ -46,7 +59,10 @@ _PROVIDER_DEFAULTS: dict[str, dict[str, bool]] = {
|
||||
"nvidia": _caps(chat=True),
|
||||
"qwencloud": _caps(chat=True),
|
||||
"xiaomi": _caps(chat=True),
|
||||
"mistral": _caps(chat=True, embeddings=True),
|
||||
# Mistral: chat only. ``embeddings`` used to be assumed for every Mistral
|
||||
# model, which wrongly labelled mistral-large / codestral as embedders
|
||||
# (BUG-044); the ``embed`` rule below covers the real embedding models.
|
||||
"mistral": _caps(chat=True),
|
||||
}
|
||||
|
||||
# Ordered (substrings, capabilities) rules — the first matching rule wins.
|
||||
@@ -76,6 +92,8 @@ _MODEL_RULES: list[tuple[tuple[str, ...], dict[str, bool]]] = [
|
||||
),
|
||||
# Video generation.
|
||||
(("veo-", "sora", "video-gen", "-video"), _caps(video=True)),
|
||||
# Document OCR models — image input, but not a chat endpoint.
|
||||
(("mistral-ocr",), _caps(vision=True)),
|
||||
# Vision-capable chat models (multimodal input).
|
||||
(
|
||||
(
|
||||
@@ -97,15 +115,36 @@ _MODEL_RULES: list[tuple[tuple[str, ...], dict[str, bool]]] = [
|
||||
"minicpm-v",
|
||||
"llama-3.2-vision",
|
||||
"mimo-vl",
|
||||
# Mistral vision families (BUG-044): text-only mistral-large,
|
||||
# codestral and voxtral are deliberately absent.
|
||||
"ministral",
|
||||
"magistral",
|
||||
"mistral-small",
|
||||
"mistral-medium",
|
||||
"mistral-vibe-cli",
|
||||
"labs-leanstral",
|
||||
),
|
||||
_caps(chat=True, vision=True),
|
||||
),
|
||||
]
|
||||
|
||||
|
||||
def _curated_capabilities(provider: str, model: str) -> dict[str, bool]:
|
||||
"""Curated table lookup (model rules first, then the provider default)."""
|
||||
if model:
|
||||
for needles, caps in _MODEL_RULES:
|
||||
if any(needle in model for needle in needles):
|
||||
return dict(caps)
|
||||
return dict(_PROVIDER_DEFAULTS.get(provider, _caps(chat=True)))
|
||||
|
||||
|
||||
def get_model_capabilities(provider: str, model: str) -> dict[str, bool]:
|
||||
"""Return the capability flags for a ``provider``/``model`` pair.
|
||||
|
||||
A provider declaration (see :mod:`backend.provider_capabilities`) overrides
|
||||
the curated table for every flag it mentions; the curated table supplies the
|
||||
rest.
|
||||
|
||||
Args:
|
||||
provider: Provider identifier (e.g. ``"deepseek"``). Case-insensitive.
|
||||
model: Model identifier (e.g. ``"deepseek-chat"``). May be empty, in
|
||||
@@ -116,11 +155,11 @@ def get_model_capabilities(provider: str, model: str) -> dict[str, bool]:
|
||||
"""
|
||||
provider = (provider or "").strip().lower()
|
||||
name = (model or "").strip().lower()
|
||||
if name:
|
||||
for needles, caps in _MODEL_RULES:
|
||||
if any(needle in name for needle in needles):
|
||||
return dict(caps)
|
||||
return dict(_PROVIDER_DEFAULTS.get(provider, _caps(chat=True)))
|
||||
caps = _curated_capabilities(provider, name)
|
||||
declared = get_declared_capabilities(provider, name)
|
||||
if declared:
|
||||
caps.update({key: value for key, value in declared.items() if key in CAPABILITY_KEYS})
|
||||
return caps
|
||||
|
||||
|
||||
def get_capabilities_for_models(
|
||||
|
||||
@@ -0,0 +1,174 @@
|
||||
"""Provider-declared model capabilities (live) with a short-lived cache.
|
||||
|
||||
The curated table in :mod:`backend.model_capabilities` has to be edited by hand
|
||||
every time a provider ships or renames a model, and it ages badly: Mistral alone
|
||||
declares ``vision`` on 28 of its models while the curated table knew none of
|
||||
them (BUG-044). Providers that *do* publish per-model capabilities in their
|
||||
public models endpoint are therefore asked first; the curated table is only
|
||||
used to fill the flags the provider stays silent about (e.g. Mistral never
|
||||
declares ``embedding``, only the absence of ``completion_chat``).
|
||||
|
||||
Supported declarations, detected by *payload shape* so a provider that starts
|
||||
exposing them is picked up without a code change:
|
||||
|
||||
* ``capabilities`` dict (Mistral): ``completion_chat`` → ``chat``, ``vision``,
|
||||
``audio_transcription`` (+ ``audio_transcription_realtime``),
|
||||
``audio_speech``.
|
||||
* ``architecture`` dict (OpenRouter): ``input_modalities`` /
|
||||
``output_modalities`` → ``vision`` (image input), ``images`` (image output),
|
||||
``audio_transcription`` (audio input), ``audio_speech`` (audio output),
|
||||
``video`` (video output), ``chat`` (text output).
|
||||
|
||||
The cache is in-process and shared by every request. Its TTL only bounds how
|
||||
long a *stale* declaration can survive: a fresh provider call (the model list
|
||||
endpoint) overwrites the provider entry immediately.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import os
|
||||
import time
|
||||
from typing import Any
|
||||
|
||||
logger = logging.getLogger("obsigate.ai.capabilities")
|
||||
|
||||
#: How long a provider-declared snapshot stays usable, in seconds.
|
||||
#: ``0`` disables expiry (the snapshot lives until the next provider call).
|
||||
TTL_SECONDS = float(os.getenv("AI_CAPABILITIES_TTL_SECONDS", "1800") or 0)
|
||||
|
||||
#: Guard against a provider returning a runaway model list.
|
||||
MAX_MODELS_PER_PROVIDER = 2000
|
||||
|
||||
# provider → (timestamp, model count, {normalized model id: declared flags})
|
||||
_CACHE: dict[str, tuple[float, int, dict[str, dict[str, bool]]]] = {}
|
||||
|
||||
|
||||
def _norm(value: str) -> str:
|
||||
"""Lower-case and strip the ``models/`` prefix Gemini uses."""
|
||||
return (value or "").strip().lower().removeprefix("models/")
|
||||
|
||||
|
||||
def _from_capability_flags(flags: dict[str, Any]) -> dict[str, bool]:
|
||||
"""Map a Mistral-style ``capabilities`` dict onto ObsiGate flags."""
|
||||
declared: dict[str, bool] = {}
|
||||
if isinstance(flags.get("completion_chat"), bool):
|
||||
declared["chat"] = flags["completion_chat"]
|
||||
if isinstance(flags.get("vision"), bool):
|
||||
declared["vision"] = flags["vision"]
|
||||
transcription = flags.get("audio_transcription")
|
||||
realtime = flags.get("audio_transcription_realtime")
|
||||
if isinstance(transcription, bool) or isinstance(realtime, bool):
|
||||
declared["audio_transcription"] = bool(transcription or realtime)
|
||||
if isinstance(flags.get("audio_speech"), bool):
|
||||
declared["audio_speech"] = flags["audio_speech"]
|
||||
return declared
|
||||
|
||||
|
||||
def _from_architecture(architecture: dict[str, Any]) -> dict[str, bool]:
|
||||
"""Map an OpenRouter-style ``architecture`` dict onto ObsiGate flags."""
|
||||
inputs = architecture.get("input_modalities")
|
||||
outputs = architecture.get("output_modalities")
|
||||
if not isinstance(inputs, list) and not isinstance(outputs, list):
|
||||
return {}
|
||||
in_modalities = [str(m).lower() for m in inputs] if isinstance(inputs, list) else []
|
||||
out_modalities = [str(m).lower() for m in outputs] if isinstance(outputs, list) else []
|
||||
return {
|
||||
"chat": "text" in out_modalities,
|
||||
"vision": "image" in in_modalities,
|
||||
"images": "image" in out_modalities,
|
||||
"audio_transcription": "audio" in in_modalities,
|
||||
"audio_speech": "audio" in out_modalities,
|
||||
"video": "video" in out_modalities,
|
||||
}
|
||||
|
||||
|
||||
def parse_declared_capabilities(entry: Any) -> dict[str, bool] | None:
|
||||
"""Extract the capability flags a single provider model entry declares.
|
||||
|
||||
Args:
|
||||
entry: One item of a provider models payload (``/v1/models``).
|
||||
|
||||
Returns:
|
||||
A partial ``{flag: bool}`` mapping (only the flags the provider
|
||||
actually declares), or ``None`` when the entry declares nothing.
|
||||
"""
|
||||
if not isinstance(entry, dict):
|
||||
return None
|
||||
flags = entry.get("capabilities")
|
||||
declared = _from_capability_flags(flags) if isinstance(flags, dict) else {}
|
||||
if not declared:
|
||||
architecture = entry.get("architecture")
|
||||
declared = _from_architecture(architecture) if isinstance(architecture, dict) else {}
|
||||
return declared or None
|
||||
|
||||
|
||||
def remember_declared_capabilities(provider: str, payload: Any) -> int:
|
||||
"""Cache the capabilities declared by a provider models payload.
|
||||
|
||||
Args:
|
||||
provider: Provider identifier (e.g. ``"mistral"``).
|
||||
payload: Raw JSON body of the provider models endpoint, or the model
|
||||
list itself.
|
||||
|
||||
Returns:
|
||||
The number of models with declared capabilities that were cached.
|
||||
"""
|
||||
provider = (provider or "").strip().lower()
|
||||
entries: Any = payload.get("data") if isinstance(payload, dict) else payload
|
||||
if not isinstance(entries, list):
|
||||
return 0
|
||||
|
||||
parsed: dict[str, dict[str, bool]] = {}
|
||||
for entry in entries[:MAX_MODELS_PER_PROVIDER]:
|
||||
if not isinstance(entry, dict):
|
||||
continue
|
||||
model_id = entry.get("id") or entry.get("name") or ""
|
||||
if not isinstance(model_id, str) or not model_id.strip():
|
||||
continue
|
||||
declared = parse_declared_capabilities(entry)
|
||||
if declared:
|
||||
parsed[_norm(model_id)] = declared
|
||||
|
||||
if not parsed:
|
||||
return 0
|
||||
_CACHE[provider] = (time.time(), len(parsed), parsed)
|
||||
logger.info(f"Capabilities declared by {provider}: {len(parsed)} models cached")
|
||||
return len(parsed)
|
||||
|
||||
|
||||
def get_declared_capabilities(provider: str, model: str) -> dict[str, bool] | None:
|
||||
"""Return the cached declared capabilities for one provider/model pair.
|
||||
|
||||
Returns ``None`` when nothing was declared for that pair (cache cold,
|
||||
expired, or the provider is silent about this model).
|
||||
"""
|
||||
snapshot = _CACHE.get((provider or "").strip().lower())
|
||||
if not snapshot:
|
||||
return None
|
||||
timestamp, _count, table = snapshot
|
||||
if TTL_SECONDS and (time.time() - timestamp) > TTL_SECONDS:
|
||||
return None
|
||||
declared = table.get(_norm(model))
|
||||
return dict(declared) if declared else None
|
||||
|
||||
|
||||
def clear_declared_capabilities(provider: str | None = None) -> None:
|
||||
"""Drop the cached snapshot for one provider, or all of them (tests/ops)."""
|
||||
if provider is None:
|
||||
_CACHE.clear()
|
||||
return
|
||||
_CACHE.pop(provider.strip().lower(), None)
|
||||
|
||||
|
||||
def cache_info() -> dict[str, dict[str, Any]]:
|
||||
"""Diagnostics: per-provider cache age and model count."""
|
||||
now = time.time()
|
||||
return {
|
||||
provider: {
|
||||
"models": count,
|
||||
"age_seconds": round(now - timestamp, 1),
|
||||
"expired": bool(TTL_SECONDS and (now - timestamp) > TTL_SECONDS),
|
||||
}
|
||||
for provider, (timestamp, count, _table) in _CACHE.items()
|
||||
}
|
||||
@@ -14,7 +14,7 @@
|
||||
|
||||
- **Projet** : ObsiGate — Porte d'entrée web pour vaults Obsidian
|
||||
- **Stack** : Python 3.11+ (backend FastAPI) · JavaScript/Vanilla (frontend) · Tauri/Rust (desktop)
|
||||
- **Dernière mise à jour** : 2026-09-14
|
||||
- **Dernière mise à jour** : 2026-09-15
|
||||
|
||||
---
|
||||
|
||||
@@ -153,6 +153,8 @@ Avant de corriger quoi que ce soit, un agent IA doit :
|
||||
| *BUG-041* | [🟡 IMPORTANT] Assistant IA : échec sur un répertoire vide (« Aucun fichier markdown trouvé dans ce dossier ») au lieu de répondre | 🟢 corrigé | P1 | 📱 frontend + ⚙️ backend | IA | `backend/bookslm_routes.py`, `backend/bookslm.py`, `frontend/js/bookslm.js` | Ouvrir l'assistant sur un dossier vide puis envoyer une question | `_resolve_system_prompt` dégrade vers le prompt Général + bloc « Dossier vide » (plus de 404) ; contexte applicatif `app_context` enrichi (documents ouverts, répertoire, recherche, fichiers récents) | Le 404 bloquait toute la requête. Feature #88, fiche `docs/features/ai-app-context.md`. Tests : `tests/test_bookslm.py` (+3), `tests/frontend/ai.test.mjs` |
|
||||
| *BUG-042* | [🟡 IMPORTANT] Assistant IA : liens de fichiers non fiables (« File not found: ») — pas de règle déterministe nom / dossier / chemin | 🟢 corrigé | P1 | 📱 frontend | IA | `frontend/js/bookslm.js` | Cliquer les liens de fichiers/dossiers dans une réponse de l'assistant (noms avec espaces et/ou accents, chemin préfixé par le nom du vault) | `_classifyPath` distingue `name` (copie presse-papiers) / `dir` (révélation arborescence) / `file` (ouverture) ; `_activatePath()` résout le chemin contre l'index du vault (exact → suffixe → basename unique) avant d'agir ; espaces + accents pris en charge (classes Unicode `\p{L}\p{N}\p{M}`, comparaison normalisée NFC, markdown `<…>`/`%20`, code inline, mentions brutes confirmées par l'index) ; `_splitVaultPrefix` retire un préfixe `Vault/…` et ouvre dans ce vault (`_fetchPathsForVault`) | Les liens morts ouvraient un fichier inexistant. Feature #88. Tests : `tests/frontend/ai.test.mjs` (+11) |
|
||||
| *BUG-043* | [🟡 IMPORTANT] Assistant IA : la liste des fournisseurs de la barre latérale ne suit pas les ajouts/retraits de clés API dans la configuration du projet | 🟢 corrigé | P1 | 📱 frontend | IA | `frontend/js/ai.js`, `frontend/js/bookslm.js`, `frontend/js/config.js` | Ajouter (ou supprimer) une clé de fournisseur AI dans la configuration puis observer le menu Fournisseur de l'assistant sans recharger la page | Le picker lit `/api/ai/status` **une seule fois**, à sa construction, et le panneau de l'assistant est un singleton monté pour toute la session → liste figée. Nouveau `refreshAIPickers()` (exporté par `ai.js`) qui reconstruit chaque picker monté dans son emplacement `.ai-picker-slot` (conservé même sans fournisseur configuré, donc un premier fournisseur s'y monte aussi) ; appelé après `saveAIKeys()` et `deleteAIKey()` (`config.js`) ; une sélection dont le fournisseur n'est plus configuré est purgée de `obsigate_ai_picker` (retour au défaut + modèle effacé au lieu d'un nom fantôme) | Il fallait recharger la page pour voir un nouveau fournisseur (ou en voir disparaître un). Feature #82. Tests : `tests/frontend/ai.test.mjs` (+4) |
|
||||
| *BUG-044* | [🟡 IMPORTANT] Capacités des modèles IA erronées : aucun modèle Mistral n'est détecté « Vision capable » (et des modèles texte sont annoncés « Embeddings ») | 🟢 corrigé | P1 | ⚙️ backend + 🔌 api | IA | `backend/model_capabilities.py`, `backend/provider_capabilities.py`, `backend/main.py`, `backend/ai_routes.py` | `curl -H "Authorization: Bearer $MISTRAL_API_KEY" https://api.mistral.ai/v1/models` puis `GET /api/ai/model-capabilities?provider=mistral&model=mistral-small-latest` | Nouveau calque **déclaratif** (`backend/provider_capabilities.py`) : les capacités publiées par le fournisseur (Mistral `capabilities`, OpenRouter `architecture`) sont lues et mises en cache par `GET /api/config/ai-models` ; elles **priment** sur la table statique, qui ne comble plus que les drapeaux non déclarés (et sert de repli hors ligne / sans clé). Table statique corrigée : familles vision Mistral (`ministral`, `magistral`, `mistral-small`, `mistral-medium`, `mistral-vibe-cli`, `labs-leanstral`), `mistral-ocr` = vision sans chat, défaut fournisseur Mistral sans `embeddings`. Tests : `tests/test_provider_capabilities.py`, `tests/test_model_capabilities.py::TestMistralFamilies`, `tests/test_ai_models.py::TestDeclaredCapabilities` | Le panneau de modèle par défaut et la bulle ⓘ n'affichaient **aucun** Mistral vision alors que l'API en déclare 28 ; seul le motif `pixtral` (retiré de l'API) matchait. Effet secondaire corrigé : `mistral-large-latest` / `codestral-latest` étaient annoncés « Embeddings » (défaut fournisseur). La porte vision de l'assistant (`bookslm_routes.py`) accepte désormais les images avec un modèle Mistral vision |
|
||||
| | | | | | | | | | | |
|
||||
|
||||
### TODOs techniques (améliorations / nouvelles tâches)
|
||||
|
||||
@@ -192,7 +194,8 @@ Avant de corriger quoi que ce soit, un agent IA doit :
|
||||
| 2026-09-14 | BUG-042 (complément) | Correction | `frontend/js/bookslm.js`, `tests/frontend/ai.test.mjs`, `docs/features/ai-app-context.md` | Prise en charge des **espaces** dans les noms de fichiers et chemins des liens de l'assistant : cibles markdown avec espaces / `<…>` / `%20` décodés, code inline reconnu (`_looksLikePath` élargi, rejette toujours les extraits de code), mentions brutes liées uniquement si présentes dans l'index du vault (`_linkifySpacePaths` + `_confirmPathInCache`, plus long suffixe aligné sur un mot pour ne pas avaler le mot de prose précédent). Vérifié : tests frontend 57/57 (IA) + 9 suites JSDOM, validate-imports 36 modules, pytest 963 passed / 6 skipped. | 🟢 corrigé (en attente vérif utilisateur) |
|
||||
| 2026-09-14 | BUG-041, BUG-042, #88 | Correction + feature | `frontend/js/bookslm.js`, `backend/bookslm.py`, `backend/bookslm_routes.py`, `frontend/locales/{fr,en}.json`, `tests/test_bookslm.py`, `tests/frontend/ai.test.mjs`, `docs/features/ai-app-context.md` | BUG-041 : contexte de dossier vide → dégradation gracieuse vers le prompt Général + bloc « Dossier vide » (fin du 404). BUG-042 : liens de fichiers déterministes (nom → presse-papiers, dossier → arborescence, chemin → viewer) + résolution du chemin contre l'index du vault avant ouverture. #88 : `app_context` (documents ouverts, répertoire/vault courants, recherche + résultats affichés) et fichiers récemment modifiés injectés dans le prompt Général. Vérifié : pytest 963 passed / 6 skipped, ruff 0, mypy 0, tests frontend 52/52 + validate-imports 36 modules. | 🟢 corrigé (en attente vérif utilisateur) |
|
||||
| 2026-09-14 | BUG-042 (accents + préfixe vault) | Correction | `frontend/js/bookslm.js`, `tests/frontend/ai.test.mjs`, `docs/features/ai-app-context.md` | **Caractères accentués** : classes de caractères Unicode (`\p{L}\p{N}\p{M}`) pour `PATH_WITH_DIR_RE` et `PATH_NAME_RE` (`_looksLikePath`), comparaison normalisée `NFC` via `_normKey()` → une mention décomposée (`e` + accent combinant, style macOS) correspond à une entrée d'index précomposée, et les liens sans espace mais accentués sont enfin produits. **Préfixe de vault** : `_splitVaultPrefix()` retire un premier segment égal à un vault connu (`TestVault/Recettes/Pizza Maison.md`) ; `_fetchPathsForVault()` interroge l'index de ce vault sans écraser le cache du vault actif ; `_openFileLink`/`_revealPath` reçoivent le vault cible ; repli « retirer le premier segment » si le préfixe est inconnu. Vérifié : tests frontend 62/62 (IA) + 9 suites JSDOM, validate-imports 36 modules, pytest 963 passed / 6 skipped, et vérification contre l'instance live (données réelles accentuées/espacées : `Recettes/Préparation.md`, `Recettes/Pâtes carbonara.md`, `Recettes/Pizza Maison.md`) — 8/8 contrôles. | 🟢 corrigé (en attente vérif utilisateur) |
|
||||
| 2026-09-14 | BUG-043, #82 | Correction | `frontend/js/ai.js`, `frontend/js/bookslm.js`, `frontend/js/config.js`, `tests/frontend/ai.test.mjs`, `docs/features/ai-provider-picker.md` | La liste des fournisseurs de la barre latérale de l'assistant suit désormais la configuration du projet : nouveau `refreshAIPickers()` (`ai.js`) qui reconstruit chaque picker monté dans son emplacement `.ai-picker-slot` (host conservé dans la barre de l'assistant, montage par `replaceChildren`), appelé après `saveAIKeys()` et `deleteAIKey()` (`config.js`) ; slot préservé même sans fournisseur configuré (le premier fournisseur ajouté s'y monte) ; sélection persistée d'un fournisseur non configuré purgée de `obsigate_ai_picker` (retour au défaut, modèle effacé). Vérifié : tests frontend 66/66 (IA) + 9 suites JSDOM vertes, validate-imports 36 modules, pytest 963 passed / 6 skipped, ruff OK, et vérification en navigateur (Playwright) sur l'instance de test : ajout de `nvidia` → fournisseur visible sans rechargement, retrait → disparition + sélection réinitialisée (`{"provider":null,"model":null}`), un seul montage du picker dans son host. CI Gitea verte (lint, test, security, build, e2e). | 🟢 corrigé (en attente vérif utilisateur) |
|
||||
| 2026-09-14 | BUG-043, #82 | Correction | `frontend/js/ai.js`, `frontend/js/bookslm.js`, `frontend/js/config.js`, `tests/frontend/ai.test.mjs`, `docs/features/ai-provider-picker.md` | La liste des fournisseurs de la barre latérale de l'assistant suit désormais la configuration du projet : nouveau `refreshAIPickers()` (`ai.js`) qui reconstruit chaque picker monté dans son emplacement `.ai-picker-slot` (host conservé dans la barre de l'assistant, montage par `replaceChildren`), appelé après `saveAIKeys()` et `deleteAIKey()` (`config.js`) ; slot préservé même sans fournisseur configuré (le premier fournisseur ajouté s'y monte) ; sélection persistée d'un fournisseur non configuré purgée de `obsigate_ai_picker` (retour au défaut, modèle effacé). Vérifié : tests frontend 66/66 (IA) + 9 suites JSDOM vertes, validate-imports 36 modules, pytest 963 passed / 6 skipped, ruff OK, et vérification en navigateur (Playwright) sur l'instance de test : ajout de `nvidia` → fournisseur visible sans rechargement, retrait → disparition + sélection réinitialisée (`{"provider":null,"model":null}`), un seul montage du picker dans son host. CI Gitea verte (lint, test, security, build, e2e). |
|
||||
| 2026-09-15 | BUG-044 | Correction | `backend/provider_capabilities.py` (nouveau), `backend/model_capabilities.py`, `backend/main.py`, `backend/ai_routes.py`, `tests/test_provider_capabilities.py` (nouveau), `tests/test_model_capabilities.py`, `tests/test_ai_models.py`, `docs/features/ai-provider-picker.md`, `CHANGELOG.md` | BUG-044 : les capacités des modèles IA sont désormais lues chez le fournisseur quand il les publie (Mistral `capabilities`, OpenRouter `architecture`, détection par forme du payload), mises en cache par `GET /api/config/ai-models` (aucune requête supplémentaire) et prioritaires sur la table statique qui ne comble plus que les drapeaux non déclarés (repli hors ligne / sans clé). Table statique corrigée : familles vision Mistral (`ministral`, `magistral`, `mistral-small`, `mistral-medium`, `mistral-vibe-cli`, `labs-leanstral`), `mistral-ocr` = vision sans chat, défaut fournisseur Mistral sans `embeddings` (mistral-large / codestral n'étaient plus des « embedders »). Vérifié : diagnostic live avant/après sur l'API Mistral (28 modèles vision déclarés, **0** détectés avant → **28** après, 0 écart dans les deux sens), pytest 1007 passed / 6 skipped, ruff 0 (backend), mypy 0 (68 fichiers), tests frontend (unit 9/9, IA 66/66, validate-imports 37 modules), et vérification sur l'instance de test reconstruite (`obsigate-test`, http://localhost:2020) via les deux endpoints du panneau : `mistral-small/medium-latest`, `ministral-8b-latest`, `magistral-small-latest` → `chat,vision` ; `mistral-ocr-latest` → `vision` seul ; `mistral-large-latest`/`codestral-latest` → `chat` ; `mistral-embed` → `embeddings` seul ; `deepseek-chat` (fournisseur muet) inchangé. | 🟢 corrigé (en attente vérif utilisateur) |
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -113,7 +113,34 @@
|
||||
et garde-fou sur le câblage `saveAIKeys`/`deleteAIKey` (`tests/frontend/ai.test.mjs`).
|
||||
|
||||
## I. Points d'attention
|
||||
- La table de capacités reste **statique** (`backend/model_capabilities.py`) : un modèle inconnu
|
||||
retombe sur le défaut du fournisseur.
|
||||
- La table statique (`backend/model_capabilities.py`) n'est plus la source principale : elle sert
|
||||
de **repli** (fournisseur muet, pas de clé API, cache froid) et comble les drapeaux non déclarés.
|
||||
Un modèle inconnu retombe toujours sur le défaut du fournisseur.
|
||||
- Le cache de déclarations est **en mémoire, par process** : un redémarrage ou un cache froid ne
|
||||
casse rien (repli statique), et `GET /api/config/ai-models` le repeuple au premier affichage de
|
||||
la liste des modèles.
|
||||
- Le select masqué est conservé uniquement pour la rétro-compatibilité ; ne pas le supprimer sans
|
||||
migrer `_cmdSwitchModel` / `_applyPickerSelection` dans `frontend/js/bookslm.js`.
|
||||
|
||||
## L. Capacités fournies par le provider (BUG-044) — ✅ livré
|
||||
- [x] `backend/provider_capabilities.py` (nouveau) : lecture des capacités **déclarées** par
|
||||
l'API du fournisseur (`capabilities` Mistral, `architecture` OpenRouter), détectées par la
|
||||
**forme** du payload — un fournisseur qui se met à les publier est pris en charge sans
|
||||
modification de code.
|
||||
- [x] Snapshot en cache process-wide (TTL 30 min par défaut, `AI_CAPABILITIES_TTL_SECONDS`),
|
||||
rempli par `GET /api/config/ai-models` (appel déjà effectué pour lister les modèles :
|
||||
aucune requête supplémentaire) ; `clear_declared_capabilities()` + `cache_info()` pour les
|
||||
tests et le diagnostic.
|
||||
- [x] `backend/model_capabilities.py` : une déclaration **prime** sur la table statique pour
|
||||
chaque drapeau qu'elle mentionne ; la table ne comble que le reste (Mistral ne déclare
|
||||
jamais `embedding`, seulement l'absence de `completion_chat`).
|
||||
- [x] Table statique corrigée pour le repli hors ligne : familles vision Mistral (`ministral`,
|
||||
`magistral`, `mistral-small`, `mistral-medium`, `mistral-vibe-cli`, `labs-leanstral`),
|
||||
`mistral-ocr` = vision sans chat, et défaut du fournisseur Mistral sans `embeddings`
|
||||
(un modèle inconnu n'est plus présenté comme un modèle d'embeddings).
|
||||
- [x] `GET /api/ai/model-capabilities` reste **sans appel réseau** : il répond depuis le cache
|
||||
(froid → table statique), donc la bulle ⓘ ne bloque jamais.
|
||||
- [x] Tests : `tests/test_provider_capabilities.py` (parsing, cache, TTL, fusion),
|
||||
`tests/test_model_capabilities.py::TestMistralFamilies` (repli hors ligne) et
|
||||
`tests/test_ai_models.py::TestDeclaredCapabilities` (bout en bout : payload Mistral simulé →
|
||||
`capabilities` de `/api/config/ai-models` puis `/api/ai/model-capabilities`).
|
||||
|
||||
@@ -14,6 +14,16 @@ from fastapi.testclient import TestClient
|
||||
|
||||
from backend.main import _FALLBACK_MODELS, app
|
||||
|
||||
#: Trimmed-down copy of what api.mistral.ai/v1/models really returns (BUG-044).
|
||||
MISTRAL_LIVE_PAYLOAD = {
|
||||
"object": "list",
|
||||
"data": [
|
||||
{"id": "mistral-small-latest", "capabilities": {"completion_chat": True, "vision": True}},
|
||||
{"id": "mistral-medium-latest", "capabilities": {"completion_chat": True, "vision": True}},
|
||||
{"id": "mistral-embed", "capabilities": {"completion_chat": False, "vision": False}},
|
||||
],
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def admin_client(tmp_path):
|
||||
@@ -83,6 +93,16 @@ def _login_admin(client):
|
||||
return resp.json()["access_token"]
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _no_declared_caps():
|
||||
"""Provider declarations are cached process-wide: isolate every test."""
|
||||
from backend.provider_capabilities import clear_declared_capabilities
|
||||
|
||||
clear_declared_capabilities()
|
||||
yield
|
||||
clear_declared_capabilities()
|
||||
|
||||
|
||||
# ── Unit tests for the fallback table itself ─────────────────────────────
|
||||
|
||||
|
||||
@@ -339,3 +359,112 @@ class TestListModelsEndpoint:
|
||||
assert xiaomi_hdrs["api-key"] == "fake-key"
|
||||
# Bearer prefix must NOT be in the api-key value
|
||||
assert "Bearer" not in xiaomi_hdrs["api-key"]
|
||||
|
||||
|
||||
class TestDeclaredCapabilities:
|
||||
"""BUG-044 — the provider's own ``capabilities[]`` must reach the UI.
|
||||
|
||||
``GET /api/config/ai-models`` is what fills the capability cache; the picker
|
||||
bubble (``GET /api/ai/model-capabilities``) and the vision gate then read it.
|
||||
Before the fix no Mistral model was ever reported as vision-capable.
|
||||
"""
|
||||
|
||||
def _serve_payload(self, monkeypatch, payload):
|
||||
"""Make the provider models call return ``payload`` verbatim."""
|
||||
import json
|
||||
import urllib.request as global_urllib_mod
|
||||
|
||||
body = json.dumps(payload).encode()
|
||||
|
||||
def fake_Request(url, *args, **kwargs):
|
||||
class FakeResp:
|
||||
def __enter__(self): return self
|
||||
def __exit__(self, *a): pass
|
||||
def read(self): return body
|
||||
|
||||
return FakeResp()
|
||||
|
||||
monkeypatch.setattr(global_urllib_mod, "Request", fake_Request, raising=True)
|
||||
monkeypatch.setattr(
|
||||
global_urllib_mod, "urlopen", lambda req, timeout=None: req, raising=True
|
||||
)
|
||||
|
||||
def _fake_key(self, monkeypatch):
|
||||
"""main.py binds get_ai_key at import: patch that binding too."""
|
||||
import backend.ai as aimod
|
||||
import backend.main as bmain
|
||||
|
||||
monkeypatch.setattr(aimod, "get_ai_key", lambda name: "fake-mistral-key")
|
||||
if hasattr(bmain, "get_ai_key"):
|
||||
monkeypatch.setattr(bmain, "get_ai_key", lambda name: "fake-mistral-key")
|
||||
|
||||
def test_declared_vision_reaches_both_endpoints(self, admin_client, monkeypatch):
|
||||
self._fake_key(monkeypatch)
|
||||
self._serve_payload(monkeypatch, MISTRAL_LIVE_PAYLOAD)
|
||||
|
||||
token = _login_admin(admin_client)
|
||||
headers = {"Authorization": f"Bearer {token}"}
|
||||
|
||||
listing = admin_client.get(
|
||||
"/api/config/ai-models",
|
||||
params={"provider": "mistral"},
|
||||
headers=headers,
|
||||
)
|
||||
assert listing.status_code == 200, listing.text
|
||||
data = listing.json()
|
||||
assert data["source"] == "live"
|
||||
assert data["capabilities"]["mistral-small-latest"]["vision"] is True
|
||||
assert data["capabilities"]["mistral-medium-latest"]["vision"] is True
|
||||
# The declaration also revokes: mistral-embed is not a chat model.
|
||||
assert data["capabilities"]["mistral-embed"]["chat"] is False
|
||||
assert data["capabilities"]["mistral-embed"]["embeddings"] is True
|
||||
|
||||
for model in ("mistral-small-latest", "mistral-medium-latest"):
|
||||
detail = admin_client.get(
|
||||
"/api/ai/model-capabilities",
|
||||
params={"provider": "mistral", "model": model},
|
||||
headers=headers,
|
||||
)
|
||||
assert detail.status_code == 200, detail.text
|
||||
assert detail.json()["capabilities"]["vision"] is True, model
|
||||
|
||||
def test_text_only_mistral_models_stay_text_only(self, admin_client, monkeypatch):
|
||||
"""mistral-large-latest declares no vision in the real API — keep it so."""
|
||||
self._fake_key(monkeypatch)
|
||||
self._serve_payload(monkeypatch, MISTRAL_LIVE_PAYLOAD)
|
||||
|
||||
token = _login_admin(admin_client)
|
||||
resp = admin_client.get(
|
||||
"/api/config/ai-models",
|
||||
params={"provider": "mistral"},
|
||||
headers={"Authorization": f"Bearer {token}"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
caps = resp.json()["capabilities"]["mistral-large-latest"]
|
||||
assert caps["vision"] is False
|
||||
assert caps["embeddings"] is False
|
||||
|
||||
def test_curated_fallback_covers_mistral_without_any_api_call(
|
||||
self, admin_client, monkeypatch,
|
||||
):
|
||||
"""No API key: the curated list must still answer correctly (offline)."""
|
||||
import backend.ai as aimod
|
||||
import backend.main as bmain
|
||||
|
||||
monkeypatch.setattr(aimod, "get_ai_key", lambda name: None)
|
||||
if hasattr(bmain, "get_ai_key"):
|
||||
monkeypatch.setattr(bmain, "get_ai_key", lambda name: None)
|
||||
|
||||
token = _login_admin(admin_client)
|
||||
resp = admin_client.get(
|
||||
"/api/config/ai-models",
|
||||
params={"provider": "mistral"},
|
||||
headers={"Authorization": f"Bearer {token}"},
|
||||
)
|
||||
assert resp.status_code == 200, resp.text
|
||||
data = resp.json()
|
||||
assert data["source"] == "fallback"
|
||||
caps = data["capabilities"]
|
||||
assert caps["mistral-small-latest"]["vision"] is True
|
||||
assert caps["mistral-large-latest"]["vision"] is False
|
||||
assert caps["mistral-large-latest"]["embeddings"] is False
|
||||
|
||||
@@ -2,6 +2,8 @@
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from backend.model_capabilities import (
|
||||
CAPABILITY_KEYS,
|
||||
get_capabilities_for_models,
|
||||
@@ -10,6 +12,16 @@ from backend.model_capabilities import (
|
||||
)
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _no_declared_caps():
|
||||
"""Curated-table tests must not see the process-wide declaration cache."""
|
||||
from backend.provider_capabilities import clear_declared_capabilities
|
||||
|
||||
clear_declared_capabilities()
|
||||
yield
|
||||
clear_declared_capabilities()
|
||||
|
||||
|
||||
class TestCapabilityShape:
|
||||
def test_every_result_has_all_keys(self):
|
||||
for provider, model in [
|
||||
@@ -84,3 +96,85 @@ class TestBatch:
|
||||
)
|
||||
assert result["qwen-max"]["vision"] is False
|
||||
assert result["qwen-vl-max"]["vision"] is True
|
||||
|
||||
|
||||
class TestMistralFamilies:
|
||||
"""BUG-044 — the curated fallback must know the current Mistral families.
|
||||
|
||||
This fallback is what answers when the provider declaration is unavailable
|
||||
(no API key, offline, cold cache), so it has to agree with what
|
||||
``api.mistral.ai/v1/models`` declares today. Before the fix, no Mistral
|
||||
model at all was reported as vision-capable — only the dead ``pixtral``
|
||||
pattern matched.
|
||||
"""
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"model",
|
||||
[
|
||||
"mistral-small-latest",
|
||||
"mistral-small-2603",
|
||||
"mistral-medium-latest",
|
||||
"mistral-medium-3.5",
|
||||
"ministral-3b-latest",
|
||||
"ministral-8b-2512",
|
||||
"ministral-14b-latest",
|
||||
"magistral-small-latest",
|
||||
"magistral-medium-latest",
|
||||
"mistral-vibe-cli-latest",
|
||||
"pixtral-12b-2409",
|
||||
],
|
||||
)
|
||||
def test_vision_chat_families(self, model):
|
||||
caps = get_model_capabilities("mistral", model)
|
||||
assert caps["vision"] is True, model
|
||||
assert caps["chat"] is True, model
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"model",
|
||||
[
|
||||
"mistral-large-latest",
|
||||
"codestral-latest",
|
||||
"open-mixtral-8x7b",
|
||||
"voxtral-small-latest",
|
||||
"mistral-code-latest",
|
||||
],
|
||||
)
|
||||
def test_text_only_models_are_not_vision(self, model):
|
||||
caps = get_model_capabilities("mistral", model)
|
||||
assert caps["vision"] is False, model
|
||||
assert caps["chat"] is True, model
|
||||
|
||||
def test_ocr_models_are_vision_without_chat(self):
|
||||
caps = get_model_capabilities("mistral", "mistral-ocr-latest")
|
||||
assert caps["vision"] is True
|
||||
assert caps["chat"] is False
|
||||
|
||||
def test_embeddings_only_for_the_embed_models(self):
|
||||
assert get_model_capabilities("mistral", "mistral-embed")["embeddings"] is True
|
||||
# Regression: every Mistral model used to inherit embeddings=True from
|
||||
# the provider default, wrongly branding chat models as embedders.
|
||||
assert get_model_capabilities("mistral", "mistral-large-latest")["embeddings"] is False
|
||||
assert get_model_capabilities("mistral", "some-unknown-mistral")["embeddings"] is False
|
||||
|
||||
def test_current_api_vision_models_are_all_covered(self):
|
||||
"""Every model the API declares vision=true is vision-capable offline.
|
||||
|
||||
List captured from ``GET https://api.mistral.ai/v1/models`` (2026-09-15);
|
||||
the live declaration layer covers renames, this guards the offline path.
|
||||
"""
|
||||
api_vision_models = [
|
||||
"magistral-medium-latest", "magistral-small-latest",
|
||||
"ministral-14b-2512", "ministral-14b-latest",
|
||||
"ministral-3b-2512", "ministral-3b-latest",
|
||||
"ministral-8b-2512", "ministral-8b-latest",
|
||||
"mistral-medium", "mistral-medium-2604", "mistral-medium-3",
|
||||
"mistral-medium-3-5", "mistral-medium-3.5", "mistral-medium-latest",
|
||||
"mistral-ocr-2512", "mistral-ocr-3", "mistral-ocr-3-0",
|
||||
"mistral-ocr-4", "mistral-ocr-4-0", "mistral-ocr-4-1",
|
||||
"mistral-ocr-latest",
|
||||
"mistral-small-2603", "mistral-small-latest",
|
||||
"mistral-vibe-cli-fast", "mistral-vibe-cli-latest",
|
||||
"mistral-vibe-cli-with-tools",
|
||||
]
|
||||
missing = [m for m in api_vision_models if not model_supports_vision("mistral", m)]
|
||||
assert missing == [], f"curated fallback misses vision for: {missing}"
|
||||
|
||||
@@ -0,0 +1,205 @@
|
||||
"""Tests for provider-declared capabilities + the cache (BUG-044).
|
||||
|
||||
Covers:
|
||||
- payload-shape parsing (Mistral ``capabilities``, OpenRouter ``architecture``);
|
||||
- the in-process cache (TTL, clearing, malformed payloads);
|
||||
- the merge rule: a declaration wins for the flags it mentions, the curated
|
||||
table fills the rest (``backend.model_capabilities``).
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import time
|
||||
|
||||
import pytest
|
||||
|
||||
from backend import provider_capabilities as pc
|
||||
from backend.model_capabilities import CAPABILITY_KEYS, get_model_capabilities
|
||||
|
||||
MISTRAL_PAYLOAD = {
|
||||
"object": "list",
|
||||
"data": [
|
||||
{
|
||||
"id": "mistral-small-latest",
|
||||
"capabilities": {
|
||||
"completion_chat": True,
|
||||
"vision": True,
|
||||
"audio_transcription": False,
|
||||
"audio_speech": False,
|
||||
},
|
||||
},
|
||||
{"id": "mistral-embed", "capabilities": {"completion_chat": False, "vision": False}},
|
||||
{"id": "mistral-ocr-latest", "capabilities": {"completion_chat": False, "vision": True}},
|
||||
{
|
||||
"id": "voxtral-mini-latest",
|
||||
"capabilities": {"completion_chat": False, "audio_transcription": True},
|
||||
},
|
||||
{"id": "mistral-new-thing", "capabilities": {"completion_chat": True, "vision": True}},
|
||||
{"id": "mistral-moderation-2603", "capabilities": {"completion_chat": False}},
|
||||
{"id": "no-capabilities-declared"},
|
||||
],
|
||||
}
|
||||
|
||||
OPENROUTER_PAYLOAD = {
|
||||
"data": [
|
||||
{
|
||||
"id": "openai/gpt-4o",
|
||||
"architecture": {
|
||||
"input_modalities": ["text", "image"],
|
||||
"output_modalities": ["text"],
|
||||
},
|
||||
},
|
||||
{
|
||||
"id": "text-only/model",
|
||||
"architecture": {"input_modalities": ["text"], "output_modalities": ["text"]},
|
||||
},
|
||||
{
|
||||
"id": "tts/model",
|
||||
"architecture": {"input_modalities": ["text"], "output_modalities": ["audio"]},
|
||||
},
|
||||
{
|
||||
"id": "whisper/model",
|
||||
"architecture": {"input_modalities": ["audio"], "output_modalities": ["text"]},
|
||||
},
|
||||
{
|
||||
"id": "video/model",
|
||||
"architecture": {"input_modalities": ["text"], "output_modalities": ["video"]},
|
||||
},
|
||||
]
|
||||
}
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def _clean_cache():
|
||||
"""The capability cache is process-wide: never leak between tests."""
|
||||
pc.clear_declared_capabilities()
|
||||
yield
|
||||
pc.clear_declared_capabilities()
|
||||
|
||||
|
||||
class TestParsing:
|
||||
def test_mistral_capability_flags_are_mapped(self):
|
||||
declared = pc.parse_declared_capabilities(MISTRAL_PAYLOAD["data"][0])
|
||||
assert declared == {
|
||||
"chat": True,
|
||||
"vision": True,
|
||||
"audio_transcription": False,
|
||||
"audio_speech": False,
|
||||
}
|
||||
assert set(declared) <= set(CAPABILITY_KEYS)
|
||||
|
||||
def test_mistral_ocr_is_vision_without_chat(self):
|
||||
declared = pc.parse_declared_capabilities(MISTRAL_PAYLOAD["data"][2])
|
||||
assert declared == {"chat": False, "vision": True}
|
||||
|
||||
def test_realtime_transcription_counts_as_transcription(self):
|
||||
entry = {
|
||||
"id": "model",
|
||||
"capabilities": {"completion_chat": False, "audio_transcription_realtime": True},
|
||||
}
|
||||
assert pc.parse_declared_capabilities(entry) == {
|
||||
"chat": False,
|
||||
"audio_transcription": True,
|
||||
}
|
||||
|
||||
def test_openrouter_modalities_are_mapped(self):
|
||||
entries = {e["id"]: e for e in OPENROUTER_PAYLOAD["data"]}
|
||||
assert pc.parse_declared_capabilities(entries["openai/gpt-4o"]) == {
|
||||
"chat": True,
|
||||
"vision": True,
|
||||
"images": False,
|
||||
"audio_transcription": False,
|
||||
"audio_speech": False,
|
||||
"video": False,
|
||||
}
|
||||
assert pc.parse_declared_capabilities(entries["tts/model"])["audio_speech"] is True
|
||||
assert pc.parse_declared_capabilities(entries["whisper/model"])["audio_transcription"] is True
|
||||
assert pc.parse_declared_capabilities(entries["video/model"])["video"] is True
|
||||
text_only = pc.parse_declared_capabilities(entries["text-only/model"])
|
||||
assert text_only is not None and text_only["vision"] is False
|
||||
|
||||
def test_undeclared_entries_return_none(self):
|
||||
for entry in [None, "not-a-dict", {}, {"architecture": {}}, {"capabilities": {}}]:
|
||||
assert pc.parse_declared_capabilities(entry) is None
|
||||
|
||||
|
||||
class TestCache:
|
||||
def test_remember_returns_cached_model_count(self):
|
||||
# 7 entries, 1 without any declaration.
|
||||
assert pc.remember_declared_capabilities("mistral", MISTRAL_PAYLOAD) == 6
|
||||
assert pc.cache_info()["mistral"]["models"] == 6
|
||||
|
||||
def test_unknown_model_returns_none(self):
|
||||
pc.remember_declared_capabilities("mistral", MISTRAL_PAYLOAD)
|
||||
assert pc.get_declared_capabilities("mistral", "not-in-payload") is None
|
||||
|
||||
def test_unknown_provider_returns_none(self):
|
||||
pc.remember_declared_capabilities("mistral", MISTRAL_PAYLOAD)
|
||||
assert pc.get_declared_capabilities("openrouter", "mistral-small-latest") is None
|
||||
|
||||
def test_payload_can_be_a_bare_list(self):
|
||||
assert pc.remember_declared_capabilities("mistral", MISTRAL_PAYLOAD["data"]) == 6
|
||||
|
||||
def test_shape_that_declares_nothing_is_not_cached(self):
|
||||
"""Gemini-shaped payloads (``{"models": [...]}``) declare nothing."""
|
||||
gemini_like = {"models": [{"name": "models/gemini-2.0-flash"}]}
|
||||
assert pc.remember_declared_capabilities("gemini", gemini_like) == 0
|
||||
assert "gemini" not in pc.cache_info()
|
||||
|
||||
def test_gemini_models_prefix_is_normalised(self):
|
||||
payload = [{"name": "models/gemini-x", "capabilities": {"completion_chat": True}}]
|
||||
pc.remember_declared_capabilities("gemini", payload)
|
||||
assert pc.get_declared_capabilities("gemini", "gemini-x") == {"chat": True}
|
||||
assert pc.get_declared_capabilities("gemini", "models/gemini-x") == {"chat": True}
|
||||
|
||||
def test_snapshot_expires_after_ttl(self, monkeypatch):
|
||||
pc.remember_declared_capabilities("mistral", MISTRAL_PAYLOAD)
|
||||
monkeypatch.setattr(pc, "TTL_SECONDS", 0.01)
|
||||
time.sleep(0.05)
|
||||
assert pc.get_declared_capabilities("mistral", "mistral-small-latest") is None
|
||||
assert pc.cache_info()["mistral"]["expired"] is True
|
||||
|
||||
def test_clear_one_provider_only(self):
|
||||
pc.remember_declared_capabilities("mistral", MISTRAL_PAYLOAD)
|
||||
pc.remember_declared_capabilities("openrouter", OPENROUTER_PAYLOAD)
|
||||
pc.clear_declared_capabilities("mistral")
|
||||
assert pc.get_declared_capabilities("mistral", "mistral-small-latest") is None
|
||||
assert pc.get_declared_capabilities("openrouter", "openai/gpt-4o") is not None
|
||||
|
||||
|
||||
class TestMergeWithCuratedTable:
|
||||
def test_declaration_fixes_a_model_the_curated_table_never_heard_of(self):
|
||||
model = "mistral-new-thing"
|
||||
assert get_model_capabilities("mistral", model)["vision"] is False # curated alone
|
||||
pc.remember_declared_capabilities("mistral", MISTRAL_PAYLOAD)
|
||||
caps = get_model_capabilities("mistral", model)
|
||||
assert caps["vision"] is True
|
||||
assert caps["chat"] is True
|
||||
assert set(caps) == set(CAPABILITY_KEYS)
|
||||
|
||||
def test_declaration_revokes_a_wrong_curated_guess(self):
|
||||
"""voxtral is audio-only: the curated default called it a chat model."""
|
||||
assert get_model_capabilities("mistral", "voxtral-mini-latest")["chat"] is True
|
||||
assert get_model_capabilities("mistral", "voxtral-mini-latest")["audio_transcription"] is False
|
||||
pc.remember_declared_capabilities("mistral", MISTRAL_PAYLOAD)
|
||||
caps = get_model_capabilities("mistral", "voxtral-mini-latest")
|
||||
assert caps["chat"] is False
|
||||
assert caps["audio_transcription"] is True
|
||||
|
||||
def test_curated_table_fills_flags_the_provider_stays_silent_about(self):
|
||||
"""Mistral never declares ``embedding``: only the absence of chat."""
|
||||
pc.remember_declared_capabilities("mistral", MISTRAL_PAYLOAD)
|
||||
caps = get_model_capabilities("mistral", "mistral-embed")
|
||||
assert caps["chat"] is False
|
||||
assert caps["embeddings"] is True
|
||||
|
||||
def test_openrouter_vision_comes_from_the_declaration(self):
|
||||
pc.remember_declared_capabilities("openrouter", OPENROUTER_PAYLOAD)
|
||||
assert get_model_capabilities("openrouter", "openai/gpt-4o")["vision"] is True
|
||||
assert get_model_capabilities("openrouter", "text-only/model")["vision"] is False
|
||||
|
||||
def test_clearing_the_cache_restores_the_curated_answer(self):
|
||||
pc.remember_declared_capabilities("mistral", MISTRAL_PAYLOAD)
|
||||
assert get_model_capabilities("mistral", "mistral-new-thing")["vision"] is True
|
||||
pc.clear_declared_capabilities()
|
||||
assert get_model_capabilities("mistral", "mistral-new-thing")["vision"] is False
|
||||
Reference in New Issue
Block a user