Compare commits

...
29 Commits
Author SHA1 Message Date
bruno dff32a97ee feat: styles minimaux, fusions et volets figes dans la visionneuse xlsx #153
A15 — le rendu tableur lit les metadonnees du classeur (une charge en mode
normal par feuille) et les restitue : styles inline data-driven (couleur de
police en premier, fond, gras/italique, format numerique signale en mono),
alignements non-defaut, plages fusionnees (rowSpan/colSpan cote client) et
ancrage des volets figes (sticky rows/cols). Le chargement lazy replie les
metadonnees de chaque fenetre. Contrat API : styles/aligns = {ref: css},
merges = [ranges], freeze = ancre. 9 tests backend + contre-preuve (5 echecs
sur neutralisation), JSDOM 33/33.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <[email protected]>
2026-09-28 16:07:58 -04:00
bruno d5c528fead feat: structure des classeurs Excel editable depuis la visionneuse #153
A14 : service mutate_xlsx_structure (batch ordonné d'actions
sheet_add/sheet_rename/sheet_delete/sheet_duplicate et
row/col_insert/delete, une seule réécriture verrouillée et atomique,
409 lossy sans force, backup, dernier sheet protégé) ; endpoint
PUT /api/file/{vault}/xlsx/structure (1-50 actions/requête) ;
menu « structure » dans la visionneuse : ajout/renommage/duplication/
suppression de feuilles et insertion/suppression de ligne/colonne à
partir de la cellule active, confirmations explicites pour les
destructions, re-rendu serveur après succès, reprise force après
confirmation 409.

Vérifié : test_xlsx_structure.py 11 (ops, gardes, atomicité du batch,
409 lossy + force) + contre-preuve (garde dernier sheet neutralisée ->
1 échec), xlsx-viewer.test.mjs 33/33 (3 nouveaux), ruff 0, mypy 0,
i18n parity, validate-imports.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <[email protected]>
2026-09-28 12:53:15 -04:00
bruno 38f39a10ae feat: tri, filtre, recherche et export CSV dans la visionneuse XLSX #153
A13 (affichage seul, le classeur n'est jamais réécrit) : clic sur un
en-tête pour trier la colonne (asc/desc, numérique si toutes les valeurs
le sont, localeCompare fr sinon — les cellules modifiées suivent leur
ligne) ; l'unique champ de recherche filtre les lignes ET surligne les
occurrences (marque <mark>, navigation ↑/↓/Entrée, compteur i18n,
insensible à la casse par défaut, bouton Aa) ; bouton « Réinitialiser »
qui re-rend le fichier (vérité serveur) ; export CSV de la feuille
visible (BOM UTF-8, échappement ;/" et retrait des ombres de valeurs
calculées).

Vérifié : xlsx-viewer.test.mjs 30/30 (5 nouveaux) + contre-preuves
(tri neutralisé -> 1 échec, filtre neutralisé -> 1 échec), validate-imports,
i18n parity (xlsx.find_*, xlsx.sort_*, xlsx.csv_export).

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <[email protected]>
2026-09-28 12:28:09 -04:00
bruno 48e023ba25 feat: navigation clavier et barre de formule dans la visionneuse XLSX #153
A7 : barre de formule sous la barre d'outils (nom de cellule active A1 +
miroir du contenu, édition live, Entrée valide, Échap annule puis
refocalise) ; flèches et Tab déplacent la cellule active dans les 4
directions (Shift inverse Tab), outline persistant sur la cellule active
quand le focus passe à la barre ; Maj+Entrée ne valide plus (réservé au
multiligne A7+). td.tabIndex=0 pour la focalisation clavier.

Vérifié : xlsx-viewer.test.mjs 25/25 (6 nouveaux), E2E 9/9 (barre +
flèches + Échap sur fixture réelle), validate-imports, i18n parity
(xlsx.formula_bar_placeholder, xlsx.active_cell).

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <[email protected]>
2026-09-28 12:18:24 -04:00
bruno 011ec84f23 feat: outils IA de lecture et d'edition des classeurs existants #153
A6 : list_xlsx_sheets (noms + dimensions + truncated), xlsx_to_markdown
(table bornee 100x20 pour le contexte LLM), update_xlsx_cells et
append_xlsx_rows (WRITE + confirmation, via edit_xlsx_cells : verrou,
ecriture atomique, neutralisation des formules, 409 lossy). Les lignes
ajoutees sont coercees comme dans le viewer (A10). Labels de step
ai.step.xlsx_* FR/EN ; update_xlsx_cells/append_xlsx_rows branches sur
MUTATING_TOOLS et FILE_WRITE_TOOLS (refresh viewer via obsigate:file-written).

Tests : test_spreadsheet_tools.py 17 (risque, confirmation, persistance,
garde formule heritee, coercion) + contre-preuve (cells vide -> 2 echecs).
ruff 0, mypy 0, i18n parity, validate-imports.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <[email protected]>
2026-09-28 12:05:16 -04:00
bruno 472ea9d309 feat: chargement à la demande des lignes cachées des tableurs #153
CI / lint (push) Successful in 2m24s
CI / security (push) Failing after 1m39s
CI / test (push) Successful in 4m0s
CI / build (push) Successful in 1m37s
CI / e2e (push) Successful in 15m34s
A9bis : sous une feuille tronquée, un pied de page « N lignes affichées
sur M · Charger la suite » fetch la fenêtre suivante (limit=500) au clic
ou à l'approche du bas du tableau (sentinelle de défilement, marge 120px).
Les lignes ajoutées passent par le même pipeline d'édition que le rendu
initial (setupCell factorisé) : éditables et sauvegardables immédiatement.
Fetch échoué → bouton restauré (retry) + toast ; feuille complète → pied
de page masqué (class done).

Vérifié : xlsx-viewer.test.mjs 19/19 (5 nouveaux, contre-preuve
wireLazyRows désactivé → 5 échecs), E2E 8/8 (A520 visible et éditable
après clic), validate-imports 40 modules, unit.test.mjs 12/12, i18n parity.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <[email protected]>
2026-09-28 11:52:42 -04:00
bruno 6ba04c4381 feat: troncature annoncee et lecture par fenetres des tableurs #153
CI / lint (push) Successful in 2m29s
CI / security (push) Failing after 1m39s
CI / test (push) Successful in 4m1s
CI / build (push) Successful in 1m37s
CI / e2e (push) Successful in 14m47s
- BUG-090 (#153 A8) : render_sheets() expose total_rows/total_cols,
  max_rows/max_cols et truncated ; la visionneuse affiche un bandeau
  « Feuille tronquée » (i18n FR/EN) au lieu de couper en silence, et la
  ligne d'en-têtes devient sticky (top:auto sur les numéros de ligne).
- #153 A9 : GET /api/file/{vault}/xlsx/sheet?sheet&offset&limit sert une
  fenêtre de 1 à 1000 lignes avec les vraies coordonnées A1, has_more de
  pagination et valeurs calculées A12 ; 404 feuille inconnue, 415 non-xlsx.
- Tests : TestXlsxTruncationNotice (4) + TestXlsxSheetWindow (11) avec
  contre-preuves, xlsx-viewer.test.mjs 14/14, E2E 7/7 (fixture
  sample-xlsx-large.xlsx 520 lignes), suite 1417 passed / 6 skipped,
  ruff/mypy 0, i18n parity.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <[email protected]>
2026-09-28 10:38:24 -04:00
bruno 06f8e63d06 feat: tableurs indexables, saisie typée et valeurs calculées #153
Les tableurs étaient invisibles à la recherche, chaque saisie devenait
du texte, et une formule n'affichait que sa formule.

- A5 : extract_indexable_text() indexe les noms de feuilles et les 20
  premières lignes (plafond 5 k caractères) pour le TF-IDF et la
  recherche sémantique. Un mot tapé dans une cellule rend le fichier
  trouvable ; un classeur chiffré s'indexe par son seul nom.
- A10 : la saisie est typée comme dans Excel — booléens
  (TRUE/FAUX/OUI/NON) et dates FR JJ/MM/AAAA en ordre jour-first, donc
  01/02/2026 est le 1er février. Une saisie ressemblant à une formule
  n'est jamais convertie (BUG-088 préservé).
- A12 : la valeur calculée par Excel s'affiche sous la formule, via une
  2e lecture data_only=True faite seulement si l'archive contient un
  <v>. Info-bulle traduite FR/EN, aucun texte d'interface côté backend.
- BUG-089 : un reindex manuel ne reconstruisait pas l'index inversé, et
  backend/search.py lisait l'index via un import par valeur — après un
  rechargement du module, la recherche écrivait dans un dict périmé.

Contre-preuves vérifiées pour A5, A10 et A12 (neutralisation de chaque
fonction → échec des tests concernés). Tests : 1402 pytest, 10 JSDOM,
4 E2E, suite E2E complète verte, ruff/mypy 0.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <[email protected]>
2026-09-27 22:43:24 -04:00
bruno 31d4616baf feat: garde-fous d'écriture des classeurs Excel #153 (P0)
L'édition d'un .xlsx pouvait détruire une partie du classeur, le
concurrencer en silence, ou diffuser une injection de formule.

- BUG-085 : inspect_workbook() détecte ce qu'un round-trip openpyxl perd
  (valeurs calculées en cache, slicers, contrôles, connexions, custom
  XML, signature, commentaires enrichis, macros) → xlsx_lossy_features
  exposé en lecture, bandeau FR/EN, et 409 xlsx_lossy_content sans
  `force` (confirmation explicite puis reprise). Périmètre réel
  revalidé : graphiques, images et TCD survivent au round-trip.
- BUG-086 : écriture atomique (fichier .tmp + os.replace) : un plantage
  ne peut plus tronquer le classeur, le backup reste intact.
- BUG-087 : verrou par fichier autour du read-modify-write (timeout 15 s,
  409 conflict) ; endpoint xlsx/save devenu synchrone pour que
  l'attente s'exécute dans le threadpool.
- BUG-088 : une saisie en '=' ou '@' est stockée en texte, sauf opt-in
  `allow_formula` ou le bouton f(x) de la visionneuse. Le handler
  ServiceError expose désormais code + details, que api() propage.
- BUG-084 : la suppression d'une vault purge enfin l'index inversé
  (documents fantômes qui continuaient de matcher) et is_stale() devient
  is_ready(), le nom étant trompeur (la staleness n'existe plus).

Tests : 1390 pytest, 10 JSDOM (xlsx-viewer.test.mjs, branché au CI),
3 E2E Playwright, suite E2E complète verte, ruff/mypy 0.

🤖 Generated with Codebuff
Co-Authored-By: Codebuff <[email protected]>
2026-09-27 20:39:12 -04:00
bruno 4c4b1222d5 chore: ignore les diagrammes E2E générés (excalidraw)
CI / lint (push) Successful in 2m23s
CI / security (push) Failing after 1m48s
CI / test (push) Successful in 3m49s
CI / build (push) Successful in 2m28s
CI / e2e (push) Successful in 14m5s
2026-09-27 14:46:09 -04:00
bruno a3973b981c securite: #87 T6-T8 fin dette — deps qualifiées, semgrep, Secure auto, CORS 2026-09-27 12:43:31 -04:00
bruno b6e2029770 fix: E2E node-direct, timeouts réalistes 25/30 min (complément BUG-080)
CI / lint (push) Successful in 2m22s
CI / security (push) Successful in 1m37s
CI / test (push) Successful in 4m48s
CI / build (push) Successful in 1m33s
CI / e2e (push) Successful in 14m3s
2026-09-27 11:28:35 -04:00
bruno 6b878caff3 securite: #87 T5c script-src sans unsafe-inline (nonces T5b)
CI / lint (push) Successful in 2m18s
CI / security (push) Successful in 1m37s
CI / test (push) Successful in 4m32s
CI / build (push) Successful in 1m34s
CI / e2e (push) Successful in 14m22s
2026-09-27 10:36:00 -04:00
bruno e9b7a317c1 fix: mfa/status 200 auth désactivée (garde anonymous) BUG-081 + clôture BUG-080/082/083 2026-09-27 10:35:33 -04:00
bruno 14b8032635 fix: CI lint — config-ai-keys.test.mjs rejoint l'étape JSDOM (complément BUG-082)
CI / security (push) Successful in 1m37s
CI / lint (push) Successful in 2m18s
CI / test (push) Successful in 4m8s
CI / build (push) Successful in 1m38s
CI / e2e (push) Successful in 14m28s
2026-09-27 09:44:18 -04:00
bruno 7dfe26c83d fix: CI security — echo pip-audit sans dièse (runner Act) BUG-083
CI / lint (push) Failing after 1m50s
CI / test (push) Skipped
CI / build (push) Skipped
CI / e2e (push) Skipped
CI / security (push) Successful in 1m37s
2026-09-27 09:37:34 -04:00
bruno 24229316c7 fix: CI lint — upload.test.mjs rejoint l'étape JSDOM (jsdom) BUG-082
CI / lint (push) Failing after 1m30s
CI / test (push) Skipped
CI / build (push) Skipped
CI / e2e (push) Skipped
CI / security (push) Failing after 1m33s
2026-09-27 09:23:51 -04:00
bruno 7d70e0fb75 fix: harnais E2E local anti-blocage BUG-080
CI / lint (push) Failing after 1m51s
CI / test (push) Skipped
CI / build (push) Skipped
CI / e2e (push) Skipped
CI / security (push) Failing after 1m33s
2026-09-27 09:08:58 -04:00
bruno d70ecd0968 securite: #87 T5b nonces CSP prets pour bascule (sans changement) 2026-09-26 22:44:12 -04:00
bruno 36a4030c09 securite: #87 T5a supprime 16 handlers inline au profit de listeners (CSP inchangee) 2026-09-26 22:32:01 -04:00
bruno 330462e7a5 docs: #87 T4 consigne resultats E2E locaux (108/108 desktop, 10/10 mobiles cibles) 2026-09-26 22:22:31 -04:00
bruno 922dfa2e79 test: #87 T4 E2E XSS partage et lecteur + script serveur E2E avec progression 2026-09-26 21:46:16 -04:00
bruno 34fce932cb securite: #87 T3 cookies Secure centralises + CORS atteste (defaut inchange) 2026-09-26 18:37:19 -04:00
bruno 18b1e13f34 test: #87 T2 durcissement concurrence users.json et budget temps regex 2026-09-26 18:34:06 -04:00
bruno 7bee4a237d ci: #87 T1 bandit et npm audit bloquants, 5 suites frontend au CI 2026-09-26 18:30:04 -04:00
bruno d6cca2b1af feat: #85 T10 persistance etat (verrous stores, ratelimit SQLite) et cloture refonte 2026-09-26 18:10:10 -04:00
bruno 58312e64da refactor: #85 T9 extrait realtime et render vers modules dedies (comportement inchange) 2026-09-26 16:59:25 -04:00
bruno 3b0927a8c9 refactor: #85 T8 extrait vaults-history-conflicts vers backend/routers (comportement inchange) 2026-09-26 14:46:55 -04:00
bruno 6e527c371d refactor: #85 T7 extrait le domaine config vers backend/routers (comportement inchange) 2026-09-26 14:33:40 -04:00
98 changed files with 9317 additions and 1547 deletions
+8 -2
View File
@@ -12,8 +12,12 @@ OBSIGATE_ADMIN_PASSWORD=chab30
# (ex. 0.0.0.0) sauf si l'on force l'opt-in ci-dessous. À réserver au local.
# OBSIGATE_ALLOW_INSECURE=false
# Sécurité des cookies (activer si derrière HTTPS)
# OBSIGATE_SECURE_COOKIES=false
# Sécurité des cookies : true|false|auto (défaut : auto — Secure si la
# requête arrive en https, sinon pas de flag ; les navigateurs ignorent les
# cookies `Secure` en HTTP, ce qui casserait les logins en local).
# Derrière un reverse proxy qui termine TLS, auto suffit avec
# OBSIGATE_TRUST_PROXY=true (X-Forwarded-Proto honoré).
# OBSIGATE_SECURE_COOKIES=auto
# Tokens TTL en secondes
# OBSIGATE_ACCESS_TOKEN_TTL=31536000000 # 1000 ans
@@ -23,6 +27,8 @@ OBSIGATE_ADMIN_PASSWORD=chab30
# OBSIGATE_LOGIN_MAX_ATTEMPTS=10
# OBSIGATE_ACCOUNT_MAX_ATTEMPTS=10
# OBSIGATE_LOGIN_WINDOW_SECONDS=900
# Compteurs partagés/persistants (SQLite WAL, multi-workers) — défaut : mémoire.
# OBSIGATE_RATELIMIT_DB=data/ratelimit.db
# IP client derrière un reverse proxy (fait confiance à X-Forwarded-For)
# OBSIGATE_TRUST_PROXY=false
+34 -6
View File
@@ -44,8 +44,11 @@ jobs:
node tests/frontend/config-mobile.test.mjs
node tests/frontend/settings-order-avatar.test.mjs
node tests/frontend/mobile-toolbar.test.mjs
node tests/frontend/pretty.test.mjs
node tests/frontend/media-viewer.test.mjs
node tests/frontend/mfa-settings.test.mjs
- name: Frontend JSDOM tests (PaneManager + Excalidraw + Plugins + AI + SW + Collab + Mobile + Semantic + Desktop + Inline edition)
- name: Frontend JSDOM tests (PaneManager + Excalidraw + Plugins + AI + SW + Collab + Mobile + Semantic + Desktop + Inline edition + Upload + XLSX)
run: |
cd tests/frontend
if [ -d node_modules ]; then
@@ -63,6 +66,9 @@ jobs:
node toolbar-order.test.mjs
node editor-inline.test.mjs
node ai-quick-actions.test.mjs
node upload.test.mjs
node config-ai-keys.test.mjs
node xlsx-viewer.test.mjs
else
echo "tests/frontend/node_modules missing - installing jsdom"
npm install --no-audit --no-fund --silent
@@ -80,6 +86,9 @@ jobs:
node toolbar-order.test.mjs
node editor-inline.test.mjs
node ai-quick-actions.test.mjs
node upload.test.mjs
node config-ai-keys.test.mjs
node xlsx-viewer.test.mjs
fi
# ── Tests ─────────────────────────────────────────────────────────
@@ -123,14 +132,30 @@ jobs:
- name: Install dependencies
run: |
pip install bandit pip-audit
pip install bandit pip-audit semgrep
pip install -r backend/requirements.txt
- name: Bandit (SAST)
run: bandit -r backend/ --skip B101,B110,B310 || echo "bandit found issues (non-blocking)"
- name: Bandit (SAST, bloquant — #87)
# B105 est exclu (aligné avec [tool.bandit] de pyproject.toml :
# faux positifs systématiques sur les noms de variables) ; les rares
# vrais positifs restants portent un `# nosec` justifié inline.
run: bandit -r backend/ --skip B101,B105,B110,B310
- name: Pip-audit (dependency vulnerabilities)
run: pip-audit || echo "pip-audit found vulnerabilities (non-blocking)"
- name: Semgrep (SAST local, bloquant — #87)
# Règles 100 % locales (semgrep-rules/, 8 règles) : aucun
# téléchargement de registre (runner au réseau fragile).
run: semgrep --config semgrep-rules/ backend/
- name: Pip-audit (bloquant — #87)
# Bloquant depuis T6 (#87) : dépendances qualifiées (mistune 3.3.3,
# python-multipart 0.0.31, weasyprint 70, mcp 1.28.1, fastapi 0.141.1
# + starlette 1.7.0, setuptools 84 — suite complète verte + 0 vuln).
# Seule exception documentée : PYSEC-2026-1325 (ecdsa, Minerva) —
# aucun correctif upstream ET ObsiGate ne signe/vérifie qu'en HS256
# (backend/auth/jwt_handler.py), les chemins ECDSA P-256 ne
# s'exécutent jamais.
# NOTE runner Gitea Act (BUG-083) : aucun `#` dans le `run:`.
run: pip-audit --ignore-vuln PYSEC-2026-1325
# ── Docker build ──────────────────────────────────────────────────
build:
@@ -192,6 +217,9 @@ jobs:
npm ci
npx playwright install --with-deps chromium
- name: Npm audit (bloquant — #87, 0 dépendance prod hors Playwright)
run: npm audit --omit=dev
- name: Start ObsiGate
run: |
docker rm -f obsigate-e2e 2>/dev/null || true
+14
View File
@@ -31,6 +31,20 @@ desktop/backend/
desktop/frontend/
backend/VERSION
# Artefacts générés par les runs E2E (excalidraw crée ces diagrammes)
test_vault/IT/e2e-diagram-*.excalidraw
# Fixtures de test locales non versionnées (~200 Mo, pas de fixture CI).
# Aucun test/CI ne les référence : les tests unitaires génèrent leurs fixtures
# dans tmp_path (tests/conftest.py), et l'E2E n'utilise que les fixtures
# committées (test_vault/sample-*.{mp3,png,svg,webm,pdf}, test_dir/*.md).
# → à committer volontairement : `git add -f <chemin>`.
test_dir/music/
test_dir/video/
test_vault/images/
test_vault/markdown/
test_vault/budget.xlsx
# Tauri updater signing keys (private key — never commit)
desktop/*.key
desktop/*.key.pub
+497 -29
View File
@@ -6,7 +6,7 @@ Format basé sur [Keep a Changelog](https://keepachangelog.com/fr/1.1.0/),
et [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
> **En cours de développement** : les changements à venir sont listés dans la section
> [Unreleased](#unreleased). La dernière version livrée est **2.27.9**.
> [Unreleased](#unreleased). La dernière version livrée est **2.37.0**.
---
@@ -14,36 +14,469 @@ et [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
---
## [2.37.0] — 2026-09-28
---
## [2.36.0] — 2026-09-28
---
## [2.35.0] — 2026-09-28
---
## [2.34.0] — 2026-09-28
---
## [2.33.0] — 2026-09-28
---
## [2.32.0] — 2026-09-28
---
## [2.31.0] — 2026-09-28
### Correction
- **BUG-090 — troncature silencieuse d'une feuille `.xlsx` au-delà de
500 lignes × 40 colonnes.** `render_sheets()` renvoie les dimensions
déclarées par la feuille (`total_rows`/`total_cols`), les plafonds du
moteur (`max_rows`/`max_cols`) et un flag `truncated` : la visionneuse
affiche un bandeau « Feuille tronquée — 500 lignes affichées sur 520 »
(i18n FR/EN) au lieu de présenter une table courte comme complète. La
ligne d'en-têtes est désormais figée au défilement vertical (`thead`
sticky, `top: auto` sur les numéros de ligne pour éviter leur
empilement en haut à gauche). *#153 A8/R5.*
### Ajouté
- **#153 A9 — chargement paresseux d'une feuille par fenêtres.**
`GET /api/file/{vault}/xlsx/sheet?sheet=&offset=&limit=` renvoie un
bloc de lignes (`XlsxSheetWindowResponse`, plafond 1 000 lignes par
requête, `has_more` de pagination) avec les **vraies** coordonnées A1
et numéros de ligne de la feuille — une fenêtre se comporte exactement
comme le rendu complet. Erreurs typées : 404 feuille inconnue, 415
fichier non-`.xlsx`. La lecture des valeurs calculées en cache (#153
A12) s'applique aussi aux fenêtres.
- **#153 A9bis — « Charger la suite » sous une feuille tronquée.** Un
pied de page annonce la progression et fetch la fenêtre suivante au
clic ou à l'approche du bas du tableau (sentinelle de défilement).
Les lignes ajoutées passent par le même pipeline d'édition que le
rendu initial : éditables et sauvegardables immédiatement. Un fetch
échoué restore le bouton (retry possible) ; feuille complète → pied
de page masqué.
---
## [2.30.0] — 2026-09-27
### Correction
- **BUG-089 — un reindex manuel ne reconstruisait pas l'index inversé.**
`reload_index()` / `reload_single_vault()` remplacent l'entrée de vault
en bloc, ce qui n'émet pas les notifications incrémentales : la
recherche TF-IDF continuait de servir un index périmé après un
reindex. Les deux fonctions appellent désormais `init_inverted_index()`.
Au passage, `backend/search.py` lisait l'index via
`from backend.indexer import index` — une liaison **par valeur** du
dict : un rechargement du module `backend.indexer` recréait le dict
côté indexer alors que la recherche écrivait dans l'ancien, et
l'index inversé n'indexait plus rien. Tous les accès passent par
`_indexer.index`. *Trouvé en écrivant le test de recherche d'A5 : il
passait isolément et échouait en suite complète selon l'ordre.*
### Ajouté
- **#153 A5 — les tableurs sont indexés par leur contenu.**
`extract_indexable_text()` extrait les noms de feuilles et les 20
premières lignes (plafond 5 000 caractères, 20 feuilles) pour le
TF-IDF et la recherche sémantique. Un mot tapé dans une cellule rend
désormais le classeur trouvable ; la lecture binaire pour l'affichage
est inchangée et un classeur chiffré/corrompu s'indexe par son seul
nom.
- **#153 A10 — la saisie est typée comme dans Excel.** Une valeur
`TRUE`/`FAUX`/`OUI`/`NON` devient un booléen, une date `JJ/MM/AAAA`
(avec `HH:MM` optionnel) devient une vraie date — et dans l'ordre
français : `01/02/2026` est le 1ᵉʳ février. Une saisie ressemblant à
une formule n'est jamais convertie.
- **#153 A12 — la valeur calculée s'affiche sous la formule.** Quand une
cellule porte encore le résultat de son dernier calcul Excel, celui-ci
s'affiche dans une ligne discrète sous la formule. La seconde lecture
`data_only=True` n'a lieu que si l'archive contient réellement une
valeur en cache, et toute erreur retombe sur l'affichage formules seul.
Info-bulle traduite FR/EN (`xlsx.cached_value_title`).
---
## [2.29.0] — 2026-09-27
### Correction
- **BUG-084 — l'index inversé conservait des documents fantômes après la
suppression d'une vault.** `remove_vault_from_index()`
(`backend/indexer.py`) ne notifiait pas le hook incrémental : après
suppression d'une vault, ses documents restaient dans l'index inversé
(`postings`, `doc_info`, `doc_vault`, `vault_docs`) et continuaient de
correspondre aux recherches pour une vault inexistante — seul un reindex
manuel les effaçait. Le correctif déclenche
`_on_index_change('remove', …)` pour chaque fichier de la vault, et
`_remove_doc_internals()` supprime désormais la clé `vault_docs` dont le set
devient vide (c'est un `defaultdict` : une lecture la recréait).
Test : `TestVaultRemovalPurgesInvertedIndex` (contre-preuve : échoue sans le
correctif).
### Maintenance
- **Index inversé — `is_stale()` renommé `is_ready()`.** La relecture de
`plan.md` a établi que les étapes 6 et 7 (suppression du cooldown et du hack
de coalescence) étaient **déjà livrées** : ni `_last_rebuild`, ni
`_rebuild_cooldown`, ni `_source_generation`, ni `_on_vault_change` ne
subsistent. `is_stale()` ne mesurait donc plus aucune staleness — il
indiquait seulement si l'index initial était construit, sous un nom
trompeur. Renommé `is_ready()`, cohérent avec le `is_ready()` de
`SemanticIndex` ; l'alias `is_stale()` de `SemanticIndex`, sans appelant, est
supprimé. `/api/diagnostics` expose désormais `is_ready` (libellé « Index
prêt » côté `frontend/js/config.js`). Tests :
`test_is_ready_tracks_initial_build`, `test_is_ready_survives_incremental_updates`.
- **`plan.md` recalibré.** Le fichier est désormais marqué « livré » et
suivi d'une section « État réel » : le code a divergé du plan sur quatre
points (pas de repli `_needs_rebuild`, `_ready` au lieu de `doc_count == 0`,
`rebuild()` conservé au démarrage, `is_stale()` repurposé). Les extraits de
code du plan sont explicitement signalés comme ne décrivant pas le code
actuel.
- **Fixtures de test locales exclues du suivi Git.** `test_dir/music/`,
`test_dir/video/`, `test_vault/images/`, `test_vault/markdown/` et
`test_vault/budget.xlsx` (~200 Mo) sont ajoutés au `.gitignore` : aucun test
ni job CI ne les référence — les tests unitaires génèrent leurs fixtures dans
`tmp_path` et l'E2E n'utilise que les fixtures committées
(`test_vault/sample-*.{mp3,png,svg,webm,pdf}`, `test_dir/*.md`). Ils
restaient non suivis et polluaient `git status`.
### Sécurité
- **BUG-088 — plus d'injection de formule via la visionneuse Excel.** Une
saisie `=cmd|'/c calc'!A1` (ou `@…`) était stockée comme **formule** par
openpyxl, donc exécutée par Excel à la réouverture du fichier (DDE).
`edit_xlsx_cells` force maintenant le type texte (`cell.data_type = "s"`)
pour toute valeur commençant par `=` ou `@` ; l'API accepte
`allow_formula: true` et la visionneuse expose un bouton `f(x)`
(opt-in, état de session, jamais persisté). `+`/`-` restent des nombres.
- **BUG-087 — écriture concurrente d'un classeur.** `load_workbook()` →
`save()` n'était pas sérialisé : deux sauvegardes simultanées (deux
onglets, l'agent IA et la visionneuse) faisaient gagner la dernière, en
silence. Verrou par chemin (`backend/services/mutations.py::_xlsx_write_lock`,
timeout 15 s) autour du cycle lecture → édition → remplacement ; attente
dépassée → **409** `conflict`. L'endpoint `PUT …/xlsx/save` est devenu
synchrone pour que l'attente s'exécute dans le threadpool.
### Corrigé
- **BUG-085 — la perte de données à l'enregistrement d'un `.xlsx` est
annoncée, plus silencieuse.** `GET /api/file/{vault}` renvoie
`xlsx_lossy_features` (éléments qu'un round-trip openpyxl perd) ; la
visionneuse affiche un bandeau listant ces éléments et la première
sauvegarde demande confirmation avant de renvoyer `force: true`. Sans
`force`, l'API répond **409** `xlsx_lossy_content` avec
`details.features`. Périmètre **remesuré** sur openpyxl 3.1.5 : graphiques,
images, dessins et tableaux croisés sont bien préservés ; sont perdus les
valeurs calculées en cache, slicers/chronologies, contrôles de formulaire,
connexions/requêtes, custom XML, signature numérique, commentaires
enrichis et macros.
- **BUG-086 — écriture atomique des classeurs.** `wb.save()` écrivait en
place sur le fichier du vault : un plantage laissait un `.xlsx` tronqué.
L'écriture passe désormais par un `.tmp` puis `os.replace()` (le backup
`.bak` est inchangé, le `.tmp` est ignoré par le watcher).
- Le handler global `ServiceError` expose maintenant `code` et `details` dans
la réponse JSON, et `api()` (frontend) les propage sur l'Error — nécessaire
pour que le client distingue un 409 de confirmation d'une autre erreur.
### Ajouté
- **#153 (P0) — tests de la visionneuse Excel.**
`tests/frontend/xlsx-viewer.test.mjs` (10 tests JSDOM : bannière,
confirmation + reprise `force`, refus, toggle `f(x)`, payload de
sauvegarde) et `tests/e2e/xlsx-viewer.spec.js` (3 tests Playwright sur la
fixture `test_vault/sample-xlsx-lossy.xlsx`) ; la suite JSDOM est branchée
dans le CI.
### Documentation
- **#153 — Visionneuse & édition XLSX : audit complet et backlog de
complétude.** La visionneuse `.xlsx` livrée par #152 a été auditée couche
par couche (lecture `backend/xlsx_reader.py`, écriture
`backend/services.mutations.edit_xlsx_cells`, UI `renderXlsxViewer`,
indexation, outils IA, tests). Bilan : la grille de valeurs est éditée
correctement (sécurité, backup, audit, échappement HTML), mais l'ensemble
supporté est étroit, une partie du classeur est perdue à l'enregistrement,
les tableurs sont **invisibles pour la recherche** et l'IA ne sait que les
**créer**. Ouverture de l'item **#153** dans `docs/ROADMAP.md` (17
sous-tâches suivies **A1 → A17** ; **P0 livré**, reste P1 recherche/IA/UX
puis P2 étendu) et création de la fiche
[docs/features/xlsx-viewer.md](docs/features/xlsx-viewer.md) : cartographie
du code, limites par couche, tableau des risques R1-R5 et critères
d'acceptation par sous-tâche.
---
## [2.28.16] — 2026-09-27
---
## [2.28.15] — 2026-09-27
### Sécurité
- **#87 T6 — dépendances qualifiées, `pip-audit` bloquant (0 vulnérabilité).**
mistune 3.0.2 → 3.3.3 (XSS/ReDoS/DoS dans le moteur de rendu),
python-multipart 0.0.9 → 0.0.31, weasyprint 69 → 70, mcp 1.9.4 → 1.28.1,
fastapi 0.110.3 → 0.141.1 + starlette 0.37.2 → 1.7.0, setuptools 84 ;
`cast(str, …)` aux 3 sites d'appel mistune (typage 3.3 resserré). Suite
complète 1359 passed, ruff/mypy 0. Seule exception : PYSEC-2026-1325
(ecdsa, Minerva) — aucun correctif upstream ET JWT exclusivement HS256
(`backend/auth/jwt_handler.py`), les chemins ECDSA P-256 ne s'exécutent
jamais → `--ignore-vuln` documenté.
- **#87 T7 — semgrep SAST local bloquant (8 règles, 0 finding).**
Ruleset `semgrep-rules/` (eval/exec, shell=True, os.system, pickle,
yaml.load sans Loader, verify=False, Markup, mktemp) — 100 % local,
aucun registre réseau (runner au réseau fragile). Trivy écarté :
binaire + base de vulnérabilités à télécharger à chaque run, couche
Python déjà couverte par `pip-audit` bloquant (image = slim + 4 libs).
- **#87 T8 — fin BUG-034 : cookies Secure auto, CORS same-origin explicite.**
`OBSIGATE_SECURE_COOKIES=true|false|auto` (défaut auto : Secure en https,
sinon rien — logins http locaux préservés ; `X-Forwarded-Proto` honoré
sous `TRUST_PROXY`, avertissement démarrage affiné, `TRUST_PROXY=true`
dans le compose prod) ; `CORSMiddleware` same-origin explicite (sûr :
web et desktop Tauri same-origin, API directe hors navigateur) ;
`style-src 'unsafe-inline'` conservé et assumé (189 attributs `style=` +
343 `el.style` — suppression = réécriture complète, risque nul côté
exécution une fois `script-src` verrouillé en T5c).
---
## [2.28.14] — 2026-09-27
---
## [2.28.13] — 2026-09-27
### Modifié
- **#87 (T5c) — `script-src` sans `'unsafe-inline'`.**
Seuls les scripts avec nonce frais (`backend/csp.py`, T5b) ou servis par
`'self'`/CDN listés s'exécutent ; `style-src` garde `'unsafe-inline'`
(chantier séparé). Vérifié : `test_csp_nonce.py` 5/5, 0 handler inline
restant dans les pages HTML (propriétés `onXxx = fn` en JS non concernées
par la CSP).
---
## [2.28.12] — 2026-09-27
### Corrigé
- **BUG-081 — `GET /api/auth/mfa/status` → 500 quand l'auth est désactivée.**
Le pseudo-user `anonymous` (auth désactivée, mode E2E/CI) n'a aucune entrée
en store : `get_user(...)` → `None` puis `AttributeError` sur `user.get`.
Garde `None` → payload « MFA désactivé » (`mfa_enabled: false`,
`totp_enabled: false`, `webauthn_credentials: 0`). Test : `tests/test_mfa.py`
(`TestMfaStatusAuthDisabled`, échoue en 500 sans le correctif).
---
## [2.28.11] — 2026-09-27
---
## [2.28.10] — 2026-09-27
### Corrigé
- **BUG-083 — job CI `security` rouge : le runner tronquait le `#` du `run:` pip-audit.**
Le runner Gitea Act coupe naïvement au premier `#` (même entre
guillemets) : `echo "... see #87)"` devenait une citation non fermée
(`unexpected EOF while looking for matching '"'"`). Seul `run:` du
workflow avec un `#` ; l'echo n'a plus de `#` (réf `#87` en commentaire
YAML, jamais vu par le shell). Garde-fou : `tests/test_ci_workflow.py`
(aucun `#` dans le code des `run:`, `upload.test.mjs` verrouillé dans
l'étape JSDOM — BUG-082).
---
## [2.28.9] — 2026-09-27
### Corrigé
- **BUG-082 — CI `lint` rouge : suites frontend exigeant `jsdom`.**
`tests/frontend/upload.test.mjs` puis `config-ai-keys.test.mjs` (imports
statiques `jsdom`, introduits par `#89`) étaient exécutés dans l'étape
frontend racine où `jsdom` n'est jamais installé (`ERR_MODULE_NOT_FOUND`,
rouge depuis `7bee4a2`). Déplacés dans l'étape JSDOM (les deux branches,
après install si besoin) ; garde-fou `tests/test_ci_workflow.py` :
aucun fichier de l'étape racine ne doit importer `jsdom` statiquement.
---
## [2.28.8] — 2026-09-27
### Corrigé
- **BUG-080 — harnais E2E local anti-blocage (plus de run pendu toute la nuit).**
`run-e2e-local.ps1` : Playwright lancé via `node` direct sur la CLI locale
(jamais de prompt interactif, `Start-Process` ne sachant pas exécuter `npx` ;
paramètre `$Arguments`, `$Args` étant une variable automatique qui l'écraserait),
installation Chromium sautée si déjà présent (`E2E_INSTALL_BROWSERS=1`
pour forcer), étapes `install`/`test` bornées (`E2E_TIMEOUT_SEC`,
défaut 1800 s / 600 s, exit 124 au dépassement — au-delà du globalTimeout
pour un abandon propre avec rapport) ; `run-e2e-local.sh` : `npx --yes` +
mêmes bornes ; `playwright.config.ts` : `globalTimeout` (25 min en local,
30 min en CI, `E2E_GLOBAL_TIMEOUT_MS` pour surcharger) ; `e2e-server.ps1` :
pidfile resynchronisé sur le vrai propriétaire du port et `stop` qui tue
l'arbre complet (fini les serveurs orphelins qui squattent le port 2029).
Garde-fous : `tests/test_e2e_harness.py` (8 tests).
---
## [2.28.7] — 2026-09-26
### Ajouté
- **#87 (T5b) — nonces CSP prêts pour la bascule (sans changement).**
Nonce frais par réponse dans `script-src` (`backend/csp.py`), injecté
dans les 6 pages HTML servies (dont la nouvelle route
`/excalidraw-editor.html`, utilisée par l'iframe du viewer au lieu de
`/static/`) et la page de partage ; `tests/test_csp_nonce.py` (unicité,
concordance en-tête/HTML). `unsafe-inline` conservé jusqu'en T5c.
---
## [2.28.6] — 2026-09-26
### Modifié
- **#87 (T5a) — 16 handlers inline convertis en listeners (CSP inchangée).**
`onclick`/`onerror` de `index.html` et des vues JS (`config`, `plugins`,
`sync`, `viewer`, `auth`) remplacés par `addEventListener` ; suites
frontend vertes (unit, ai, config-mobile, pdf-viewer, mfa-settings,
sidebar-filters).
---
## [2.28.5] — 2026-09-26
---
## [2.28.4] — 2026-09-26
### Ajouté
- **#87 (T4) — E2E XSS et serveur E2E piloté.**
`tests/e2e/xss.spec.js` : page publique `/s/{token}` (titre/frontmatter
hostile échappé, JSON neutralisé, aucun JS exécuté) et lecteur markdown
(sanitizer, aucun `on*`/`javascript:` vivant) — 2/2 verts en local.
`scripts/e2e-server.ps1` : `start|stop|status|logs` avec progression
visible et fichier PID (fini les serveurs orphelins sur le port 2029).
---
## [2.28.3] — 2026-09-26
### Ajouté
- **#87 (T3) — cookies `Secure` et CORS explicites.**
Helper `is_secure_cookies()` centralisé (défaut `false` conservé pour ne
pas casser les logins HTTP locaux) + avertissement au démarrage sur bind
non-loopback sans `Secure` ; `tests/test_security_headers.py` atteste
l'absence de CORS permissif (same-origin par défaut du navigateur) et les
en-têtes de durcissement.
---
## [2.28.2] — 2026-09-26
### Ajouté
- **#87 (T2) — tests de durcissement : concurrence et regex.**
`tests/test_hardening_concurrency.py` : créations/mises à jour/`login
failures` concurrents sur `users.json` (zéro mise à jour perdue, JSON
toujours récupérable) et budget temps de la politique ReDoS (motifs
catastrophiques rejetés en < 1 s, motifs acceptés < 5 s sur 200 Ko).
---
## [2.28.1] — 2026-09-26
### Modifié
- **#87 (T1) — CI sécurité durcie.**
`bandit` passe en bloquant (`# nosec` justifiés : SHA1 non-crypto,
subprocess git à argv fixe, `saxutils.escape` sans parsing ; B105 exclu
comme `pyproject.toml`) ; `npm audit --omit=dev` bloquant (0 faille) ;
les 5 suites frontend hors CI (`upload`, `pretty`, `media-viewer`,
`mfa-settings`, `config-ai-keys`, vertes en local) rejoignent le job
`lint`. `pip-audit` reste consultatif (upgrades starlette/weasyprint à
qualifier, chantier dédié).
---
## [2.28.0] — 2026-09-26
### Modifié
- **#85 (T10) — persistance d'état et clôture de la refonte architecturale.**
Verrous `RLock` sur les stores JSON sans protection (`revoked_tokens`,
`shares`, `webhooks` + secrets, clés d'outils) avec tests de concurrence
(`tests/test_store_locks.py` — pertes prouvées sans verrou) ; rate-limit
auth persisté en option (`OBSIGATE_RATELIMIT_DB`, SQLite WAL, sémantique
identique, défaut mémoire inchangé, `tests/test_ratelimit_store.py`).
Contrat `tools/registry.py` audité (permissions/quotas/redaction déjà
câblés, rien à coder). Index non persisté : rebuild différentiel #86
suffisant (décision documentée). Fiche `docs/features/archi-refonte-85.md`,
#85 sorti du backlog (index roadmap).
## [2.27.12] — 2026-09-26
### Modifié
- **#85 (T9) — extraction realtime + render hors du monolithe `backend/main.py`.**
Le stream SSE `/api/events` et le WebSocket `/ws/collab/*` sont servis par
`backend/routers/realtime.py`, le pipeline markdown (mistune, wikilinks,
slugs, sanitizer) par `backend/render.py` (imports directs, plus de
couplage différé). `main.py` (4 827 → ~760 lignes) ne contient plus que
l'assemblage : lifespan, middlewares, montage des 16 routers, racine
`/api`, statique/SPA et cales de compatibilité testées.
## [2.27.11] — 2026-09-26
### Modifié
- **#85 (T8) — extraction vaults/history/conflicts hors du monolithe `backend/main.py`.**
13 routes servies par `backend/routers/vaults.py`, `history.py` et
`conflicts.py` ; `VaultInfo`/`BookmarkToggleRequest` dans `schemas.py`,
handle watcher partagé dans `backend/watcher_state.py`.
`tests/test_api_main.py` importe `humanize_mtime` depuis son module
canonique (`services.recent`).
## [2.27.10] — 2026-09-26
### Modifié
- **#85 (T7) — extraction du domaine `config` hors du monolithe `backend/main.py`.**
`/api/config`, ai-keys (get/post/delete/test), tool-keys (×3), ai-models,
diagnostics et dashboard sont servis par `backend/routers/config.py`
(`_FALLBACK_MODELS`, store clés et config déplacés ; `main` réimporte
`_load_config` pour son lifespan, les fixtures de tests inchangées).
`tests/test_ai_models.py` patch désormais la référence du router.
## [2.27.9] — 2026-09-26
---
## [2.27.8] — 2026-09-26
---
## [2.27.7] — 2026-09-26
---
## [2.27.6] — 2026-09-26
---
## [2.27.5] — 2026-09-26
---
## [2.27.4] — 2026-09-26
---
## [2.27.3] — 2026-09-26
---
## [2.27.2] — 2026-09-26
### Modifié
- **#85 (T6c) — extraction media/pdf/export/guide hors du monolithe `backend/main.py`.**
@@ -52,12 +485,22 @@ et [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
et vault files sont servis par `backend/routers/files_media.py` ; le helper
Range partagé vit dans `backend/routers/helpers.py` (tags OpenAPI inchangés,
tests statiques frontend `media-viewer`/`image-viewer` réalignés).
## [2.27.8] — 2026-09-26
### Modifié
- **#85 (T6b) — extraction mutations fichiers/dossiers hors du monolithe `backend/main.py`.**
`PUT .../save|xlsx/save`, `DELETE/POST/PATCH /api/file`, `POST/PATCH/DELETE
/api/directory`, `POST /api/move`, `POST .../batch-upload` sont servis par
le nouveau `backend/routers/files_write.py` (effets de bord inchangés :
audit, index, SSE, webhooks, plugins, historique) ; 15 modèles dans
`schemas.py`.
## [2.27.7] — 2026-09-26
### Modifié
- **#85 (T6a) — extraction lecture fichiers hors du monolithe `backend/main.py`.**
`/api/browse/{vault}`, `/api/file/{vault}/raw|download|backlinks` et
`GET /api/file/{vault}` (vue rendue tous formats) sont servis par le
@@ -66,6 +509,11 @@ et [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
`backend/routers/helpers.py` (partagés avec les tranches suivantes).
Correctif au passage : décorateur orphelin `/s/{token}` resté en T3 et
double-enregistrement de `/api/conflicts` supprimés.
## [2.27.6] — 2026-09-26
### Modifié
- **#85 (T5) — extraction du domaine `search` hors du monolithe `backend/main.py`.**
Les 11 routes (`/api/search`, `/advanced`, `/replace`, `/tags`,
`/tree-search`, `/vault/{vault}/paths`, `/suggest`, `/tags/suggest`,
@@ -73,22 +521,42 @@ et [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
par le nouveau `backend/routers/search.py` ; les modèles search dans
`schemas.py` et le pool de threads dans `backend/search_executor.py`
(même dimensionnement, même cycle de vie) — comportement inchangé.
## [2.27.5] — 2026-09-26
### Modifié
- **#85 (T4) — extraction du domaine `backups` hors du monolithe `backend/main.py`.**
Les 9 routes (`/api/file/{vault}/backups|diff|restore`, `/api/backups`,
`/delete`, `/purge`, `/content`, `/compress`, `/auto`) sont servies par le
nouveau `backend/routers/backups.py` ; `Diff/Restore*` déménagent dans
`schemas.py` et le singleton SSE dans `backend/sse.py` (partagé avec
`main`) — comportement inchangé, aucun impact utilisateur.
## [2.27.4] — 2026-09-26
### Modifié
- **#85 (T3) — extraction du domaine `sharing` hors du monolithe `backend/main.py`.**
`POST /api/share/{vault}`, `GET /api/shares`, `DELETE /api/share/{share_id}`
et les pages publiques `/s/{token}`, `/s/{token}/raw`, `/s/{token}/pdf`
sont servis par le nouveau `backend/routers/sharing.py` — chemins,
réponses, tags OpenAPI et authentification inchangés (aucun impact
utilisateur).
## [2.27.3] — 2026-09-26
### Modifié
- **#85 (T2) — extraction du domaine `webhooks` hors du monolithe `backend/main.py`.**
Le CRUD `GET/POST/PATCH/DELETE /api/webhooks` (admin) est servi par le
nouveau `backend/routers/webhooks.py` — chemins, réponses, tags OpenAPI et
authentification inchangés (aucun impact utilisateur).
## [2.27.2] — 2026-09-26
### Modifié
- **#85 (T1) — extraction du domaine `health` hors du monolithe `backend/main.py`.**
`GET /api/health` et `GET /api/health/detailed` (admin) sont servis par le
nouveau `backend/routers/health.py` (monté dans `main.py`) et le modèle
+6 -6
View File
@@ -4,7 +4,7 @@
**Porte d'entrée web ultra-léger pour vos vaults Obsidian** — Accédez, naviguez et recherchez dans toutes vos notes Obsidian depuis n'importe quel appareil via une interface web moderne et responsive.
[![Version](https://img.shields.io/badge/Version-2.27.9-blue.svg)]()
[![Version](https://img.shields.io/badge/Version-2.37.0-blue.svg)]()
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Docker](https://img.shields.io/badge/Docker-Ready-blue.svg)](https://www.docker.com/)
[![Python](https://img.shields.io/badge/Python-3.11+-green.svg)](https://www.python.org/)
@@ -23,7 +23,7 @@ Les **guides d'utilisation** pas à pas se trouvent dans [`docs/GUIDES/`](docs/G
| Guide | Contenu |
|---|---|
| 🚀 [Prise en main](docs/GUIDES/PRISE_EN_MAIN.md) | Premier lancement, interface, navigation, vaults, raccourcis |
| 🔍 [Recherche, PDF & Excalidraw](docs/GUIDES/RECHERCHE_PDF_EXCALIDRAW.md) | Syntaxe de requête, recherche sémantique, lecteur PDF, diagrammes |
| 🔍 [Recherche, PDF, Excel & Excalidraw](docs/GUIDES/RECHERCHE_PDF_EXCALIDRAW.md) | Syntaxe de requête, recherche sémantique, lecteurs PDF/Excel, diagrammes |
| 🤖 [Assistant IA & Forge](docs/GUIDES/ASSISTANT_IA_FORGE.md) | Fournisseurs, éditeur IA, BooksLM, Forge, commandes `@` / `/` |
| 📝 [Édition & collaboration](docs/GUIDES/COLLABORATION.md) | Édition simultanée, curseurs distants, persistance |
| 📱 [PWA & hors-ligne](docs/GUIDES/PWA_HORS_LIGNE.md) | Installation, cache hors-ligne, file de synchro, notifications |
@@ -85,7 +85,7 @@ Les **guides d'utilisation** pas à pas se trouvent dans [`docs/GUIDES/`](docs/G
- **🖼️ Images Obsidian** : Support complet des syntaxes d'images Obsidian avec résolution intelligente
- **🎬 Audio & vidéo** : Lecteurs HTML5 intégrés (`.mp3 .wav .flac .mp4 .webm`…) avec streaming HTTP Range (lecture, déplacement, plein écran) et **lecture persistante** (mini-lecteur flottant / mini-fenêtre vidéo, retour au média ou arrêt à tout moment, contrôles écran verrouillé via Media Session), repli téléchargement si le format n'est pas lisible par le navigateur
- **🎨 Diagrammes Excalidraw** : Visualiseur/éditeur natif des fichiers `.excalidraw` et `.excalidraw.md` (iframe sandboxée, auto-save, thème clair/sombre, texte des diagrammes indexé pour la recherche)
- **📊 Tableurs Excel** : les fichiers `.xlsx` s'ouvrent dans un visualiseur dédié — un tableau par feuille avec onglets, en-têtes A1 et édition directe des cellules (`PUT /api/file/{vault}/xlsx/save`, backup automatique), plus le téléchargement du fichier d'origine
- **📊 Tableurs Excel** : les fichiers `.xlsx` s'ouvrent dans un visualiseur dédié — un tableau par feuille avec onglets, en-têtes A1 et édition directe des cellules (`PUT /api/file/{vault}/xlsx/save`, backup automatique, écriture atomique), plus le téléchargement du fichier d'origine. Les classeurs contenant des éléments qu'ObsiGate ne peut pas conserver (valeurs calculées, segments, contrôles de formulaire, signature…) affichent un **avertissement** et demandent confirmation avant l'enregistrement ; une saisie commençant par `=` ou `@` est stockée comme texte sauf activation du bouton `f(x)`
- **🎨 Syntax highlight** : Coloration syntaxique des blocs de code
- **🌓 Thème clair/sombre** : Toggle persisté en localStorage
- **📡 Synchronisation temps réel** : Surveillance automatique des fichiers via watchdog avec mise à jour incrémentale de l'index
@@ -673,7 +673,7 @@ curl "http://localhost:2020/api/file/Recettes?path=pizza.md"
## 🔍 Recherche avancée
> 📖 Guide complet : [Recherche, PDF & Excalidraw](docs/GUIDES/RECHERCHE_PDF_EXCALIDRAW.md)
> 📖 Guide complet : [Recherche, PDF, Excel & Excalidraw](docs/GUIDES/RECHERCHE_PDF_EXCALIDRAW.md)
### Syntaxe de requête
@@ -976,8 +976,8 @@ Ce projet est sous licence **MIT** — voir le fichier [LICENSE](LICENSE) pour l
## 📝 Changelog
Consultez le [CHANGELOG.md](./CHANGELOG.md) pour l'historique complet de toutes les versions (v1.0.0 → v2.27.9).
Consultez le [CHANGELOG.md](./CHANGELOG.md) pour l'historique complet de toutes les versions (v1.0.0 → v2.37.0).
---
*Projet : ObsiGate | Version : 2.27.9 | Dernière mise à jour : Septembre 2026*
*Projet : ObsiGate | Version : 2.37.0 | Dernière mise à jour : Septembre 2026*
+6 -6
View File
@@ -2,7 +2,7 @@
**Ultra-light web gateway for your Obsidian vaults** — Access, browse, and search all your Obsidian notes from any device via a modern, responsive web interface.
[![Version](https://img.shields.io/badge/Version-2.27.9-blue.svg)]()
[![Version](https://img.shields.io/badge/Version-2.37.0-blue.svg)]()
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Docker](https://img.shields.io/badge/Docker-Ready-blue.svg)](https://www.docker.com/)
[![Python](https://img.shields.io/badge/Python-3.11+-green.svg)](https://www.python.org/)
@@ -21,7 +21,7 @@ Step-by-step **user guides** live in [`docs/GUIDES/`](docs/GUIDES/):
| Guide | What it covers |
|---|---|
| 🚀 [Getting Started](docs/GUIDES/PRISE_EN_MAIN.md) | First run, interface, navigation, vaults, shortcuts |
| 🔍 [Search, PDF & Excalidraw](docs/GUIDES/RECHERCHE_PDF_EXCALIDRAW.md) | Query syntax, semantic search, PDF viewer, diagrams |
| 🔍 [Search, PDF, Excel & Excalidraw](docs/GUIDES/RECHERCHE_PDF_EXCALIDRAW.md) | Query syntax, semantic search, PDF/Excel viewers, diagrams |
| 🤖 [AI Assistant & Forge](docs/GUIDES/ASSISTANT_IA_FORGE.md) | Providers, AI editor, BooksLM, Forge, `@` / `/` commands |
| 📝 [Editing & Collaboration](docs/GUIDES/COLLABORATION.md) | Simultaneous editing, remote cursors, persistence |
| 📱 [PWA & Offline](docs/GUIDES/PWA_HORS_LIGNE.md) | Install as an app, offline cache, sync queue, push |
@@ -84,7 +84,7 @@ Step-by-step **user guides** live in [`docs/GUIDES/`](docs/GUIDES/):
- **🖼️ Obsidian Images** : Full support for all Obsidian image syntaxes with intelligent resolution
- **🎬 Audio & video** : Built-in HTML5 players (`.mp3 .wav .flac .mp4 .webm`…) with HTTP Range streaming (play, seek, fullscreen) and **persistent playback** (floating mini-player / mini video window, return to media or stop anytime, lock-screen controls via Media Session), falling back to download when the format is not playable in the browser
- **🎨 Excalidraw Diagrams** : Native viewer/editor for `.excalidraw` and `.excalidraw.md` files (sandboxed iframe, autosave, dark/light theme, diagram text indexed for search)
- **📊 Excel Spreadsheets** : `.xlsx` files open in a dedicated viewer — one table per sheet with tabs, A1 headers and inline cell editing (`PUT /api/file/{vault}/xlsx/save`, automatic backup), plus download of the original file
- **📊 Excel Spreadsheets** : `.xlsx` files open in a dedicated viewer — one table per sheet with tabs, A1 headers and inline cell editing (`PUT /api/file/{vault}/xlsx/save`, automatic backup, atomic write), plus download of the original file. Workbooks holding elements ObsiGate cannot preserve (cached values, slicers, form controls, signature…) show a **warning** and ask for confirmation before saving; a value starting with `=` or `@` is stored as text unless the `f(x)` toggle is enabled
- **🎨 Syntax Highlight** : Syntax highlighting for code blocks
- **🌓 Light/Dark Theme** : Toggle persisted in localStorage
- **📡 Real-time Sync** : Automatic file monitoring via watchdog with incremental index updates
@@ -804,7 +804,7 @@ curl "http://localhost:2020/api/file/Recipes?path=pizza.md"
## 🔍 Advanced Search
> 📖 Full guide: [Search, PDF & Excalidraw](docs/GUIDES/RECHERCHE_PDF_EXCALIDRAW.md)
> 📖 Full guide: [Search, PDF, Excel & Excalidraw](docs/GUIDES/RECHERCHE_PDF_EXCALIDRAW.md)
### Query Syntax
@@ -1151,8 +1151,8 @@ This project is licensed under the **MIT License** - see the [LICENSE](LICENSE)
## 📝 Changelog
See [CHANGELOG.md](./CHANGELOG.md) for the complete version history (v1.0.0 → v2.27.9).
See [CHANGELOG.md](./CHANGELOG.md) for the complete version history (v1.0.0 → v2.37.0).
---
*Project: ObsiGate | Version: 2.27.9 | Last updated: September 2026*
*Project: ObsiGate | Version: 2.37.0 | Last updated: September 2026*
+1 -1
View File
@@ -1 +1 @@
2.27.9
2.37.0
+33 -26
View File
@@ -119,25 +119,30 @@ def decode_token(token: str) -> dict | None:
_revoked_map: dict[str, int] = {}
_revoked_loaded = False
# ROADMAP #85 T10a — verrou autour du read-modify-write du store de
# révocation (perte de révocations en cas de logouts concurrents).
_revoked_lock = threading.RLock()
def _load_revoked():
"""Load revoked token JTIs from disk into memory (once)."""
global _revoked_loaded, _revoked_map
if _revoked_loaded:
return
if REVOKED_TOKENS_FILE.exists():
try:
data = json.loads(REVOKED_TOKENS_FILE.read_text())
# Drop entries whose underlying token has itself expired.
now = int(time.time())
_revoked_map = {
jti: int(exp) for jti, exp in data.items()
if int(exp) > now
}
except Exception as e:
logger.warning(f"Failed to load revoked tokens: {e}")
_revoked_map = {}
_revoked_loaded = True
with _revoked_lock:
if _revoked_loaded:
return
if REVOKED_TOKENS_FILE.exists():
try:
data = json.loads(REVOKED_TOKENS_FILE.read_text())
# Drop entries whose underlying token has itself expired.
now = int(time.time())
_revoked_map = {
jti: int(exp) for jti, exp in data.items()
if int(exp) > now
}
except Exception as e:
logger.warning(f"Failed to load revoked tokens: {e}")
_revoked_map = {}
_revoked_loaded = True
def _save_revoked():
@@ -154,24 +159,26 @@ def revoke_token(jti: str, expires_at: int | None = None):
``expires_at`` is the revoked token's own ``exp`` (unix seconds) — the
record is kept at least that long so a long-lived API token cannot
outlive its revocation. ``None`` means the token never expires (API/MCP
"sans fin") → the record is kept forever (capped at ~100 years, the JWT
"sans fin") → the record is kept forever (capped at ~100 years, the JWT
store's practical infinity). Default keeps 7 days (session tokens).
"""
_load_revoked()
now = int(time.time())
if expires_at is None:
until = now + 100 * 365 * 24 * 3600
else:
until = max(int(expires_at), now + REFRESH_TOKEN_EXPIRE_SECONDS)
_revoked_map[jti] = until
_save_revoked()
with _revoked_lock:
_load_revoked()
now = int(time.time())
if expires_at is None:
until = now + 100 * 365 * 24 * 3600
else:
until = max(int(expires_at), now + REFRESH_TOKEN_EXPIRE_SECONDS)
_revoked_map[jti] = until
_save_revoked()
logger.debug(f"Revoked token JTI: {jti[:8]}...")
def is_token_revoked(jti: str) -> bool:
"""Check if a token JTI has been revoked."""
_load_revoked()
return jti in _revoked_map
with _revoked_lock:
_load_revoked()
return jti in _revoked_map
# ---------------------------------------------------------------------------
+54 -14
View File
@@ -5,6 +5,7 @@
import base64
import binascii
import logging
import os
import re
from fastapi import APIRouter, Body, Depends, HTTPException, Request, Response
@@ -15,7 +16,7 @@ from backend.ratelimit import record_account_failure as rl_record_account_failur
from backend.ratelimit import record_account_success as rl_record_account_success
from backend.ratelimit import record_failure as rl_record_failure
from backend.ratelimit import record_success as rl_record_success
from backend.services.net import get_client_ip
from backend.services.net import get_client_ip, is_trusted_proxy
from .jwt_handler import (
ACCESS_TOKEN_EXPIRE_SECONDS,
@@ -56,6 +57,34 @@ logger = logging.getLogger("obsigate.auth.router")
router = APIRouter(prefix="/api/auth", tags=["auth"])
def is_secure_cookies(request: Request | None = None) -> bool:
"""True when auth cookies must carry the ``Secure`` flag (#87 T3/T8).
``OBSIGATE_SECURE_COOKIES=true|false|auto`` (défaut : ``auto``) :
``true``/``false`` forcent le comportement ; ``auto`` met ``Secure``
si la requête arrive en https (production derrière TLS) et l'omet
sinon (dev local en http — les navigateurs jettent les cookies
``Secure`` sur http, ce qui casserait silencieusement les logins
localhost). Derrière un reverse proxy qui termine TLS, le schéma perçu
est http : avec ``OBSIGATE_TRUST_PROXY=true``, ``X-Forwarded-Proto``
est honoré (même garde que ``get_client_ip``, BUG-030).
"""
forced = os.environ.get("OBSIGATE_SECURE_COOKIES", "auto").lower()
if forced in ("1", "true", "yes", "on"):
return True
if forced in ("0", "false", "no", "off"):
return False
if request is None:
return False
if request.url.scheme == "https":
return True
if is_trusted_proxy():
proto = request.headers.get("x-forwarded-proto", "").split(",")[0].strip().lower()
if proto == "https":
return True
return False
# ── Pydantic request models ──────────────────────────────────────────
class LoginRequest(BaseModel):
@@ -218,10 +247,11 @@ async def login(body: LoginRequest, response: Response, request: Request):
"remember_me": body.remember_me,
}
return _issue_tokens(user, body.username, body.remember_me, response)
return _issue_tokens(user, body.username, body.remember_me, response, request)
def _issue_tokens(user: dict, username: str, remember_me: bool, response: Response) -> dict:
def _issue_tokens(user: dict, username: str, remember_me: bool, response: Response,
request: Request | None = None) -> dict:
"""Issue JWT tokens after successful authentication (password or MFA verified)."""
record_login_success(username)
rl_record_account_success(username)
@@ -229,9 +259,8 @@ def _issue_tokens(user: dict, username: str, remember_me: bool, response: Respon
access_token = create_access_token(user)
refresh_token, refresh_jti = create_refresh_token(username, remember=remember_me)
import os
max_age = 2592000 if remember_me else 604800 # 30d or 7d
secure = os.environ.get("OBSIGATE_SECURE_COOKIES", "false").lower() == "true"
secure = is_secure_cookies(request)
response.set_cookie(
key="refresh_token",
value=refresh_token,
@@ -253,7 +282,8 @@ def _issue_tokens(user: dict, username: str, remember_me: bool, response: Respon
)
return {
"access_token": access_token,
"token_type": "bearer", # nosec B105 — OAuth2 token_type, pas un mot de passe
# OAuth2 token_type, pas un mot de passe (B105) :
"token_type": "bearer", # nosec B105
"expires_in": ACCESS_TOKEN_EXPIRE_SECONDS,
"user": {
"username": user["username"],
@@ -299,9 +329,7 @@ async def refresh_token_endpoint(request: Request, response: Response):
if stale:
raise HTTPException(401, "Session expirée, veuillez vous reconnecter")
import os
secure = os.environ.get("OBSIGATE_SECURE_COOKIES", "false").lower() == "true"
secure = is_secure_cookies(request)
remember_me = bool(payload.get("remember", False))
# BUG-027: rotate the refresh token — the old one is now single-use.
@@ -332,7 +360,8 @@ async def refresh_token_endpoint(request: Request, response: Response):
return {
"access_token": new_access_token,
"token_type": "bearer", # nosec B105 — OAuth2 token_type, pas un mot de passe
# OAuth2 token_type, pas un mot de passe (B105) :
"token_type": "bearer", # nosec B105
"expires_in": ACCESS_TOKEN_EXPIRE_SECONDS,
}
@@ -426,6 +455,7 @@ async def patch_me(req: UpdateMeRequest, current_user=Depends(require_auth)):
async def change_password(
req: ChangePasswordRequest,
response: Response,
request: Request,
current_user=Depends(require_auth),
):
"""Change own password.
@@ -441,7 +471,7 @@ async def change_password(
updated = get_user(current_user["username"])
result: dict = {"message": "Mot de passe mis à jour"}
if updated is not None:
result.update(_issue_tokens(updated, updated["username"], False, response))
result.update(_issue_tokens(updated, updated["username"], False, response, request))
return result
@@ -804,7 +834,7 @@ async def mfa_webauthn_verify(
rl_record_success(client_ip)
logger.info(f"User '{body.username}' logged in via WebAuthn")
return _issue_tokens(user, body.username, body.remember_me, response)
return _issue_tokens(user, body.username, body.remember_me, response, request)
@router.get("/mfa/status")
@@ -812,6 +842,16 @@ async def mfa_status(current_user=Depends(require_auth)):
"""Return current user's MFA status."""
from .user_store import get_user
user = get_user(current_user["username"])
if user is None:
# BUG-081 : auth désactivée (OBSIGATE_AUTH_ENABLED=false) → le
# pseudo-user "anonymous" n'a aucune entrée en store : pas de MFA,
# et surtout pas de 500 (`AttributeError` sur `user.get`).
return {
"mfa_enabled": False,
"mfa_method": None,
"totp_enabled": False,
"webauthn_credentials": 0,
}
return {
"mfa_enabled": user.get("mfa_enabled", False),
"mfa_method": user.get("mfa_method"),
@@ -847,7 +887,7 @@ async def mfa_totp_verify(body: MfaVerifyRequest, response: Response, request: R
# Clear IP rate limit on success
rl_record_success(client_ip)
return _issue_tokens(user, body.username, body.remember_me, response)
return _issue_tokens(user, body.username, body.remember_me, response, request)
@router.post("/mfa/recovery")
@@ -885,7 +925,7 @@ async def mfa_recovery_login(body: MfaRecoveryRequest, response: Response, reque
rl_record_success(client_ip)
logger.info(f"User '{body.username}' logged in via recovery code")
return _issue_tokens(user, body.username, False, response)
return _issue_tokens(user, body.username, False, response, request)
# ── Admin endpoints ───────────────────────────────────────────────────
+35
View File
@@ -0,0 +1,35 @@
"""Content-Security-Policy nonces (ROADMAP #87, tranche 5b).
Chaque réponse HTTP reçoit un nonce frais (``request.state.csp_nonce``)
injecté dans ``script-src``. Les routes servant du HTML avec des scripts
inline (index, popout, admin, editor-poc, excalidraw, page de partage)
l'injectent dans le balisage via :func:`inject_csp_nonce` — mêmes
emplacements, aucun script déplacé.
Tant que ``'unsafe-inline'`` reste dans la politique (retrait en T5c),
l'injection est inerte : elle prépare la bascule sans changer le
comportement.
"""
from __future__ import annotations
import re
import secrets
# Balises <script> exécutables sans `src` et sans nonce existant :
# `<script>`, `<script type="module">`, `<script type="importmap">`.
# Les blocs non-JS (ex. `type="text/plain"`) et les scripts externes
# (`src=…`, couverts par 'self'/hôtes CDN) sont laissés intacts.
_SCRIPT_TAG_RE = re.compile(
r"<script(?=>|\s+type=\"(?:module|importmap)\"\s*>)",
)
def new_nonce() -> str:
"""Generate a fresh per-response CSP nonce."""
return secrets.token_urlsafe(16)
def inject_csp_nonce(html: str, nonce: str) -> str:
"""Add ``nonce="…"`` to bare executable inline ``<script>`` tags."""
return _SCRIPT_TAG_RE.sub(f'<script nonce="{nonce}"', html)
+4 -1
View File
@@ -23,6 +23,7 @@ import re
import unicodedata
import zipfile
from pathlib import Path
from typing import cast
import frontmatter
import mistune
@@ -246,7 +247,9 @@ def _render_body(md: str, file_dir: Path, vault_path: Path, current: Path) -> st
"""Render raw markdown to an HTML fragment (images inlined, wikilinks resolved)."""
md = _inline_images(md, file_dir, vault_path)
md = _convert_wikilinks(md, vault_path, current)
return _markdown(md)
# mistune 3.3 types `Markdown.__call__` as `str | list[...]` (le renderer
# HTML renvoie toujours `str` à l'exécution).
return cast(str, _markdown(md))
def _build_nav(vault_path: Path, current: Path) -> str:
+2 -1
View File
@@ -35,7 +35,8 @@ def diagram_png_for(code: str) -> Path | None:
Mermaid, ou None. Le hash doit rester synchrone avec le script de build :
sha1(unescape(code).strip())[:16]."""
normalized = html.unescape(code).strip()
sha = hashlib.sha1(normalized.encode("utf-8")).hexdigest()[:16]
# Identifiant de cache déterministe (pas un usage sécurité).
sha = hashlib.sha1(normalized.encode("utf-8")).hexdigest()[:16] # nosec B324
png = DIAGRAMS_DIR / (sha + ".png")
return png if png.exists() else None
+45 -7
View File
@@ -351,6 +351,23 @@ def _decompress_excalidraw(compressed: str) -> dict[str, Any] | None:
return data
def extract_xlsx_indexable(file_path: Path) -> str:
"""Return searchable text for a workbook (#153 A5).
Lazy wrapper: ``openpyxl`` is only imported when a spreadsheet is actually
indexed, so a vault without workbooks never pays the import. Errors are
swallowed — a corrupt or encrypted file still gets indexed by name.
"""
try:
from backend.xlsx_reader import extract_indexable_text
except Exception: # pragma: no cover - openpyxl missing
return ""
try:
return extract_indexable_text(file_path)
except Exception: # pragma: no cover - defensive
return ""
def extract_excalidraw_indexable(raw: str) -> str:
"""Return indexable text content for a raw .excalidraw / .excalidraw.md file.
@@ -561,11 +578,12 @@ def _scan_vault(
title = fpath.stem.replace("-", " ").replace("_", " ")
content_preview = ""
elif ext == ".xlsx":
# #152 — binary workbook: metadata only, the viewer renders
# it (parity with _index_single_file_sync).
raw = ""
# #153 A5 — a workbook stays rendered by the viewer, but its
# cell values are now indexed as text so a spreadsheet is
# findable by its content (parity with _index_single_file_sync).
raw = extract_xlsx_indexable(fpath)
title = fpath.stem.replace("-", " ").replace("_", " ")
content_preview = ""
content_preview = raw[:200].strip()
else:
raw = fpath.read_text(encoding="utf-8", errors="replace")
title = fpath.stem.replace("-", " ").replace("_", " ")
@@ -807,6 +825,13 @@ async def reload_index() -> dict[str, Any]:
await build_index()
# BUG-040/#86: complete the deferred PDF + excalidraw extraction.
await enrich_pdf_texts()
# The inverted index is NOT updated by the hooks here: the rebuild above
# replaces whole vault entries, so the incremental notifications are not
# emitted for the files that only changed content. Without this, a manual
# reindex left TF-IDF search serving a stale index (BUG-089).
from backend.search import init_inverted_index
init_inverted_index()
stats = {}
for name, data in index.items():
stats[name] = {"file_count": len(data["files"]), "tag_count": len(data["tags"])}
@@ -882,6 +907,13 @@ async def reload_single_vault(vault_name: str) -> dict[str, Any]:
# BUG-040/#86: complete the deferred PDF + excalidraw extraction.
await enrich_pdf_texts(vault_name)
# Same as reload_index: the vault entry was replaced wholesale, so rebuild
# the inverted index or TF-IDF search keeps serving stale postings
# (BUG-089).
from backend.search import init_inverted_index
init_inverted_index()
stats = {"file_count": len(vault_data["files"]), "tag_count": len(vault_data["tags"])}
logger.info(f"Vault '{vault_name}' reindexed: {stats['file_count']} files, {stats['tag_count']} tags")
return stats
@@ -955,9 +987,9 @@ def _index_single_file_sync(vault_name: str, vault_path: str, file_path: str, va
raw = ""
content_preview = ""
elif ext == ".xlsx":
# #152 — binary workbook: metadata only (parity with _scan_vault).
raw = ""
content_preview = ""
# #153 A5 — index sheet names + header rows as text (see _scan_vault).
raw = extract_xlsx_indexable(fpath)
content_preview = raw[:200].strip()
else:
raw = fpath.read_text(encoding="utf-8", errors="replace")
content_preview = raw[:200].strip()
@@ -1226,6 +1258,12 @@ async def remove_vault_from_index(vault_name: str):
if not _file_lookup[key]:
_file_lookup.pop(key, None)
# Notify the inverted index, otherwise every document of the vault
# stays in it as a ghost (postings, doc_info, doc_vault, vault_docs)
# and keeps matching searches for a vault that no longer exists.
if _on_index_change:
_on_index_change('remove', vault_name, rel_path, f) # type: ignore[misc]
# Clean path_index
path_index.pop(vault_name, None)
+142 -1118
View File
File diff suppressed because it is too large Load Diff
+2 -1
View File
@@ -32,7 +32,8 @@ def thumb_cache_path(file_path: Path, size: int) -> Path:
stamp = f"{st.st_mtime_ns}:{st.st_size}"
except OSError:
stamp = "0:0"
key = hashlib.sha1(f"{file_path}:{stamp}:{size}".encode()).hexdigest()
# Clé de cache miniature (pas un usage sécurité).
key = hashlib.sha1(f"{file_path}:{stamp}:{size}".encode()).hexdigest() # nosec B324
return thumbs_cache_dir() / f"{key}.webp"
+21 -1
View File
@@ -182,9 +182,29 @@ _ENDPOINT_EXAMPLES: dict[tuple[str, str], dict[str, Any]] = {
"response": {"status": "ok", "vault": "TestVault", "path": "notes/Accueil.md", "size": 26},
},
("put", "/api/file/{vault_name}/xlsx/save"): {
"request": {"sheet": "Budget", "cells": {"B1": "250"}},
"request": {"sheet": "Budget", "cells": {"B1": "250"}, "allow_formula": False, "force": False},
"response": {"status": "ok", "vault": "TestVault", "path": "data/budget.xlsx", "size": 1},
},
# GET : pas d'exemple de requête (un requestBody sur un GET serait un OpenAPI
# invalide) — les paramètres sont documentés par leurs Query().
("get", "/api/file/{vault_name}/xlsx/sheet"): {
"response": {
"vault": "TestVault",
"path": "data/budget.xlsx",
"sheet": "Budget",
"offset": 0,
"limit": 200,
"rows": 2,
"cols": 2,
"total_rows": 640,
"total_cols": 12,
"max_rows": 500,
"max_cols": 40,
"truncated": True,
"has_more": True,
"html": "<table>…</table>",
},
},
("post", "/api/search/replace"): {
"request": {"query": "Python", "replacement": "Python 3", "vault": "all", "dry_run": True},
"response": {"matches": [{"vault": "TestVault", "path": "note1.md", "title": "Python", "match_count": 3}], "total_matches": 3, "dry_run": True},
+172 -1
View File
@@ -12,14 +12,24 @@ the per-account lockout in ``user_store.py``.
deployment, front this service with a shared store (Redis) or a single
worker. This limitation is intentional and documented (BUG-031).
Opt-in persistence (ROADMAP #85 T10b) : if ``OBSIGATE_RATELIMIT_DB`` points
to a SQLite file, counters are stored there instead (WAL mode, one short
connection per call — safe across threads, processes and restarts sharing
the same file). Semantics (windows, budgets, success reset) are identical
to the in-memory store, which remains the default when the variable is
unset.
Configuration via environment variables:
OBSIGATE_LOGIN_MAX_ATTEMPTS Max failures per IP (default: 10)
OBSIGATE_ACCOUNT_MAX_ATTEMPTS Max failures per account (default: 10)
OBSIGATE_LOGIN_WINDOW_SECONDS Lockout window in seconds (default: 900)
OBSIGATE_RATELIMIT_DB SQLite file for shared/persistent counters (default: unset = memory)
"""
import logging
import os
import sqlite3
import threading
import time
from collections import defaultdict
@@ -37,6 +47,127 @@ _last_cleanup = time.time()
CLEANUP_INTERVAL = 60 # seconds
def _db_path() -> str | None:
"""SQLite file for shared counters, or ``None`` for the in-memory store."""
path = os.environ.get("OBSIGATE_RATELIMIT_DB", "").strip()
return path or None
def _db_connect(path: str) -> sqlite3.Connection:
"""Open a short-lived connection (WAL + busy timeout for concurrent workers)."""
_db_ensure_schema(path)
conn = sqlite3.connect(path, timeout=10.0)
conn.execute("PRAGMA busy_timeout=10000")
return conn
_schema_ready: set[str] = set()
_schema_lock = threading.Lock()
def _db_ensure_schema(path: str) -> None:
"""Create the store schema once per file (DDL under a process-wide lock)."""
with _schema_lock:
if path in _schema_ready:
return
conn = sqlite3.connect(path, timeout=10.0)
try:
conn.execute("PRAGMA journal_mode=WAL")
conn.execute(
"CREATE TABLE IF NOT EXISTS attempts"
" (kind TEXT NOT NULL, key TEXT NOT NULL, ts REAL NOT NULL, success INTEGER NOT NULL)"
)
conn.execute(
"CREATE INDEX IF NOT EXISTS idx_attempts_kind_key_ts"
" ON attempts (kind, key, ts)"
)
conn.commit()
finally:
conn.close()
_schema_ready.add(path)
def _db_write(fn, *args):
"""Run a write op, retrying once on lock contention (concurrent workers)."""
try:
return fn(*args)
except sqlite3.OperationalError as e:
if "locked" not in str(e).lower():
raise
time.sleep(0.05)
return fn(*args)
def _db_prune(conn: sqlite3.Connection, cutoff: float) -> None:
"""Drop expired entries (best-effort cap on disk growth)."""
conn.execute("DELETE FROM attempts WHERE ts <= ?", (cutoff,))
def _db_record(kind: str, key: str, success: bool) -> int:
"""Record one attempt in SQLite; return the live failure count."""
path = _db_path()
assert path is not None
now = time.time()
cutoff = now - WINDOW_SECONDS
def _write() -> int:
with _db_connect(path) as conn:
_db_prune(conn, cutoff)
if success:
# Mirror the in-memory reset: replace history with one success.
conn.execute("DELETE FROM attempts WHERE kind = ? AND key = ?", (kind, key))
conn.execute(
"INSERT INTO attempts (kind, key, ts, success) VALUES (?, ?, ?, ?)",
(kind, key, now, int(success)),
)
conn.commit()
(failures,) = conn.execute(
"SELECT COUNT(*) FROM attempts WHERE kind = ? AND key = ? AND ts > ? AND success = 0",
(kind, key, cutoff),
).fetchone()
return failures
return _db_write(_write)
def _db_failures(kind: str, key: str) -> int:
"""Live failure count in SQLite (expired entries never count)."""
path = _db_path()
assert path is not None
cutoff = time.time() - WINDOW_SECONDS
with _db_connect(path) as conn:
(failures,) = conn.execute(
"SELECT COUNT(*) FROM attempts WHERE kind = ? AND key = ? AND ts > ? AND success = 0",
(kind, key, cutoff),
).fetchone()
return failures
def _db_tracked(kind: str) -> int:
"""Number of distinct keys ever seen for one budget (SQLite)."""
path = _db_path()
assert path is not None
with _db_connect(path) as conn:
(n,) = conn.execute(
"SELECT COUNT(DISTINCT key) FROM attempts WHERE kind = ?", (kind,)
).fetchone()
return n
def _db_limited_count(kind: str, max_attempts: int) -> int:
"""Number of keys currently over budget (SQLite)."""
path = _db_path()
assert path is not None
cutoff = time.time() - WINDOW_SECONDS
with _db_connect(path) as conn:
rows = conn.execute(
"SELECT key, COUNT(*) FROM attempts"
" WHERE kind = ? AND ts > ? AND success = 0 GROUP BY key",
(kind, cutoff),
).fetchall()
return sum(1 for _, n in rows if n >= max_attempts)
def _prune(store: dict[str, list], cutoff: float) -> None:
"""Drop expired entries from one store in place."""
expired = []
@@ -66,6 +197,12 @@ def record_failure(ip: str) -> tuple[int, int]:
Returns:
(current_failure_count, remaining_attempts)
"""
if _db_path() is not None:
failures = _db_record("ip", ip, False)
remaining = max(0, MAX_ATTEMPTS - failures)
if failures >= MAX_ATTEMPTS:
logger.warning(f"IP {ip} rate-limited after {failures} failed logins")
return failures, remaining
_cleanup_expired()
_ip_attempts[ip].append((time.time(), False))
failures = sum(1 for _, success in _ip_attempts[ip] if not success)
@@ -77,12 +214,17 @@ def record_failure(ip: str) -> tuple[int, int]:
def record_success(ip: str):
"""Clear rate limit state for an IP after successful login."""
if _db_path() is not None:
_db_record("ip", ip, True)
return
_cleanup_expired()
_ip_attempts[ip] = [(time.time(), True)]
def is_rate_limited(ip: str) -> bool:
"""Check if an IP has exceeded the rate limit."""
if _db_path() is not None:
return _db_failures("ip", ip) >= MAX_ATTEMPTS
_cleanup_expired()
failures = sum(1 for _, success in _ip_attempts.get(ip, []) if not success)
return failures >= MAX_ATTEMPTS
@@ -94,8 +236,14 @@ def record_account_failure(account: str) -> tuple[int, int]:
Returns:
(current_failure_count, remaining_attempts)
"""
_cleanup_expired()
key = account.lower()
if _db_path() is not None:
failures = _db_record("account", key, False)
remaining = max(0, ACCOUNT_MAX_ATTEMPTS - failures)
if failures >= ACCOUNT_MAX_ATTEMPTS:
logger.warning(f"Account {account} rate-limited after {failures} failed attempts")
return failures, remaining
_cleanup_expired()
_account_attempts[key].append((time.time(), False))
failures = sum(1 for _, success in _account_attempts[key] if not success)
remaining = max(0, ACCOUNT_MAX_ATTEMPTS - failures)
@@ -106,12 +254,17 @@ def record_account_failure(account: str) -> tuple[int, int]:
def record_account_success(account: str):
"""Clear the per-account rate limit state after a successful login."""
if _db_path() is not None:
_db_record("account", account.lower(), True)
return
_cleanup_expired()
_account_attempts[account.lower()] = [(time.time(), True)]
def is_account_rate_limited(account: str) -> bool:
"""Check if an account has exceeded the per-account rate limit."""
if _db_path() is not None:
return _db_failures("account", account.lower()) >= ACCOUNT_MAX_ATTEMPTS
_cleanup_expired()
failures = sum(
1 for _, success in _account_attempts.get(account.lower(), []) if not success
@@ -121,6 +274,24 @@ def is_account_rate_limited(account: str) -> bool:
def get_status(ip: str | None = None) -> dict:
"""Get rate limit status for an IP (for diagnostics)."""
if _db_path() is not None:
if ip:
failures = _db_failures("ip", ip)
return {
"ip": ip,
"failures": failures,
"max": MAX_ATTEMPTS,
"limited": failures >= MAX_ATTEMPTS,
"window_seconds": WINDOW_SECONDS,
}
return {
"tracked_ips": _db_tracked("ip"),
"tracked_accounts": _db_tracked("account"),
"max_attempts": MAX_ATTEMPTS,
"account_max_attempts": ACCOUNT_MAX_ATTEMPTS,
"window_seconds": WINDOW_SECONDS,
"limited_ips": _db_limited_count("ip", MAX_ATTEMPTS),
}
_cleanup_expired()
if ip:
attempts = _ip_attempts.get(ip, [])
+210
View File
@@ -0,0 +1,210 @@
"""Markdown rendering pipeline (ROADMAP #85, tranche 9).
Helpers extraits de :mod:`backend.main` sans changement de comportement :
slugification des headings, IDs d'ancrage, rendu mistune singleton,
wikilinks, normalisation des sauts de ligne et pipeline complet
:func:`_render_markdown` (rendu + sanitizer XSS BUG-021).
Les noms gardent leur préfixe ``_`` d'origine pour un déplacement
strictement verbatim (tests et routers pointent ici désormais).
"""
from __future__ import annotations
import html as html_mod
import re
import unicodedata
from pathlib import Path
from typing import cast
import mistune
from backend.image_processor import preprocess_images
from backend.indexer import find_file_in_index, get_vault_data
from backend.secret_redactor import redact_file_content
from backend.services.sanitizer import sanitize_html
def _heading_slugify(text: str) -> str:
"""Generate a URL-safe slug from heading text.
Matches the JavaScript slugify algorithm exactly using
Unicode-aware character classification:
1. Strip HTML tags (e.g. wikilink spans rendered inside headings)
2. Decode HTML entities (e.g. ``&amp;`` → ``&``)
3. Lowercase
4. NFD normalize + strip combining marks
5. Keep only Unicode letters, numbers, spaces, hyphens
6. Replace spaces with hyphens, collapse multiple hyphens
Args:
text: The heading text content (may contain inline HTML).
Returns:
A URL-safe slug string.
"""
# Strip any inline HTML so it does not pollute the slug
text = re.sub(r"<[^>]+>", "", text)
# Decode HTML entities so &amp; becomes & before slugification
text = html_mod.unescape(text)
text = text.lower()
text = unicodedata.normalize("NFD", text)
text = "".join(ch for ch in text if not unicodedata.combining(ch))
# Unicode-aware: keep letters (L*), numbers (N*), spaces, and hyphens
cleaned = []
for ch in text:
cat = unicodedata.category(ch)
if cat.startswith('L') or cat.startswith('N') or ch in (' ', '-'):
cleaned.append(ch)
text = "".join(cleaned)
text = re.sub(r"\s+", "-", text)
text = re.sub(r"-+", "-", text)
result = text.strip("-")
return result if result else "heading"
def _add_heading_ids(html: str) -> str:
"""Post-process rendered HTML to add IDs to heading tags.
Adds an ``id`` attribute to every ``<h1>`` through ``<h6>`` tag
using a slug generated from the heading's text content.
Duplicate slugs get a ``-2``, ``-3``, etc. suffix.
Args:
html: Rendered HTML string.
Returns:
HTML with heading IDs injected.
"""
used_ids: dict[str, int] = {}
def _replace_heading(match):
tag = match.group(1)
content = match.group(2)
slug = _heading_slugify(content)
count = used_ids.get(slug, 0)
used_ids[slug] = count + 1
if count > 0:
slug = f"{slug}-{count + 1}"
return f'<{tag} id="{slug}">{content}</{tag}>'
# Match h1-h6 tags with text content (no existing id attribute)
return re.sub(
r'<(h[1-6])>([^<]*(?:<(?!/?h[1-6])[^<]*)*)</h[1-6]>',
_replace_heading,
html,
)
# Cached mistune renderer — avoids re-creating on every request
_markdown_renderer = mistune.create_markdown(
escape=False,
plugins=["table", "strikethrough", "footnotes", "task_lists"],
)
def _convert_wikilinks(content: str, current_vault: str) -> str:
"""Convert ``[[wikilinks]]`` and ``[[target|display]]`` to clickable HTML.
Supports:
- Internal file links: ``[[My Note]]`` / ``[[My Note|display]]``
- Same-document anchors: ``[[#Heading]]`` / ``[[#Heading|display]]``
Resolved file links get a ``data-vault`` / ``data-path`` attribute pair.
Anchor links target the slugified heading ID in the current document.
Unresolved links are rendered as ``<span class="wikilink-missing">``.
Args:
content: Markdown string potentially containing wikilinks.
current_vault: Active vault name for resolution priority.
Returns:
Markdown string with wikilinks replaced by HTML anchors.
"""
def _replace(match):
target = match.group(1).strip()
display = match.group(2).strip() if match.group(2) else target
# Same-document anchor link: [[#Heading|display]]
if target.startswith("#"):
anchor_text = target[1:].strip()
anchor_slug = _heading_slugify(anchor_text)
link_display = display if display != target else anchor_text
return f'<a class="wikilink-anchor" href="#{anchor_slug}">{link_display}</a>'
found = find_file_in_index(target, current_vault)
if found:
return (
f'<a class="wikilink" href="#" '
f'data-vault="{found["vault"]}" '
f'data-path="{found["path"]}">{display}</a>'
)
return f'<span class="wikilink-missing">{display}</span>'
pattern = r'\[\[([^\]|]+)(?:\|([^\]]+))?\]\]'
return re.sub(pattern, _replace, content)
def _normalize_line_breaks(text: str) -> str:
"""Convert single newlines to hard breaks (matching Obsidian default behavior).
In standard Markdown, a single ``\\n`` is a "soft break" — it renders as a space,
not a visible line break. Obsidian defaults to treating single newlines as hard
breaks (equivalent to ``<br>``). This function pre-processes the Markdown source
so that mistune renders standalone lines on separate rows, while still honouring
blank lines as paragraph separators.
Fenced code blocks (`` ``` ``) are left untouched so their internal newlines are
preserved verbatim.
"""
parts = re.split(r"(```[\s\S]*?```)", text)
for i, part in enumerate(parts):
if part.startswith("```"):
continue # Protect fenced code blocks
# Single \n (not preceded or followed by another \n) → two spaces + \n
parts[i] = re.sub(r"(?<!\n)\n(?!\n)", " \n", part)
return "".join(parts)
def _render_markdown(raw_md: str, vault_name: str, current_file_path: Path | None = None) -> str:
"""Render a markdown string to HTML with wikilink and image support.
Uses the cached singleton mistune renderer for performance.
Args:
raw_md: Raw markdown text (frontmatter already stripped).
vault_name: Current vault for wikilink resolution context.
current_file_path: Absolute path to the current markdown file.
Returns:
HTML string.
"""
# Get vault data for image resolution
vault_data = get_vault_data(vault_name)
vault_root = Path(vault_data["path"]) if vault_data else None
attachments_path = vault_data.get("config", {}).get("attachmentsPath") if vault_data else None
# Redact secrets before rendering (P0 security)
raw_md = redact_file_content(raw_md, str(current_file_path) if current_file_path else "")
# Preprocess images first
if vault_root:
raw_md = preprocess_images(raw_md, vault_name, vault_root, current_file_path, attachments_path)
# Convert wikilinks
converted = _convert_wikilinks(raw_md, vault_name)
# Normalize line breaks to match Obsidian behavior (single \n → hard break)
converted = _normalize_line_breaks(converted)
# mistune 3.3 types `Markdown.__call__` as `str | list[...]` (les
# renderers HTML renvoient toujours `str` à l'exécution).
rendered = cast(str, _markdown_renderer(converted))
# Add heading IDs for TOC navigation
rendered = _add_heading_ids(rendered)
# Sanitize: raw HTML in vault content must never reach the DOM (BUG-021).
rendered = sanitize_html(rendered)
return rendered
+6 -6
View File
@@ -1,9 +1,9 @@
fastapi==0.110.3
uvicorn==0.30.0
fastapi==0.141.1
uvicorn==0.54.0
websockets>=12.0
python-frontmatter==1.1.0
mistune==3.0.2
python-multipart==0.0.9
mistune==3.3.3
python-multipart==0.0.31
aiofiles==23.2.1
aiohttp>=3.9.0
watchdog>=4.0.0
@@ -11,7 +11,7 @@ argon2-cffi>=23.1.0
python-jose>=3.3.0
sortedcontainers>=2.4.0
snowballstemmer>=2.2.0
weasyprint>=60.0
weasyprint>=70.0
httpx>=0.27.0
pypdf>=4.0
pyotp>=2.10.0
@@ -19,7 +19,7 @@ segno>=1.5.0
webauthn==2.6.0
psutil>=5.9
pywebpush>=2.3.0
mcp==1.9.4
mcp==1.28.1
sse-starlette==2.1.3
openpyxl>=3.1
python-docx>=1.1
+531
View File
@@ -0,0 +1,531 @@
"""Configuration, AI keys, diagnostics & dashboard endpoints (ROADMAP #85, tranche 7).
Handlers déplacés depuis :mod:`backend.main` sans changement de
comportement : mêmes chemins (``/api/config*``, ``/api/diagnostics``,
``/api/dashboard``), mêmes modèles de réponse, mêmes dépendances
d'authentification.
Adaptations strictement équivalentes :
- ``_load_config`` / ``_save_config`` / ``_DEFAULT_CONFIG`` /
``_CONFIG_PATH`` / ``_BASE_DIR`` ont déménagé ici : ``main`` les
réimporte pour son lifespan (pas de cycle : ce module ne dépend pas de
``main``).
- ``AI_KEYS_FILE`` / ``_write_ai_keys`` / ``_FALLBACK_MODELS`` ont déménagé
ici (``AI_KEYS_FILE`` garde son chemin relatif ``data/api_keys.json``,
résolu depuis le même CWD au runtime).
"""
import json as _json
import logging
import os
import urllib.request
from pathlib import Path
from fastapi import APIRouter, Body, Depends, HTTPException, Query
from backend.ai import PROVIDERS, _read_ai_keys, get_ai_key
from backend.auth.middleware import require_admin, require_auth
from backend.indexer import index
from backend.media_types import IMAGE_EXTENSIONS
from backend.schemas import (
AIKeyDeleteResponse,
AIKeysResponse,
AIModelsResponse,
AITestResponse,
AppConfigResponse,
DashboardResponse,
DiagnosticsResponse,
StatusResponse,
)
from backend.search_executor import get_search_executor
from backend.tools.secrets import (
TOOL_KEY_NAMES as _TOOL_KEY_NAMES,
)
from backend.tools.secrets import (
delete_tool_key as _delete_tool_key,
)
from backend.tools.secrets import (
get_tool_key as _get_tool_key,
)
from backend.tools.secrets import (
mask_value as _mask_tool_value,
)
from backend.tools.secrets import (
set_tool_key as _set_tool_key,
)
logger = logging.getLogger("obsigate")
router = APIRouter(tags=["System"])
_BASE_DIR = Path(__file__).resolve().parent.parent.parent
_CONFIG_PATH = _BASE_DIR / "data" / "config.json"
_DEFAULT_CONFIG = {
"search_workers": 2,
"debounce_ms": 300,
"results_per_page": 50,
"min_query_length": 2,
"search_timeout_ms": 30000,
"max_content_size": 100000,
"snippet_context_chars": 120,
"max_snippet_highlights": 5,
"title_boost": 3.0,
"path_boost": 1.5,
"watcher_enabled": True,
"watcher_use_polling": False,
"watcher_polling_interval": 5.0,
"watcher_debounce": 2.0,
"tag_boost": 2.0,
"prefix_max_expansions": 50,
"recent_files_limit": 20,
"max_backups_per_file": 10,
"ai_default_provider": "deepseek",
"ai_default_models": {},
}
def _load_config() -> dict:
"""Load config from disk, merging with defaults."""
config = dict(_DEFAULT_CONFIG)
if _CONFIG_PATH.exists():
try:
stored = _json.loads(_CONFIG_PATH.read_text(encoding="utf-8"))
config.update(stored)
except Exception as e:
logger.warning(f"Failed to read config.json: {e}")
return config
def _save_config(config: dict) -> None:
"""Persist config to disk."""
try:
_CONFIG_PATH.write_text(
_json.dumps(config, indent=2, ensure_ascii=False),
encoding="utf-8",
)
except Exception as e:
logger.error(f"Failed to write config.json: {e}")
raise HTTPException(status_code=500, detail=f"Failed to save config: {e}")
AI_KEYS_FILE = Path("data/api_keys.json")
def _write_ai_keys(data: dict):
AI_KEYS_FILE.parent.mkdir(parents=True, exist_ok=True)
tmp = AI_KEYS_FILE.with_suffix(".tmp")
tmp.write_text(_json.dumps(data, indent=2), encoding="utf-8")
tmp.replace(AI_KEYS_FILE)
@router.get("/api/config", response_model=AppConfigResponse)
async def api_get_config(current_user=Depends(require_auth)):
"""Return current configuration with defaults for missing keys."""
return _load_config()
@router.post("/api/config", response_model=AppConfigResponse)
async def api_set_config(body: dict = Body(...), current_user=Depends(require_admin)):
"""Update configuration. Only known keys are accepted.
Keys matching ``_DEFAULT_CONFIG`` are validated and persisted.
Unknown keys are silently ignored.
Returns the full merged config after update.
"""
current = _load_config()
updated_keys = []
for key, value in body.items():
if key in _DEFAULT_CONFIG:
expected_type = type(_DEFAULT_CONFIG[key])
if isinstance(value, expected_type) or (expected_type is float and isinstance(value, (int, float))):
current[key] = value
updated_keys.append(key)
else:
raise HTTPException(
status_code=400,
detail=f"Invalid type for '{key}': expected {expected_type.__name__}, got {type(value).__name__}",
)
_save_config(current)
if any(k.startswith("ai_") for k in updated_keys):
try:
from backend.ai import reload_ai_config
reload_ai_config()
except Exception as e:
logger.warning(f"Failed to reload AI config: {e}")
logger.info(f"Config updated: {updated_keys}")
return current
@router.get("/api/config/ai-keys", response_model=AIKeysResponse)
async def api_get_ai_keys(current_user=Depends(require_admin)):
"""Return stored AI keys (values masked)."""
keys = _read_ai_keys()
masked = {}
for k in ["DEEPSEEK_API_KEY", "OPENROUTER_API_KEY", "GEMINI_API_KEY", "NVIDIA_API_KEY", "QWENCLOUD_API_KEY", "XIAOMI_API_KEY", "MISTRAL_API_KEY"]:
val = keys.get(k, "") or os.environ.get(k, "")
if val:
masked[k] = val[:4] + "..." + val[-4:] if len(val) > 8 else "***"
else:
masked[k] = ""
return masked
@router.post("/api/config/ai-keys", response_model=StatusResponse)
async def api_set_ai_keys(body: dict = Body(...), current_user=Depends(require_admin)):
"""Save AI keys. Pass {"DEEPSEEK_API_KEY":"sk-...","OPENROUTER_API_KEY":"...","GEMINI_API_KEY":"..."}"""
keys = _read_ai_keys()
for k in ["DEEPSEEK_API_KEY", "OPENROUTER_API_KEY", "GEMINI_API_KEY", "NVIDIA_API_KEY", "QWENCLOUD_API_KEY", "XIAOMI_API_KEY", "MISTRAL_API_KEY"]:
if body.get(k):
keys[k] = body[k]
_write_ai_keys(keys)
logger.info("AI keys updated")
return {"status": "ok"}
@router.delete("/api/config/ai-keys/{provider_env}", response_model=AIKeyDeleteResponse)
async def api_delete_ai_key(provider_env: str, current_user=Depends(require_admin)):
"""Delete a specific AI provider key from storage."""
allowed = {"DEEPSEEK_API_KEY", "OPENROUTER_API_KEY", "GEMINI_API_KEY",
"NVIDIA_API_KEY", "QWENCLOUD_API_KEY", "XIAOMI_API_KEY", "MISTRAL_API_KEY"}
key_name = provider_env.upper()
if key_name not in allowed:
raise HTTPException(status_code=400, detail=f"Clé inconnue: {provider_env}")
keys = _read_ai_keys()
if key_name in keys:
del keys[key_name]
_write_ai_keys(keys)
# Also clear from env at runtime so get_ai_key() no longer finds it
os.environ.pop(key_name, None)
logger.info(f"AI key deleted: {key_name}")
return {"status": "deleted", "key": key_name}
@router.get("/api/config/tool-keys", response_model=AIKeysResponse)
async def api_get_tool_keys(current_user=Depends(require_admin)):
"""Return tool/connected-source configuration (tokens masked, URLs clear)."""
masked = {}
for name in _TOOL_KEY_NAMES:
masked[name] = _mask_tool_value(name, _get_tool_key(name))
return masked
@router.post("/api/config/tool-keys", response_model=StatusResponse)
async def api_set_tool_keys(body: dict = Body(...), current_user=Depends(require_admin)):
"""Save tool/connected-source keys.
Only whitelisted names (``backend.tools.secrets.TOOL_KEY_NAMES``) are
accepted: Tavily/Brave/SerpAPI/Exa API keys, Gitea URL + token, GitHub
token. Empty values delete the stored entry.
"""
updated = []
for name, value in body.items():
if name not in _TOOL_KEY_NAMES:
raise HTTPException(status_code=400, detail=f"Clé inconnue: {name}")
if value is not None and not isinstance(value, str):
raise HTTPException(status_code=400, detail=f"Type invalide pour {name}")
_set_tool_key(name, value or "")
updated.append(name)
logger.info(f"Tool keys updated: {updated}")
return {"status": "ok"}
@router.delete("/api/config/tool-keys/{name}", response_model=AIKeyDeleteResponse)
async def api_delete_tool_key(name: str, current_user=Depends(require_admin)):
"""Delete a stored tool key (the environment fallback still applies)."""
key_name = name.upper()
try:
existed = _delete_tool_key(key_name)
except ValueError as e:
raise HTTPException(status_code=400, detail=str(e))
logger.info(f"Tool key deleted: {key_name} (existed={existed})")
return {"status": "deleted", "key": key_name}
@router.post("/api/config/ai-keys/test", response_model=AITestResponse)
async def api_test_ai_keys(current_user=Depends(require_admin)):
"""Test which AI providers are configured.
Each provider has a dedicated (URL, header-name) test pair.
- Most OpenAI-compatible APIs use `Authorization: Bearer KEY`
- Xiaomi MiMo uses `api-key: KEY`
- Gemini uses a query-string key
"""
results = {}
for key_name, label, test_url_tmpl, header_name in [
# OpenAI-compatible — Authorization: Bearer
("DEEPSEEK_API_KEY", "deepseek", "https://api.deepseek.com/v1/models", "Authorization"),
("OPENROUTER_API_KEY","openrouter", "https://openrouter.ai/api/v1/models", "Authorization"),
("NVIDIA_API_KEY", "nvidia", "https://integrate.api.nvidia.com/v1/models", "Authorization"),
("QWENCLOUD_API_KEY", "qwencloud", "https://dashscope.aliyuncs.com/compatible-mode/v1/models", "Authorization"),
("MISTRAL_API_KEY", "mistral", "https://api.mistral.ai/v1/models", "Authorization"),
# Xiaomi MiMo — dedicated api-key header (NOT Authorization: Bearer)
("XIAOMI_API_KEY", "xiaomi", "https://api.xiaomimimo.com/v1/models", "api-key"),
# Gemini — key in query string
("GEMINI_API_KEY", "gemini", "https://generativelanguage.googleapis.com/v1beta/models?key={key}", None),
]:
key = get_ai_key(key_name)
if not key:
results[label] = "non configuré"
continue
try:
url = test_url_tmpl.replace("{key}", key) if "{key}" in test_url_tmpl else test_url_tmpl
if header_name:
req = urllib.request.Request(url, headers={header_name: key})
else:
req = urllib.request.Request(url)
urllib.request.urlopen(req, timeout=5)
results[label] = "ok"
except Exception as e:
# Truncate the error to keep the response small.
results[label] = "erreur: " + str(e)[:80]
return results
@router.get("/api/config/ai-models", response_model=AIModelsResponse)
async def api_list_ai_models(provider: str = Query(...), current_user=Depends(require_admin)):
"""List available models for a given AI provider.
Strategy:
1. Try the provider's public models endpoint (OpenAI-compatible /v1/models or Gemini).
2. If the network call fails (timeout, 4xx, 5xx, DNS, etc.), fall back to a
curated static list of known-good models for that provider.
3. Always return a non-empty list when the provider is known, so the UI
dropdown is never empty.
"""
provider = provider.lower()
from backend.model_capabilities import get_capabilities_for_models
from backend.provider_capabilities import remember_declared_capabilities
all_providers = ("deepseek", "openrouter", "gemini", "nvidia", "qwencloud", "xiaomi", "mistral")
if provider not in all_providers:
return {"models": [], "error": f"Unknown provider: {provider}", "source": "validation"}
key_name = f"{provider.upper()}_API_KEY"
key = get_ai_key(key_name)
if not key:
# No key configured — return curated fallback list so the UI can
# still show what WOULD be available once a key is set.
fallback = _FALLBACK_MODELS.get(provider, [])
return {"models": fallback, "source": "fallback",
"capabilities": get_capabilities_for_models(provider, fallback),
"note": "API key not configured — showing default model list"}
# Build URL
if provider == "gemini":
url = f"https://generativelanguage.googleapis.com/v1beta/models?key={key}"
elif provider == "deepseek":
url = "https://api.deepseek.com/v1/models"
elif provider == "openrouter":
url = "https://openrouter.ai/api/v1/models"
elif provider == "nvidia":
url = "https://integrate.api.nvidia.com/v1/models"
elif provider == "qwencloud":
url = "https://dashscope.aliyuncs.com/compatible-mode/v1/models"
elif provider == "xiaomi":
# Xiaomi MiMo — dedicated api-key header (NOT Authorization: Bearer).
# Endpoint: https://api.xiaomimimo.com/v1/models
url = "https://api.xiaomimimo.com/v1/models"
models = [] # parsed below with the custom header
elif provider == "mistral":
url = "https://api.mistral.ai/v1/models"
try:
if provider == "gemini":
req = urllib.request.Request(url)
elif provider == "xiaomi":
# Xiaomi MiMo uses a dedicated api-key header.
req = urllib.request.Request(url, headers={"api-key": key})
else:
req = urllib.request.Request(url, headers={"Authorization": "Bearer " + key})
with urllib.request.urlopen(req, timeout=10) as resp:
data = _json.loads(resp.read().decode())
if provider == "gemini":
models = [m.get("name", "") for m in data.get("models", []) if m.get("name")]
# Gemini returns names like "models/gemini-1.5-flash" — strip prefix
models = [m.replace("models/", "") for m in models]
else:
models = [m.get("id", "") for m in data.get("data", []) if m.get("id")]
# Cache the capabilities the provider declares for these models
# (BUG-044) — get_capabilities_for_models() below then returns the
# provider's own truth for the flags it declares, the curated table
# for the rest. Providers that declare nothing are left untouched.
remember_declared_capabilities(provider, data)
if models:
# Prepend the configured default if not already present
default = PROVIDERS.get(provider, {}).get("model")
if default and default not in models:
models = [default] + models
return {"models": models, "source": "live", "count": len(models),
"capabilities": get_capabilities_for_models(provider, models)}
# Empty list from API — fall through to fallback
raise ValueError("empty model list from provider API")
except Exception as e:
# Network error, auth error, parsing error — use curated fallback
fallback = _FALLBACK_MODELS.get(provider, [])
return {"models": fallback, "source": "fallback", "error": str(e)[:200],
"capabilities": get_capabilities_for_models(provider, fallback),
"note": "Could not reach provider API — showing default model list"}
# ── Curated fallback model lists ──────────────────────────────────────────
# Used when the provider API is unreachable or returns empty.
# Keep these short and focused on models known to work with the
# OpenAI-compatible chat completions interface (or Gemini's generateContent).
_FALLBACK_MODELS: dict[str, list[str]] = {
"deepseek": [
"deepseek-chat",
"deepseek-reasoner",
],
"openrouter": [
"openai/gpt-4o-mini",
"openai/gpt-4o",
"anthropic/claude-3.5-sonnet",
"anthropic/claude-3-haiku",
"google/gemini-2.0-flash-exp:free",
"meta-llama/llama-3.1-70b-instruct",
"meta-llama/llama-3.1-8b-instruct:free",
"mistralai/mistral-large-latest",
],
"gemini": [
"gemini-2.0-flash",
"gemini-2.0-flash-exp",
"gemini-1.5-pro",
"gemini-1.5-flash",
"gemini-1.5-flash-8b",
],
"nvidia": [
"meta/llama-3.1-405b-instruct",
"meta/llama-3.1-70b-instruct",
"meta/llama-3.1-8b-instruct",
"mistralai/mistral-large",
"google/gemma-2-27b-it",
"nvidia/llama-3.1-nemotron-70b-instruct",
],
"qwencloud": [
"qwen-max",
"qwen-plus",
"qwen-turbo",
"qwen-long",
"qwen-vl-max",
"qwen-vl-plus",
],
"xiaomi": [
# Xiaomi MiMo models — the public /v1/models endpoint requires the
# `api-key` custom header (NOT Authorization: Bearer), so the live
# call often fails with 401 even with the right key. We ship a
# known-good list as fallback. See https://mimo.mi.com/docs/
"mimo-v2.5-pro",
"mimo-v2.5",
"mimo-v2.5-asr",
"mimo-v2.5-tts",
"mimo-v2.5-tts-voiceclone",
"mimo-v2.5-tts-voicedesign",
],
"mistral": [
"mistral-large-latest",
"mistral-medium-latest",
"mistral-small-latest",
"open-mistral-7b",
"open-mixtral-8x7b",
"codestral-latest",
],
}
@router.get("/api/diagnostics", response_model=DiagnosticsResponse)
async def api_diagnostics(current_user=Depends(require_admin)):
"""Return index statistics and system diagnostics.
Includes document counts, token counts, memory estimates,
and inverted index status.
"""
import sys
from backend.search import get_inverted_index
inv = get_inverted_index()
# Per-vault stats
vault_stats = {}
total_files = 0
total_tags = 0
# Snapshot both dicts first: the indexer mutates them from background
# threads, and iterating a live dict raises "dictionary changed size".
for vname, vdata in list(index.items()):
file_count = len(vdata.get("files", []))
tag_count = len(vdata.get("tags", {}))
vault_stats[vname] = {"file_count": file_count, "tag_count": tag_count}
total_files += file_count
total_tags += tag_count
# Memory estimate for inverted index
word_index = inv.word_index.copy()
word_index_entries = sum(len(docs) for docs in word_index.values())
mem_estimate_mb = round(
(sys.getsizeof(inv.word_index) + word_index_entries * 80
+ len(inv.doc_info) * 200
+ len(inv._sorted_tokens) * 60) / (1024 * 1024), 2
)
return {
"index": {
"total_files": total_files,
"total_tags": total_tags,
"vaults": vault_stats,
},
"inverted_index": {
"unique_tokens": len(word_index),
"total_postings": word_index_entries,
"documents": inv.doc_count,
"sorted_tokens": len(inv._sorted_tokens),
"is_ready": inv.is_ready(),
"memory_estimate_mb": mem_estimate_mb,
},
"config": _load_config(),
"search_executor": {
"active": get_search_executor() is not None,
"max_workers": get_search_executor()._max_workers if get_search_executor() else 0,
},
}
@router.get("/api/dashboard", response_model=DashboardResponse)
async def api_dashboard(current_user=Depends(require_auth)):
"""Aggregated dashboard statistics across all accessible vaults."""
user_vaults = current_user.get("_token_vaults") or current_user.get("vaults", [])
vault_stats = []
total_files = 0
total_tags = set()
total_size = 0
total_images = 0
for vname, vdata in index.items():
if "*" not in user_vaults and vname not in user_vaults:
continue
files = vdata.get("files", [])
fc = len(files)
total_files += fc
vtags = set()
vsize = 0
vimages = 0
for f in files:
vtags.update(f.get("tags", []))
vsize += f.get("size", 0)
if (f.get("extension") or "").lower() in IMAGE_EXTENSIONS:
vimages += 1
total_tags.update(vtags)
total_size += vsize
total_images += vimages
vault_stats.append({
"name": vname, "file_count": fc, "tag_count": len(vtags),
"total_size_bytes": vsize, "image_count": vimages,
})
return {
"vaults": vault_stats,
"total_files": total_files,
"total_tags": len(total_tags),
"total_size_bytes": total_size,
"total_images": total_images,
}
+73
View File
@@ -0,0 +1,73 @@
"""Syncthing conflict endpoints (ROADMAP #85, tranche 8).
Handlers déplacés depuis :mod:`backend.main` sans changement de
comportement : mêmes chemins (``/api/conflicts*``), mêmes modèles de
réponse, mêmes dépendances d'authentification.
Adaptations strictement équivalentes :
- ``_resolve_safe_path`` / ``_backup_file`` → :mod:`backend.services.paths`
et :mod:`backend.services.backups` (pass-through).
"""
import logging
import shutil
from pathlib import Path
from fastapi import APIRouter, Body, Depends, HTTPException
from backend.audit import log_file_delete
from backend.auth.middleware import check_vault_access, require_auth
from backend.indexer import get_conflicts, get_vault_data, remove_single_file
from backend.schemas import ConflictResolveResponse, ConflictsResponse
from backend.services.backups import create_backup
from backend.services.paths import resolve_safe_path
from backend.sse import sse_manager
logger = logging.getLogger("obsigate")
router = APIRouter(tags=["conflicts"])
@router.get("/api/conflicts", response_model=ConflictsResponse)
async def api_conflicts(current_user=Depends(require_auth)):
"""List sync-conflict files across accessible vaults."""
user_vaults = current_user.get("_token_vaults") or current_user.get("vaults", [])
all_conflicts = get_conflicts()
if "*" not in user_vaults:
all_conflicts = [c for c in all_conflicts if c["vault"] in user_vaults]
return {"conflicts": all_conflicts, "total": len(all_conflicts)}
@router.post("/api/conflicts/resolve", response_model=ConflictResolveResponse)
async def api_conflict_resolve(body: dict = Body(...), current_user=Depends(require_auth)):
"""Resolve a conflict: keep_local (delete conflict file) or keep_conflict (replace original)."""
vault_name = body.get("vault")
conflict_path = body.get("conflict_path")
original_path = body.get("original_path")
action = body.get("action") # "keep_local" or "keep_conflict"
# mypy: narrow down from dict values
assert isinstance(vault_name, str), "'vault' is required and must be a string"
assert isinstance(conflict_path, str), "'conflict_path' is required and must be a string"
assert isinstance(original_path, str), "'original_path' is required and must be a string"
if not check_vault_access(vault_name, current_user):
raise HTTPException(403, f"Accès refusé à la vault '{vault_name}'")
vault_data = get_vault_data(vault_name)
if not vault_data:
raise HTTPException(404, "Vault not found")
vault_root = Path(vault_data["path"])
conf_file = resolve_safe_path(vault_root, conflict_path)
orig_file = resolve_safe_path(vault_root, original_path)
if not conf_file.exists():
raise HTTPException(404, "Conflict file not found")
try:
if action == "keep_conflict":
create_backup(orig_file, vault_name, original_path)
shutil.copy2(conf_file, orig_file)
logger.info(f"Conflict resolved (keep_conflict): {conflict_path} → {original_path}")
conf_file.unlink()
await remove_single_file(vault_name, conflict_path)
log_file_delete(current_user["username"], vault_name, conflict_path)
await sse_manager.broadcast("file_deleted", {"vault": vault_name, "path": conflict_path})
return {"status": "resolved", "action": action}
except Exception as e:
raise HTTPException(500, f"Error resolving conflict: {e!s}")
+3 -3
View File
@@ -9,7 +9,8 @@ d'authentification.
Adaptations strictement équivalentes :
- ``_resolve_safe_path`` → :mod:`backend.services.paths` (pass-through).
- ``_render_markdown`` reste dans ``main`` (import différé).
- ``_render_markdown`` vient de :mod:`backend.render` (#85 T9, sans cycle
d'import).
- ``_resolve_export_target`` / ``_safe_export_name`` (export uniquement)
sont définis ici ; ``stream_file_with_range`` vit dans
:mod:`backend.routers.helpers` (partagé).
@@ -29,6 +30,7 @@ from backend.history import record_open
from backend.indexer import get_vault_data, index, parse_markdown_file
from backend.media_thumbs import generate_thumbnail, is_decodable
from backend.media_types import is_audio, is_image, is_video, media_mime_type
from backend.render import _render_markdown
from backend.routers.helpers import media_max_inline_bytes, stream_file_with_range
from backend.schemas import (
AllVaultSettingsResponse,
@@ -85,8 +87,6 @@ def _safe_export_name(name: str) -> str:
)
async def api_file_pdf(vault_name: str, path: str = Query(..., description="Relative path to file"), current_user=Depends(require_auth)):
"""Download a markdown file as PDF."""
from backend.main import _render_markdown # différé : évite l'import circulaire (#85)
if generate_pdf is None:
raise HTTPException(501, "PDF export unavailable (WeasyPrint/GTK not available)")
if not check_vault_access(vault_name, current_user):
+76 -5
View File
@@ -7,8 +7,8 @@ lecture), mêmes modèles de réponse (déménagés dans
Adaptations strictement équivalentes :
- ``_resolve_safe_path`` → :mod:`backend.services.paths` (pass-through).
- ``_render_markdown`` reste dans ``main`` (import différé, extraction
prévue dans une tranche ultérieure).
- ``_render_markdown`` vient de :mod:`backend.render` (#85 T9, sans cycle
d'import).
- ``_content_disposition`` / ``_media_max_inline_bytes`` / ``EXT_TO_LANG``
ont déménagé : helpers partagés dans :mod:`backend.routers.helpers`
(``EXT_TO_LANG`` n'était utilisé que par la vue fichier).
@@ -31,12 +31,14 @@ from backend.indexer import (
parse_markdown_file,
)
from backend.media_types import is_audio, is_image, is_video, media_mime_type
from backend.render import _render_markdown
from backend.routers.helpers import media_max_inline_bytes
from backend.schemas import (
BacklinksResponse,
BrowseResponse,
FileContentResponse,
FileRawResponse,
XlsxSheetWindowResponse,
)
from backend.services.files import read_raw_file
from backend.services.paths import resolve_safe_path
@@ -177,6 +179,71 @@ async def api_file_backlinks(
}
@router.get(
"/api/file/{vault_name}/xlsx/sheet", response_model=XlsxSheetWindowResponse
)
def api_file_xlsx_sheet(
vault_name: str,
path: str = Query(..., description="Relative path to the .xlsx file"),
sheet: str = Query(..., description="Sheet name (as shown in the viewer tab)"),
offset: int = Query(0, ge=0, description="0-based index of the first row to return"),
limit: int = Query(
200, ge=1, le=1000, description="Rows to return (server-capped)"
),
current_user=Depends(require_auth),
):
"""Return a window of rows of one sheet of an .xlsx workbook (#153 A9).
Backs the viewer's lazy loading: instead of every sheet in a single JSON
payload, the client asks for the block it is about to display. The row
numbers and the ``data-cell`` references are the real A1 coordinates of the
sheet, so a window behaves like the full render (editing a cell in it
targets the right cell).
The response also carries ``total_rows``/``total_cols`` and the ``truncated``
flag, so the client can say what is hidden behind the 500x40 render caps
instead of silently hiding it.
Args:
vault_name: Name of the vault.
path: Relative path of the .xlsx file within the vault.
sheet: Sheet name; **404** if the workbook has no such sheet.
offset: 0-based index of the first row to return.
limit: Rows to return, capped server-side at 1000.
Returns:
``XlsxSheetWindowResponse`` with the rendered ``html`` of the window.
Raises:
HTTPException: 403 (vault access), 404 (vault, file or sheet unknown),
415 (not an .xlsx file), 500 (unreadable workbook).
"""
if not check_vault_access(vault_name, current_user):
raise HTTPException(status_code=403, detail=f"Accès refusé à la vault '{vault_name}'")
vault_data = get_vault_data(vault_name)
if not vault_data:
raise HTTPException(status_code=404, detail=f"Vault '{vault_name}' not found")
file_path = resolve_safe_path(Path(vault_data["path"]), path)
if not file_path.is_file():
raise HTTPException(status_code=404, detail=f"File not found: {path}")
if file_path.suffix.lower() != ".xlsx":
raise HTTPException(status_code=415, detail="Le fichier n'est pas un classeur .xlsx")
# Import tardif : openpyxl n'est chargé que si un .xlsx est réellement demandé.
from backend.xlsx_reader import read_sheet_window
try:
window = read_sheet_window(file_path, sheet, offset=offset, limit=limit)
except Exception as e:
logger.error(f"XLSX sheet read error for {path}: {e}")
raise HTTPException(status_code=500, detail=f"Error reading XLSX: {e!s}")
if window is None:
raise HTTPException(status_code=404, detail=f"Feuille introuvable: {sheet}")
return {"vault": vault_name, "path": path, **window}
@router.get("/api/file/{vault_name}", response_model=FileContentResponse)
async def api_file(vault_name: str, path: str = Query(..., description="Relative path to file"), current_user=Depends(require_auth)):
"""Return rendered HTML and metadata for a file.
@@ -192,8 +259,6 @@ async def api_file(vault_name: str, path: str = Query(..., description="Relative
Returns:
``FileContentResponse`` with HTML, metadata, and tags.
"""
from backend.main import _render_markdown # différé : évite l'import circulaire (#85)
if not check_vault_access(vault_name, current_user):
raise HTTPException(status_code=403, detail=f"Accès refusé à la vault '{vault_name}'")
vault_data = get_vault_data(vault_name)
@@ -242,8 +307,11 @@ async def api_file(vault_name: str, path: str = Query(..., description="Relative
# === Excel .xlsx: render sheets as HTML tables (binary, before read_text) ===
if ext == ".xlsx":
try:
from backend.xlsx_reader import render_sheets
from backend.xlsx_reader import inspect_workbook, render_sheets
# #153 A15 — every sheet dict already carries its styles, aligns,
# merges and freeze anchor (read_workbook_meta, one normal-mode
# load inside render_sheets).
sheets = render_sheets(file_path)
size = file_path.stat().st_size
return {
@@ -258,6 +326,9 @@ async def api_file(vault_name: str, path: str = Query(..., description="Relative
"is_markdown": False,
"is_xlsx": True,
"xlsx_sheets": sheets,
# #153 A1 — parts a save would drop; the viewer warns and asks
# for an explicit confirmation before forcing the write.
"xlsx_lossy_features": inspect_workbook(file_path),
"unsupported": False,
"size_bytes": size,
}
+89 -5
View File
@@ -64,6 +64,9 @@ from backend.services.mutations import (
from backend.services.mutations import (
move_path as service_move_path,
)
from backend.services.mutations import (
mutate_xlsx_structure as service_mutate_xlsx_structure,
)
from backend.services.mutations import (
rename_directory as service_rename_directory,
)
@@ -114,17 +117,35 @@ async def api_file_save(
@router.put("/api/file/{vault_name}/xlsx/save", response_model=FileSaveResponse)
async def api_file_xlsx_save(
def api_file_xlsx_save(
vault_name: str,
path: str = Query(..., description="Relative path to the .xlsx file"),
body: dict = Body(..., description='{"sheet": str, "cells": {"A1": value}}'),
body: dict = Body(
...,
description=(
'{"sheet": str, "cells": {"A1": value}, '
'"allow_formula": false, "force": false}'
),
),
current_user=Depends(require_auth),
):
"""Apply cell edits to an .xlsx workbook.
Expects a JSON body with ``sheet`` and ``cells`` (A1 references to new
scalar values, max 500 per request). A backup is created before the
workbook is rewritten.
scalar values, max 500 per request) plus two optional boolean flags:
* ``allow_formula`` — keep values starting with ``=``/``@`` as real
formulas. Off by default (#153 A4): such a value is stored as text so a
later Excel session cannot execute it (DDE).
* ``force`` — write a workbook carrying features openpyxl cannot re-serialize
(slicers, form controls, connections, custom XML, signature, cached formula
results). Without it the call fails **409** ``xlsx_lossy_content`` and the
client asks the user to confirm (#153 A1).
A backup is created before the workbook is rewritten, and the new archive
swaps in atomically. Declared as a sync endpoint on purpose: the openpyxl
round-trip and the per-file lock wait (#153 A3) then run in the threadpool
instead of blocking the event loop.
"""
if not check_vault_access(vault_name, current_user):
raise HTTPException(status_code=403, detail=f"Accès refusé à la vault '{vault_name}'")
@@ -138,8 +159,16 @@ async def api_file_xlsx_save(
for ref, value in cells.items():
if not isinstance(ref, str) or not isinstance(value, (str, int, float, bool, type(None))):
raise HTTPException(status_code=400, detail=f"Cellule invalide: {ref!r}")
flags: dict[str, bool] = {}
for name in ("allow_formula", "force"):
raw = body.get(name, False)
if not isinstance(raw, bool):
raise HTTPException(status_code=400, detail=f"Flag invalide: {name}")
flags[name] = raw
result = service_edit_xlsx_cells(vault_name, path, sheet, cells)
result = service_edit_xlsx_cells(
vault_name, path, sheet, cells, **flags
)
log_file_save(
current_user["username"], vault_name, path,
sum(len(str(v)) for v in cells.values()),
@@ -148,6 +177,61 @@ async def api_file_xlsx_save(
return {"status": "ok", "vault": result["vault"], "path": result["path"], "size": result["size"]}
@router.put("/api/file/{vault_name}/xlsx/structure", response_model=FileSaveResponse)
def api_file_xlsx_structure(
vault_name: str,
path: str = Query(..., description="Relative path to the .xlsx file"),
body: dict = Body(
...,
description=(
'{"actions": [{"op": "sheet_add", "name": "X"}, '
'{"op": "row_insert", "sheet": "X", "at": 2, "count": 1}], '
'"force": false}'
),
),
current_user=Depends(require_auth),
):
"""Apply structural changes to an .xlsx workbook (#153 A14).
``actions`` is an ordered list applied in one locked, atomic rewrite:
``sheet_add`` (``name``, optional ``at`` 0-based), ``sheet_rename``
(``from``/``to``), ``sheet_delete`` (refused on the last sheet),
``sheet_duplicate`` (``name``/``as``) and ``row_insert``/``row_delete``/
``col_insert``/``col_delete`` (``sheet``, 1-based ``at``, ``count``).
Without ``force`` the call fails **409** ``xlsx_lossy_content`` when the
workbook carries features openpyxl cannot rewrite (same gate as the cell
edits). A backup is created before the archive is replaced.
Args:
vault_name: Name of the vault.
path: Relative path to the ``.xlsx`` file.
body: JSON body with ``actions`` (1 to 50) and optional ``force``.
Returns:
``FileSaveResponse`` confirming the write.
"""
if not check_vault_access(vault_name, current_user):
raise HTTPException(status_code=403, detail=f"Accès refusé à la vault '{vault_name}'")
actions = body.get("actions")
if not isinstance(actions, list) or not actions or len(actions) > 50:
raise HTTPException(status_code=400, detail="Actions invalides (1 à 50 par requête)")
raw_force = body.get("force", False)
if not isinstance(raw_force, bool):
raise HTTPException(status_code=400, detail="Flag invalide: force")
result = service_mutate_xlsx_structure(
vault_name, path, actions, force=raw_force
)
log_file_save(
current_user["username"], vault_name, path,
len(actions),
current_user.get("_request_ip", "unknown"),
)
return {"status": "ok", "vault": result["vault"], "path": result["path"], "size": len(result["applied"])}
@router.delete("/api/file/{vault_name}", response_model=FileDeleteResponse)
async def api_file_delete(vault_name: str, path: str = Query(..., description="Relative path to file"), current_user=Depends(require_auth)):
"""Delete a file from the vault.
+160
View File
@@ -0,0 +1,160 @@
"""History endpoints — recent, bookmarks, saved searches (ROADMAP #85, tranche 8).
Handlers déplacés depuis :mod:`backend.main` sans changement de
comportement : mêmes chemins, mêmes modèles (``BookmarkToggleRequest``
déménagé dans :mod:`backend.schemas`), mêmes dépendances
d'authentification.
Adaptations strictement équivalentes :
- ``_resolve_safe_path`` / ``_backup_file`` → :mod:`backend.services.paths`
et :mod:`backend.services.backups` (pass-through).
- ``_load_config`` vient de :mod:`backend.routers.config`.
"""
import logging
from pathlib import Path
import frontmatter
from fastapi import APIRouter, Body, Depends, HTTPException, Query
from backend.auth.middleware import check_vault_access, require_auth
from backend.history import get_bookmarks, toggle_bookmark
from backend.indexer import find_file_in_index, get_vault_data, update_single_file
from backend.routers.config import _load_config
from backend.saved_searches import delete_saved, get_saved, save_search
from backend.schemas import (
BookmarksResponse,
BookmarkToggleRequest,
BookmarkToggleResponse,
RecentResponse,
SavedSearch,
StatusResponse,
)
from backend.services.backups import create_backup
from backend.services.paths import resolve_safe_path
from backend.services.recent import humanize_mtime, list_recent
logger = logging.getLogger("obsigate")
router = APIRouter(tags=["Bookmarks"])
@router.get("/api/recent", response_model=RecentResponse)
async def api_recent(limit: int | None = Query(None), vault: str | None = Query(None), mode: str | None = Query("opened"), current_user=Depends(require_auth)):
config = _load_config()
actual_limit = limit if limit is not None else config.get("recent_files_limit", 20)
username = current_user.get("username")
user_vaults = current_user.get("_token_vaults") or current_user.get("vaults", [])
return list_recent(
username,
user_vaults,
vault=vault,
limit=actual_limit,
mode=mode or "opened",
)
@router.get("/api/bookmarks", response_model=BookmarksResponse)
async def api_bookmarks(vault: str | None = Query(None), current_user=Depends(require_auth)):
username = current_user.get("username")
user_vaults = current_user.get("_token_vaults") or current_user.get("vaults", [])
if not username:
return {"files": []}
history = get_bookmarks(username, vault_filter=vault)
files_resp = []
for item in history:
v_name = item["vault"]
if "*" not in user_vaults and v_name not in user_vaults:
continue
# Find in index to get metadata
f_idx = find_file_in_index(item["path"], v_name)
if f_idx:
files_resp.append({
"path": f_idx["path"],
"title": f_idx.get("title") or item["path"].split("/")[-1],
"vault": v_name,
"mtime": item["bookmarked_at"],
"mtime_human": humanize_mtime(item["bookmarked_at"]),
"size_bytes": f_idx.get("size", 0),
"tags": [f"#{t}" for t in f_idx.get("tags", [])][:5],
"bookmarked": True
})
else:
files_resp.append({
"path": item["path"],
"title": item.get("title") or item["path"].split("/")[-1],
"vault": v_name,
"mtime": item["bookmarked_at"],
"mtime_human": humanize_mtime(item["bookmarked_at"]),
"tags": [],
"bookmarked": True
})
return {
"files": files_resp,
"total": len(files_resp)
}
@router.post("/api/bookmarks/toggle", response_model=BookmarkToggleResponse)
async def api_toggle_bookmark(req: BookmarkToggleRequest, current_user=Depends(require_auth)):
username = current_user.get("username")
if not username:
raise HTTPException(status_code=401, detail="Not authenticated")
# Check vault access
if not check_vault_access(req.vault, current_user):
raise HTTPException(status_code=403, detail="Access denied to vault")
is_now_bookmarked = toggle_bookmark(username, req.vault, req.path, req.title or "")
# Update the file's YAML frontmatter: favoris: true/false
vault_data = get_vault_data(req.vault)
if vault_data:
file_path = resolve_safe_path(Path(vault_data["path"]), req.path)
if file_path.exists() and file_path.suffix == ".md":
try:
raw = file_path.read_text(encoding="utf-8", errors="replace")
post = frontmatter.loads(raw)
if is_now_bookmarked:
post.metadata["favoris"] = True
elif "favoris" in post.metadata:
del post.metadata["favoris"]
new_raw = frontmatter.dumps(post)
create_backup(file_path, req.vault, req.path)
file_path.write_text(new_raw, encoding="utf-8")
await update_single_file(req.vault, str(file_path))
except Exception as e:
logger.warning(f"Failed to update favoris metadata on {req.vault}/{req.path}: {e}")
return {"bookmarked": is_now_bookmarked}
@router.get("/api/saved-searches", response_model=list[SavedSearch])
async def api_saved_searches(current_user=Depends(require_auth)):
username = current_user.get("username")
if not username:
raise HTTPException(401)
return get_saved(username)
@router.post("/api/saved-searches", response_model=SavedSearch)
async def api_save_search(body: dict = Body(...), current_user=Depends(require_auth)):
username = current_user.get("username")
if not username:
raise HTTPException(401)
return save_search(username, body)
@router.delete("/api/saved-searches/{search_id}", response_model=StatusResponse)
async def api_delete_saved_search(search_id: str, current_user=Depends(require_auth)):
username = current_user.get("username")
if not username:
raise HTTPException(401)
if not delete_saved(username, search_id):
raise HTTPException(404, "Not found")
return {"status": "deleted"}
+105
View File
@@ -0,0 +1,105 @@
"""Real-time endpoints — SSE stream & collaboration WebSocket (ROADMAP #85, tranche 9).
Handlers déplacés depuis :mod:`backend.main` sans changement de
comportement : mêmes chemins (``/api/events``,
``/ws/collab/{vault}/{path}``), même authentification (Depend pour le SSE,
manuelle pour le WebSocket — les ``Depends`` FastAPI ne s'exécutent pas sur
les routes WebSocket).
Pas de tags déclarés : assignation par chemin via
``openapi_docs.tag_for_path`` comme avant (``/api/events`` → System).
"""
import asyncio
import json as _json
from fastapi import APIRouter, Depends, WebSocket
from fastapi.responses import StreamingResponse
from backend.auth.middleware import check_vault_access, require_auth
from backend.collab import authenticate_websocket, collab_manager
from backend.services.paths import resolve_safe_path
from backend.services.vaults import get_vault_root
from backend.sse import sse_manager
router = APIRouter()
@router.get(
"/api/events",
response_class=StreamingResponse,
responses={200: {"content": {"text/event-stream": {}}, "description": "Server-Sent Events stream"}},
)
async def api_events(current_user=Depends(require_auth)):
"""SSE stream for real-time index update notifications.
Sends keepalive comments every 30s. Events:
- ``index_updated``: partial index change (file create/modify/delete/move)
- ``index_reloaded``: full re-index completed
- ``vault_added``: new vault added dynamically
- ``vault_removed``: vault removed dynamically
"""
queue = await sse_manager.connect()
async def event_generator():
try:
# Send initial connection event
yield f"event: connected\ndata: {_json.dumps({'sse_clients': sse_manager.client_count})}\n\n"
while True:
try:
msg = await asyncio.wait_for(queue.get(), timeout=30.0)
yield f"event: {msg['event']}\ndata: {msg['data']}\n\n"
except asyncio.TimeoutError:
# Keepalive comment
yield ": keepalive\n\n"
except asyncio.CancelledError:
break
finally:
sse_manager.disconnect(queue)
return StreamingResponse(
event_generator(),
media_type="text/event-stream",
headers={
"Cache-Control": "no-cache",
"Connection": "keep-alive",
"X-Accel-Buffering": "no",
},
)
@router.websocket("/ws/collab/{vault_name}/{path:path}")
async def collab_websocket(websocket: WebSocket, vault_name: str, path: str):
"""Real-time collaborative editing over WebSocket (ROADMAP #62).
One *room* is created per ``vault::path``; all clients editing the same
file share Yjs/CRDT updates, awareness (cursors/selection) and a debounced
server-side persistence of the markdown content.
Authentication is performed manually (FastAPI ``Depends`` do not run for
WebSocket routes) and vault access is enforced per connection.
"""
from backend.services.errors import ServiceError
user = authenticate_websocket(websocket)
if user is None:
await websocket.close(code=4401)
return
if not check_vault_access(vault_name, user):
await websocket.close(code=4403)
return
try:
vault_root = get_vault_root(vault_name)
file_path = resolve_safe_path(vault_root, path)
except ServiceError:
await websocket.close(code=4404)
return
if not file_path.exists() or not file_path.is_file():
await websocket.close(code=4404)
return
await websocket.accept()
await collab_manager.connect(websocket, vault_name, path, file_path, user)
+19 -13
View File
@@ -11,9 +11,8 @@ Adaptations strictement équivalentes (pas de changement de comportement) :
wrappers directs : appelés ici via :mod:`backend.services.paths` et
:mod:`backend.services.backups` (mêmes signatures, mêmes exceptions
``ServiceError`` toujours mappées par le handler global de ``main``).
- ``_render_markdown`` reste défini dans ``main`` (extraction prévue dans
une tranche ultérieure) : import différé à l'intérieur des handlers, donc
sans import circulaire au chargement.
- ``_render_markdown`` vient de :mod:`backend.render` (#85 T9, sans cycle
d'import).
"""
import html as html_mod
@@ -22,11 +21,12 @@ import logging
from pathlib import Path
import frontmatter
from fastapi import APIRouter, Body, Depends, HTTPException, Query
from fastapi import APIRouter, Body, Depends, HTTPException, Query, Request
from fastapi.responses import FileResponse, HTMLResponse, Response
from backend.auth.middleware import check_vault_access, require_auth
from backend.indexer import get_vault_data, parse_markdown_file, update_single_file
from backend.render import _render_markdown
from backend.schemas import ShareModel, StatusResponse
from backend.secret_redactor import redact_file_content
from backend.services.backups import create_backup
@@ -115,8 +115,6 @@ async def api_share_revoke(share_id: str, current_user=Depends(require_auth)):
)
async def public_share_pdf_download(token: str):
"""Download shared document as real PDF via WeasyPrint."""
from backend.main import _render_markdown # différé : évite l'import circulaire (#85)
if generate_pdf is None:
raise HTTPException(501, "PDF export unavailable (WeasyPrint/GTK not available)")
share = get_share_by_token(token)
@@ -166,9 +164,9 @@ async def public_share_raw(token: str):
@router.get("/s/{token}", response_class=HTMLResponse)
async def public_share_view(token: str):
async def public_share_view(request: Request, token: str):
"""Public share view — no authentication required."""
from backend.main import _render_markdown # différé : évite l'import circulaire (#85)
from backend.csp import inject_csp_nonce
share = get_share_by_token(token)
if not share:
@@ -234,7 +232,9 @@ async def public_share_view(token: str):
if fm_items:
fm_html = f'<div class="fm-section"><div class="fm-header">Frontmatter</div><div class="fm-body">{"".join(fm_items)}</div></div>'
return HTMLResponse(f"""<!DOCTYPE html><html lang="fr" data-theme="dark"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
return HTMLResponse(
inject_csp_nonce(
f"""<!DOCTYPE html><html lang="fr" data-theme="dark"><head><meta charset="utf-8"><meta name="viewport" content="width=device-width,initial-scale=1">
<title>{title_esc} — ObsiGate Share</title>
<style>
:root {{ --bg:#1a1a2e; --bg-card:#16213e; --text:#e0e0e0; --text-muted:#888; --accent:#6366f1; --border:#2a2a4a; --banner-bg:var(--accent); --banner-text:#fff; }}
@@ -278,15 +278,15 @@ body{{font-family:system-ui,-apple-system,sans-serif;background:var(--bg);color:
</div>
<div class="toolbar">
<span class="toolbar-title">{title_esc}</span>
<button class="toolbar-btn" onclick="toggleTheme()" title="Thème clair/sombre">
<button class="toolbar-btn" data-share-theme title="Thème clair/sombre">
<svg id="theme-icon-dark" xmlns="http://www.w3.org/2000/svg" width="15" height="15" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M21 12.79A9 9 0 1 1 11.21 3 7 7 0 0 0 21 12.79z"/></svg>
<svg id="theme-icon-light" xmlns="http://www.w3.org/2000/svg" width="15" height="15" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round" style="display:none"><circle cx="12" cy="12" r="5"/><line x1="12" y1="1" x2="12" y2="3"/><line x1="12" y1="21" x2="12" y2="23"/><line x1="4.22" y1="4.22" x2="5.64" y2="5.64"/><line x1="18.36" y1="18.36" x2="19.78" y2="19.78"/><line x1="1" y1="12" x2="3" y2="12"/><line x1="21" y1="12" x2="23" y2="12"/><line x1="4.22" y1="19.78" x2="5.64" y2="18.36"/><line x1="18.36" y1="5.64" x2="19.78" y2="4.22"/></svg>
</button>
<button class="toolbar-btn" onclick="exportMD()" title="Télécharger en Markdown">
<button class="toolbar-btn" data-share-md title="Télécharger en Markdown">
<svg xmlns="http://www.w3.org/2000/svg" width="15" height="15" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M21 15v4a2 2 0 0 1-2 2H5a2 2 0 0 1-2-2v-4"/><polyline points="7 10 12 15 17 10"/><line x1="12" y1="15" x2="12" y2="3"/></svg>
.md
</button>
<button class="toolbar-btn" onclick="location.href=location.pathname+'/pdf'" title="Télécharger en PDF">
<button class="toolbar-btn" data-share-pdf title="Télécharger en PDF">
<svg xmlns="http://www.w3.org/2000/svg" width="15" height="15" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M14 2H6a2 2 0 0 0-2 2v16a2 2 0 0 0 2-2V8z"/><polyline points="14 2 14 8 20 8"/><line x1="16" y1="13" x2="8" y2="13"/><line x1="16" y1="17" x2="8" y2="17"/><polyline points="10 9 9 9 8 9"/></svg>
PDF
</button>
@@ -297,4 +297,10 @@ body{{font-family:system-ui,-apple-system,sans-serif;background:var(--bg);color:
function toggleTheme(){{var t=document.documentElement;var isDark=t.dataset.theme==="dark";t.dataset.theme=isDark?"light":"dark";document.getElementById("theme-icon-dark").style.display=isDark?"none":"";document.getElementById("theme-icon-light").style.display=isDark?"":"none";localStorage.setItem("obsigate-share-theme",t.dataset.theme)}}
(function(){{var s=localStorage.getItem("obsigate-share-theme");if(!s)s="dark";document.documentElement.dataset.theme=s;var isDark=s==="dark";document.getElementById("theme-icon-dark").style.display=isDark?"":"none";document.getElementById("theme-icon-light").style.display=isDark?"none":""}})();
function exportMD(){{var raw=JSON.parse(document.getElementById("raw-content").textContent);var b=new Blob([raw],{{type:"text/markdown"}});var a=document.createElement("a");a.href=URL.createObjectURL(b);a.download={title_download_js};a.click()}}
</script></body></html>""")
document.querySelector("[data-share-theme]").addEventListener("click",toggleTheme);
document.querySelector("[data-share-md]").addEventListener("click",exportMD);
document.querySelector("[data-share-pdf]").addEventListener("click",function(){{location.href=location.pathname+"/pdf"}});
</script></body></html>""",
request.state.csp_nonce,
),
)
+107
View File
@@ -0,0 +1,107 @@
"""Vault management endpoints (ROADMAP #85, tranche 8).
Handlers déplacés depuis :mod:`backend.main` sans changement de
comportement : mêmes chemins (``/api/vaults*``), mêmes modèles de réponse
(``VaultInfo`` déménagé dans :mod:`backend.schemas`), mêmes dépendances
d'authentification.
Le handle du file-watcher vit désormais dans :mod:`backend.watcher_state`
(partagé avec le lifespan de ``main``) au lieu du global de ``main``.
"""
from pathlib import Path
from fastapi import APIRouter, Body, Depends, HTTPException
from backend.auth.middleware import require_admin, require_auth
from backend.indexer import add_vault_to_index, index, remove_vault_from_index
from backend.schemas import VaultActionResponse, VaultInfo, VaultsStatusResponse, VaultStatsResponse
from backend.services.vaults import list_accessible_vaults
from backend.sse import sse_manager
from backend.watcher_state import get_watcher
router = APIRouter(tags=["vaults"])
@router.get("/api/vaults", response_model=list[VaultInfo])
async def api_vaults(current_user=Depends(require_auth)):
"""List configured vaults the user has access to.
Returns:
List of vault summary objects filtered by user permissions.
"""
return list_accessible_vaults(current_user)
@router.post("/api/vaults/add", response_model=VaultStatsResponse)
async def api_add_vault(body: dict = Body(...), current_user=Depends(require_admin)):
"""Add a new vault dynamically without restarting.
Body:
name: Display name for the vault.
path: Absolute filesystem path to the vault directory.
"""
name = body.get("name", "").strip()
vault_path = body.get("path", "").strip()
if not name or not vault_path:
raise HTTPException(status_code=400, detail="Both 'name' and 'path' are required")
if name in index:
raise HTTPException(status_code=409, detail=f"Vault '{name}' already exists")
if not Path(vault_path).exists():
raise HTTPException(status_code=400, detail=f"Path does not exist: {vault_path}")
stats = await add_vault_to_index(name, vault_path)
# Start watching the new vault
watcher = get_watcher()
if watcher:
await watcher.add_vault(name, vault_path)
await sse_manager.broadcast("vault_added", {"vault": name, "stats": stats})
return {"status": "ok", "vault": name, "stats": stats}
@router.delete("/api/vaults/{vault_name}", response_model=VaultActionResponse)
async def api_remove_vault(vault_name: str, current_user=Depends(require_admin)):
"""Remove a vault from the index and stop watching it.
Args:
vault_name: Name of the vault to remove.
"""
if vault_name not in index:
raise HTTPException(status_code=404, detail=f"Vault '{vault_name}' not found")
# Stop watching
watcher = get_watcher()
if watcher:
await watcher.remove_vault(vault_name)
await remove_vault_from_index(vault_name)
await sse_manager.broadcast("vault_removed", {"vault": vault_name})
return {"status": "ok", "vault": vault_name}
@router.get("/api/vaults/status", response_model=VaultsStatusResponse)
async def api_vaults_status(current_user=Depends(require_auth)):
"""Detailed status of all vaults including watcher state.
Returns per-vault: file count, tag count, watching status, vault path.
"""
watcher = get_watcher()
statuses = {}
for vname, vdata in index.items():
watching = watcher is not None and vname in watcher.observers
statuses[vname] = {
"file_count": len(vdata.get("files", [])),
"tag_count": len(vdata.get("tags", {})),
"path": vdata.get("path", ""),
"watching": watching,
}
return {
"vaults": statuses,
"watcher_active": watcher is not None,
"sse_clients": sse_manager.client_count,
}
+86 -1
View File
@@ -214,6 +214,30 @@ class RestoreResponse(BaseModel):
current_backed_up: int | None = Field(default=None, description="Timestamp of the backup created from the current version before restore, if any")
class BackupEntry(BaseModel):
"""A single backup version of a file (#85 — extrait de backend.main, inchangé)."""
timestamp: int = Field(description="Unix timestamp of when the backup was created")
datetime: str = Field(description="ISO 8601 datetime string")
size: int = Field(description="File size in bytes")
filename: str = Field(description="Backup filename on disk")
class BackupListResponse(BaseModel):
"""Response listing all available backups for a file (#85 — extrait de backend.main, inchangé)."""
vault: str = Field(description="Vault name")
path: str = Field(description="Relative file path")
backups: list[BackupEntry] = Field(description="Available backups, newest first")
class DiffRequest(BaseModel):
"""Request parameters for generating a diff (#85 — extrait de backend.main, inchangé)."""
version: int = Field(description="Timestamp of the backup version to compare")
compare_with: int | None = Field(default=None, description="Timestamp of another backup version. If omitted, compares with the current file.")
# ---------------------------------------------------------------------------
# Files — browse / read (#85 — extrait de backend.main, inchangé)
# ---------------------------------------------------------------------------
@@ -262,7 +286,20 @@ class FileContentResponse(BaseModel):
is_csv: bool | None = Field(default=None, description="True for CSV files")
is_xlsx: bool | None = Field(default=None, description="True for Excel .xlsx files")
xlsx_sheets: list[dict[str, Any]] | None = Field(
default=None, description="Rendered xlsx sheets [{name, html}]"
default=None,
description=(
"Rendered xlsx sheets [{name, html, rows, cols, total_rows, "
"total_cols, max_rows, max_cols, truncated}] — `truncated` is true "
"when the sheet exceeds the 500x40 render caps (#153 A8)"
),
)
xlsx_lossy_features: list[str] | None = Field(
default=None,
description=(
"Workbook parts an openpyxl save would drop (#153 A1) — e.g. "
"cached_values, slicers, form_controls, connections, custom_xml, "
"signature, rich_comments, macros. Empty/absent = nothing at risk."
),
)
is_json: bool | None = Field(default=None, description="True for JSON files")
is_excalidraw: bool | None = Field(default=None, description="True for Excalidraw diagram files")
@@ -273,6 +310,32 @@ class FileContentResponse(BaseModel):
image_mime: str | None = Field(default=None, description="MIME type for image files")
class XlsxSheetWindowResponse(BaseModel):
"""One window of rows of a single .xlsx sheet (lazy loading, #153 A9).
Served by ``GET /api/file/{vault_name}/xlsx/sheet``; the row numbers and
the ``data-cell`` references in ``html`` are the real A1 coordinates of the
sheet, whatever the window.
"""
vault: str = Field(description="Vault name")
path: str = Field(description="Relative file path within the vault")
sheet: str = Field(description="Sheet name (as shown in the tab)")
offset: int = Field(description="0-based index of the first returned row")
limit: int = Field(description="Maximum number of rows returned (capped server-side)")
rows: int = Field(description="Rows actually returned in this window")
cols: int = Field(description="Columns of the rendered window")
total_rows: int = Field(description="Rows the sheet declares")
total_cols: int = Field(description="Columns the sheet declares")
max_rows: int = Field(description="Row cap of the renderer (500) — the coverage of this window")
max_cols: int = Field(description="Column cap of the renderer (40)")
truncated: bool = Field(
description="True when the sheet exceeds the 500x40 render caps"
)
has_more: bool = Field(description="True when rows remain after this window")
html: str = Field(description="Rendered HTML table for the window")
class FileRawResponse(BaseModel):
"""Raw text content of a file."""
@@ -411,6 +474,28 @@ class FileMoveResponse(BaseModel):
item_type: str = Field(description="Type of item moved: 'file' or 'directory'")
# ---------------------------------------------------------------------------
# Vaults & history (#85 — extrait de backend.main, inchangé)
# ---------------------------------------------------------------------------
class VaultInfo(BaseModel):
"""Summary information about a configured vault."""
name: str = Field(description="Display name of the vault")
file_count: int = Field(description="Number of indexed files")
tag_count: int = Field(description="Number of unique tags")
type: str = Field(default="VAULT", description="Type of the vault mapping (VAULT or DIR)")
class BookmarkToggleRequest(BaseModel):
"""Request to toggle a bookmark on a file."""
vault: str
path: str
title: str | None = None
# ---------------------------------------------------------------------------
# Search / suggest / graph (#85 — extrait de backend.main, inchangé)
# ---------------------------------------------------------------------------
+24 -9
View File
@@ -12,7 +12,12 @@ from sortedcontainers import SortedList
from backend import indexer as _indexer
from backend import semantic_search as _semantic
from backend.indexer import index
# NOTE: the shared index is read through ``_indexer.index`` everywhere, never
# via ``from backend.indexer import index``. That import binds the dict object
# once, so a module reload of ``backend.indexer`` (tests, dev reload) rebinds
# the module-level name to a FRESH dict while this module keeps writing to the
# stale one — the inverted index then silently indexes nothing (BUG-089).
from backend.services.regex_safety import (
MAX_REGEX_MATCHES,
truncate_for_regex,
@@ -371,9 +376,15 @@ class InvertedIndex:
self._sorted_tokens: SortedList = SortedList()
self._ready: bool = False # True after initial build
def is_stale(self) -> bool:
"""Return True if the index has not been built yet."""
return not self._ready
def is_ready(self) -> bool:
"""Return True once the initial build has completed.
The index is then kept current incrementally by ``add_document()`` /
``remove_document()``, so it never goes stale: there is no generation
counter, no cooldown and no lazy rebuild. Searches simply fall back to
a full scan while this is False (see ``search()``).
"""
return self._ready
def rebuild(self) -> None:
"""Rebuild inverted index from the global ``index`` dict.
@@ -393,7 +404,7 @@ class InvertedIndex:
self.vault_docs = defaultdict(set)
self.tag_docs = defaultdict(set)
for vault_name, vault_data in index.items():
for vault_name, vault_data in _indexer.index.items():
for file_info in vault_data.get("files", []):
doc_key = f"{vault_name}::{file_info['path']}"
self.doc_count += 1
@@ -537,6 +548,10 @@ class InvertedIndex:
self.doc_vault.pop(doc_key, None)
if vault_name in self.vault_docs:
self.vault_docs[vault_name].discard(doc_key)
# Drop the empty entry so a fully removed vault leaves no trace
# (it is a defaultdict: a bare lookup would recreate the key).
if not self.vault_docs[vault_name]:
del self.vault_docs[vault_name]
# Tags (per-document, NOT the global tag_norm_map)
for tag in file_info.get("tags", []):
td = self.tag_docs.get(tag.lower())
@@ -678,7 +693,7 @@ _indexer.set_index_change_hook(_on_index_change_hook)
def init_inverted_index():
"""Force initial inverted index build. Called after build_index completes on startup."""
if any(vdata.get("files") for vdata in index.values()):
if any(vdata.get("files") for vdata in _indexer.index.values()):
_inverted_index.rebuild()
logger.info("Inverted index initialized.")
@@ -739,7 +754,7 @@ def search(
results: list[dict[str, Any]] = []
inv = get_inverted_index()
use_index = (not inv.is_stale()) and inv.doc_count > 0
use_index = inv.is_ready() and inv.doc_count > 0
if use_index:
# BUG-033: retrieve candidates from the inverted index instead of
@@ -774,7 +789,7 @@ def search(
else:
candidates = [
(vault_name, file_info)
for vault_name, vault_data in index.items()
for vault_name, vault_data in _indexer.index.items()
if vault_filter == "all" or vault_name == vault_filter
for file_info in vault_data["files"]
]
@@ -1603,7 +1618,7 @@ def get_all_tags(vault_filter: str | None = None) -> dict[str, int]:
Dict mapping tag names to their total occurrence count.
"""
merged: dict[str, int] = {}
for vault_name, vault_data in index.items():
for vault_name, vault_data in _indexer.index.items():
if vault_filter and vault_filter != "all" and vault_name != vault_filter:
continue
for tag, count in vault_data.get("tags", {}).items():
-4
View File
@@ -457,10 +457,6 @@ class SemanticIndex:
"""Return True once a full rebuild has completed."""
return self._ready
def is_stale(self) -> bool:
"""Alias used by callers that check index freshness."""
return not self._ready
def _ensure_provider(self) -> EmbeddingProvider:
if self.provider is None:
self.provider = get_embedding_provider()
+1 -1
View File
@@ -31,7 +31,7 @@ DEFAULT_MAX_BACKUPS = 10
def _default_max_backups() -> int:
"""Read ``max_backups_per_file`` from app config (lazy, best-effort)."""
try:
from backend.main import _load_config
from backend.routers.config import _load_config # ROADMAP #85 T7 — déménagé depuis backend.main
return int(_load_config().get("max_backups_per_file", DEFAULT_MAX_BACKUPS))
except Exception: # pragma: no cover - config unavailable
+350 -28
View File
@@ -16,7 +16,10 @@ import logging
import os
import re
import shutil
from collections.abc import Callable
import threading
from collections.abc import Callable, Iterator
from contextlib import contextmanager
from datetime import date, datetime
from pathlib import Path
from typing import Any
@@ -230,10 +233,67 @@ _XLSX_CELL_RE = re.compile(r"^[A-Z]{1,3}[1-9][0-9]{0,7}$")
# number; dates/booleans stay text (upgrade path: parse locale dates too).
_XLSX_INT_RE = re.compile(r"^[+-]?\d+$")
_XLSX_FLOAT_RE = re.compile(r"^[+-]?(?:\d+\.\d*|\.\d+)$")
# #153 A4 — openpyxl turns any string starting with "=" into a formula, which
# Excel then evaluates on open (DDE / =cmd|… / =HYPERLINK exfiltration). "@" is
# the legacy Lotus-style trigger. "+"/"-" are left alone: they are numbers here.
_XLSX_FORMULA_RE = re.compile(r"^[=@]")
# #153 A10 — types recognised when a user types into a cell. Excel infers them
# too; storing everything as text would make a spreadsheet unusable (a boolean
# column stays a string, a date column sorts lexicographically).
_XLSX_TRUE_LITERALS = {"true", "vrai", "oui", "yes"}
_XLSX_FALSE_LITERALS = {"false", "faux", "non", "no"}
# Shape check before strptime: keeps the hot path free of format attempts.
_XLSX_DATE_RE = re.compile(r"^\d{1,2}[-/]\d{1,2}[-/]\d{4}(?:[ T]\d{1,2}:\d{2})?$")
# #153 A3 — per-file write lock. Two concurrent saves (two tabs, the AI agent
# and the viewer, a watcher restore) would otherwise read-modify-write on the
# same archive and the last writer silently wins. Kept deliberately small: the
# lock only covers the load → edit → atomic-replace window.
_XLSX_LOCK_TIMEOUT = 15.0
_xlsx_locks: dict[str, threading.Lock] = {}
_xlsx_locks_guard = threading.Lock()
@contextmanager
def _xlsx_write_lock(key: str) -> Iterator[None]:
"""Serialize the read-modify-write of one workbook path.
Raises:
ServiceError: ``conflict`` (409) when the lock is still held after
:data:`_XLSX_LOCK_TIMEOUT` seconds.
"""
with _xlsx_locks_guard:
lock = _xlsx_locks.setdefault(key, threading.Lock())
if not lock.acquire(timeout=_XLSX_LOCK_TIMEOUT):
raise ServiceError(
"Workbook is being modified by another operation, retry shortly",
code="conflict",
status=409,
details={"path": key, "timeout_seconds": _XLSX_LOCK_TIMEOUT},
)
try:
yield
finally:
lock.release()
def _coerce_xlsx_value(value: Any) -> Any:
"""Turn the string sent by the cell editor back into a scalar."""
"""Turn the string sent by the cell editor back into a scalar (#153 A10).
The coercion is symmetric with :func:`backend.xlsx_reader._fmt`: a value
typed by the user comes back as a string, and Excel would have inferred a
type when typing the same thing. Recognised here:
* an empty cell -> ``None`` (clears it)
* ``1234`` / ``-1`` -> ``int``
* ``1.5`` / ``.5`` -> ``float``
* ``TRUE``/``FAUX`` (case-insensitive) -> ``bool``
* ``31/12/2026`` / ``31/12/2026 14:30`` -> ``date``/``datetime`` (FR)
Anything else stays text. A date-looking string typed with a leading
``=`` is a formula and never reaches here as a date.
"""
if not isinstance(value, str):
return value
text = value.strip()
@@ -243,9 +303,48 @@ def _coerce_xlsx_value(value: Any) -> Any:
return int(text)
if _XLSX_FLOAT_RE.match(text):
return float(text)
lowered = text.lower()
if lowered in _XLSX_TRUE_LITERALS:
return True
if lowered in _XLSX_FALSE_LITERALS:
return False
if not _XLSX_FORMULA_RE.match(text):
parsed = _parse_fr_datetime(text)
if parsed is not None:
return parsed
return value
def _parse_fr_datetime(text: str) -> date | datetime | None:
"""Parse a FR-localised date/datetime, or return ``None``.
Accepts ``JJ/MM/AAAA`` and ``JJ/MM/AAAA HH:MM`` (also ``JJ-MM-AAAA``).
``dayfirst`` is what makes ``01/02/2026`` the 1st of February rather than
the 2nd of January — the French convention.
"""
if not _XLSX_DATE_RE.match(text):
return None
for fmt in ("%d/%m/%Y %H:%M", "%d/%m/%Y", "%d-%m-%Y %H:%M", "%d-%m-%Y"):
try:
return datetime.strptime(text, fmt)
except ValueError:
continue
return None
def _write_cell(ws: Any, ref: str, value: Any, *, allow_formula: bool) -> None:
"""Assign one cell, forcing text when it looks like a formula.
``cell.data_type = "s"`` is what stops openpyxl from emitting ``<f>``: the
text is then stored as an inline/shared string and Excel shows it verbatim.
"""
cell = ws[ref]
coerced = _coerce_xlsx_value(value)
cell.value = coerced
if not allow_formula and isinstance(coerced, str) and _XLSX_FORMULA_RE.match(coerced):
cell.data_type = "s"
def edit_xlsx_cells(
vault_name: str,
path: str,
@@ -253,15 +352,29 @@ def edit_xlsx_cells(
cells: dict[str, Any],
*,
backup: bool = True,
allow_formula: bool = False,
force: bool = False,
) -> dict[str, Any]:
"""Apply a batch of cell edits to an ``.xlsx`` workbook.
Raises:
ServiceError: ``not_found`` (404), ``read_only`` (403) or
``invalid`` (400) for a bad sheet, cell reference or value.
Args:
vault_name: Name of the vault the workbook belongs to.
path: Vault-relative path of the ``.xlsx`` file.
sheet: Worksheet title to edit.
cells: Mapping of A1 references to new scalar values.
backup: Create a timestamped ``.bak`` before rewriting the archive.
allow_formula: Keep values starting with ``=``/``@`` as real formulas.
Off by default (#153 A4): a typed ``=cmd|…`` is a DDE payload when
the file is later opened in Excel.
force: Write even when the workbook carries features openpyxl drops
(slicers, form controls, connections, custom XML, signature, cached
formula results — see :data:`backend.xlsx_reader.LOSSY_PARTS`).
ponytail: openpyxl round-trips values/formulas/styles but drops charts,
images and pivot tables; use the SheetJS path if a workbook needs those.
Raises:
ServiceError: ``not_found`` (404), ``read_only`` (403), ``conflict``
(409, concurrent write), ``xlsx_lossy_content`` (409, a lossy write was
attempted without ``force``) or ``invalid`` (400) for a bad sheet, cell
reference or value.
"""
root = get_vault_root(vault_name)
_ensure_writable(root)
@@ -286,30 +399,52 @@ def edit_xlsx_cells(
f"Invalid cell reference: {ref!r}", code="invalid", status=400
)
from openpyxl import load_workbook
if not force:
from backend.xlsx_reader import inspect_workbook
try:
wb = load_workbook(file_path)
except Exception as exc:
raise ServiceError(
f"Cannot open workbook: {exc}", code="invalid", status=400
) from exc
if sheet not in wb.sheetnames:
raise ServiceError(
f"Unknown sheet: {sheet}",
code="invalid",
status=400,
details={"sheets": wb.sheetnames},
)
lossy = inspect_workbook(file_path)
if lossy:
raise ServiceError(
"Saving this workbook would drop features ObsiGate cannot "
"preserve; retry with force=true after confirmation",
code="xlsx_lossy_content",
status=409,
details={"path": path, "features": lossy},
)
rel_path = _rel(root, file_path)
if backup:
create_backup(file_path, vault_name, rel_path)
with _xlsx_write_lock(str(file_path)):
from openpyxl import load_workbook
ws = wb[sheet]
for ref, value in cells.items():
ws[ref].value = _coerce_xlsx_value(value)
wb.save(file_path)
try:
wb = load_workbook(file_path)
except Exception as exc:
raise ServiceError(
f"Cannot open workbook: {exc}", code="invalid", status=400
) from exc
if sheet not in wb.sheetnames:
raise ServiceError(
f"Unknown sheet: {sheet}",
code="invalid",
status=400,
details={"sheets": wb.sheetnames},
)
rel_path = _rel(root, file_path)
if backup:
create_backup(file_path, vault_name, rel_path)
ws = wb[sheet]
for ref, value in cells.items():
_write_cell(ws, ref, value, allow_formula=allow_formula)
# #153 A2 — write beside the target then swap: a crash mid-save leaves
# the original workbook intact instead of a truncated archive.
tmp_path = file_path.with_name(f"{file_path.name}.{os.getpid()}.tmp")
try:
wb.save(tmp_path)
os.replace(tmp_path, file_path)
except Exception:
tmp_path.unlink(missing_ok=True)
raise
logger.info(f"XLSX cells saved: {vault_name}/{rel_path} [{sheet}] +{len(cells)}")
return {
@@ -320,6 +455,193 @@ def edit_xlsx_cells(
}
def mutate_xlsx_structure(
vault_name: str,
path: str,
actions: list[dict[str, Any]],
*,
backup: bool = True,
force: bool = False,
) -> dict[str, Any]:
"""Apply structural changes to an ``.xlsx`` workbook (#153 A14).
``actions`` is an ordered list — the workbook is loaded once and every
action is applied in sequence inside the same per-file lock and the same
atomic replace, so a half-applied batch can never reach the disk:
* ``{"op": "sheet_add", "name": "X", "at": 1}`` — new sheet (at =
optional 0-based position);
* ``{"op": "sheet_rename", "from": "X", "to": "Y"}``;
* ``{"op": "sheet_delete", "name": "X"}`` — refused when it is the
last sheet (an openpyxl workbook must keep one);
* ``{"op": "sheet_duplicate", "name": "X", "as": "Y"}`` — values,
styles and merged ranges are copied (not the data-dependent objects);
* ``{"op": "row_insert"|"row_delete"|"col_insert"|"col_delete",
"sheet": "X", "at": N, "count": k}`` — 1-based position, default 1.
All of it rides the same guards as the cell edits (P0): per-file lock,
``.tmp`` + ``os.replace`` atomic write and the ``force`` gate on lossy
round-trips. The UI proposes these actions with an explicit confirmation
— deletions are NOT recoverable from the viewer (only via the ``.bak``).
"""
root = get_vault_root(vault_name)
_ensure_writable(root)
file_path = resolve_safe_path(root, path)
if not file_path.exists() or not file_path.is_file():
raise ServiceError(
f"File not found: {path}",
code="not_found",
status=404,
details={"vault": vault_name, "path": path},
)
if not actions or len(actions) > 50:
raise ServiceError(
"Invalid actions (1 to 50 per request)", code="invalid", status=400
)
if not force:
from backend.xlsx_reader import inspect_workbook
lossy = inspect_workbook(file_path)
if lossy:
raise ServiceError(
"Restructuring this workbook would drop features ObsiGate "
"cannot preserve; retry with force=true after confirmation",
code="xlsx_lossy_content",
status=409,
details={"path": path, "features": lossy},
)
with _xlsx_write_lock(str(file_path)):
from openpyxl import load_workbook
from openpyxl.worksheet.copier import WorksheetCopy
try:
wb = load_workbook(file_path)
except Exception as exc:
raise ServiceError(
f"Cannot open workbook: {exc}", code="invalid", status=400
) from exc
rel_path = _rel(root, file_path)
applied: list[str] = []
try:
for i, action in enumerate(actions):
op = action.get("op")
try:
if op == "sheet_add":
name = str(action.get("name", "")).strip()
if not name or name in wb.sheetnames:
raise ServiceError(
f"Nom de feuille invalide ou déjà pris: {name!r}",
code="invalid", status=400,
)
ws = wb.create_sheet(name[:31])
at = action.get("at")
# create_sheet appends at the end: shift left by the
# distance between the last index and the target.
if isinstance(at, int) and 0 <= at < len(wb.sheetnames):
wb.move_sheet(ws, offset=at - (len(wb.sheetnames) - 1))
applied.append(f"sheet_add:{ws.title}")
elif op == "sheet_rename":
src, dst = str(action.get("from", "")), str(action.get("to", "")).strip()
if src not in wb.sheetnames or not dst or dst in wb.sheetnames:
raise ServiceError(
f"Renommage invalide: {src!r} -> {dst!r}",
code="invalid", status=400,
)
wb[src].title = dst[:31]
applied.append(f"sheet_rename:{src}->{dst}")
elif op == "sheet_delete":
name = str(action.get("name", ""))
if name not in wb.sheetnames:
raise ServiceError(
f"Feuille introuvable: {name}", code="invalid", status=400
)
if len(wb.sheetnames) <= 1:
raise ServiceError(
"Impossible de supprimer la dernière feuille",
code="invalid", status=400,
)
del wb[name]
applied.append(f"sheet_delete:{name}")
elif op == "sheet_duplicate":
name = str(action.get("name", ""))
new_name = str(action.get("as", "")).strip()
if name not in wb.sheetnames or not new_name or new_name in wb.sheetnames:
raise ServiceError(
f"Duplication invalide: {name!r} -> {new_name!r}",
code="invalid", status=400,
)
# WorksheetCopy is the documented dup path (openpyxl
# 3.1); it copies values, styles and merges — not
# charts/images, which openpyxl itself cannot clone.
copy = wb.create_sheet(new_name[:31])
WorksheetCopy(wb[name], copy).copy_worksheet()
applied.append(f"sheet_duplicate:{name}->{copy.title}")
elif op in ("row_insert", "row_delete", "col_insert", "col_delete"):
sheet = str(action.get("sheet", ""))
if sheet not in wb.sheetnames:
raise ServiceError(
f"Feuille introuvable: {sheet}", code="invalid", status=400
)
ws = wb[sheet]
at = action.get("at", 1)
count = action.get("count", 1)
if not isinstance(at, int) or at < 1 or not isinstance(count, int) or count < 1:
raise ServiceError(
"Position 'at' / 'count' invalides", code="invalid", status=400
)
if op == "row_insert":
ws.insert_rows(at, count)
elif op == "row_delete":
ws.delete_rows(at, count)
elif op == "col_insert":
ws.insert_cols(at, count)
else:
ws.delete_cols(at, count)
applied.append(f"{op}:{sheet}@{at}x{count}")
else:
raise ServiceError(
f"Action inconnue: {op!r}", code="invalid", status=400
)
except ServiceError:
raise
except Exception as exc:
raise ServiceError(
f"Action {i + 1} ({op}) a échoué: {exc}",
code="invalid", status=400,
) from exc
except ServiceError:
wb.close()
raise
if backup:
create_backup(file_path, vault_name, rel_path)
tmp_path = file_path.with_name(f"{file_path.name}.{os.getpid()}.tmp")
try:
wb.save(tmp_path)
os.replace(tmp_path, file_path)
except Exception:
tmp_path.unlink(missing_ok=True)
wb.close()
raise
wb.close()
logger.info(
f"XLSX structure: {vault_name}/{rel_path} {applied}"
)
return {
"success": True,
"vault": vault_name,
"path": rel_path,
"applied": applied,
}
def append_to_file(
vault_name: str,
path: str,
+48 -39
View File
@@ -10,6 +10,7 @@ No authentication required for public share views.
import json
import logging
import secrets
import threading
from datetime import datetime, timedelta, timezone
from pathlib import Path
@@ -17,6 +18,10 @@ logger = logging.getLogger("obsigate.share")
SHARES_FILE = Path("data/shares.json")
# ROADMAP #85 T10a — verrou autour des read-modify-write (perte de mises à
# jour en cas de créations/accès/révocations concurrents).
_lock = threading.RLock()
def _read() -> dict:
if not SHARES_FILE.exists():
@@ -41,26 +46,27 @@ def create_share(
expires_in_hours: int | None = None,
) -> dict:
"""Create a new share token for a document."""
data = _read()
token = secrets.token_hex(32) # 64-char hex token
with _lock:
data = _read()
token = secrets.token_hex(32) # 64-char hex token
expires_at = None
if expires_in_hours:
expires_at = (datetime.now(timezone.utc) + timedelta(hours=expires_in_hours)).isoformat()
expires_at = None
if expires_in_hours:
expires_at = (datetime.now(timezone.utc) + timedelta(hours=expires_in_hours)).isoformat()
share = {
"id": token,
"token": token,
"vault": vault,
"path": path,
"created_by": created_by,
"created_at": datetime.now(timezone.utc).isoformat(),
"expires_at": expires_at,
"access_count": 0,
"last_accessed": None,
}
data["shares"][token] = share
_write(data)
share = {
"id": token,
"token": token,
"vault": vault,
"path": path,
"created_by": created_by,
"created_at": datetime.now(timezone.utc).isoformat(),
"expires_at": expires_at,
"access_count": 0,
"last_accessed": None,
}
data["shares"][token] = share
_write(data)
logger.info(f"Created share for {vault}/{path} by {created_by}")
return share
@@ -80,22 +86,24 @@ def get_share_by_token(token: str) -> dict | None:
def record_access(token: str):
"""Increment access counter for a share."""
data = _read()
share = data["shares"].get(token)
if share:
share["access_count"] = share.get("access_count", 0) + 1
share["last_accessed"] = datetime.now(timezone.utc).isoformat()
_write(data)
with _lock:
data = _read()
share = data["shares"].get(token)
if share:
share["access_count"] = share.get("access_count", 0) + 1
share["last_accessed"] = datetime.now(timezone.utc).isoformat()
_write(data)
def revoke_share(share_id: str) -> bool:
"""Revoke (delete) a share by its token."""
data = _read()
if share_id in data["shares"]:
del data["shares"][share_id]
_write(data)
logger.info(f"Revoked share {share_id}")
return True
with _lock:
data = _read()
if share_id in data["shares"]:
del data["shares"][share_id]
_write(data)
logger.info(f"Revoked share {share_id}")
return True
return False
@@ -112,12 +120,13 @@ def list_shares(vault_filter: str | None = None) -> list:
def update_shares_after_rename(vault: str, old_path: str, new_path: str):
"""Update all shares when a file is renamed."""
data = _read()
updated = False
for sid, s in data["shares"].items():
if s.get("vault") == vault and s.get("path") == old_path:
s["path"] = new_path
updated = True
logger.info(f"Updated share {sid}: {vault}/{old_path} -> {new_path}")
if updated:
_write(data)
with _lock:
data = _read()
updated = False
for sid, s in data["shares"].items():
if s.get("vault") == vault and s.get("path") == old_path:
s["path"] = new_path
updated = True
logger.info(f"Updated share {sid}: {vault}/{old_path} -> {new_path}")
if updated:
_write(data)
+1
View File
@@ -13,6 +13,7 @@ from backend.tools import connected as _connected # noqa: F401 (registers conn
from backend.tools import crawler as _crawler # noqa: F401 (registers the site crawler)
from backend.tools import documents as _documents # noqa: F401 (registers document tools)
from backend.tools import service as _service # noqa: F401 (registers tools)
from backend.tools import spreadsheets as _spreadsheets # noqa: F401 (registers existing-workbook tools #153 A6)
from backend.tools import web as _web # noqa: F401 (registers web tools)
from backend.tools.context import (
ToolConfirmationRequired,
+7 -3
View File
@@ -18,8 +18,10 @@ import csv as csv_lib
import io
import logging
import re
from typing import Any
from xml.sax import saxutils
from typing import Any, cast
# saxutils.escape uniquement (échappement de chaînes, aucun parsing XML).
from xml.sax import saxutils # nosec B406
from backend.services.errors import ServiceError
from backend.services.mutations import save_raw_file
@@ -171,7 +173,9 @@ def _render_markdown_pdf(content: str, title: str) -> bytes | None:
escape=False,
plugins=["table", "strikethrough", "footnotes", "task_lists"],
)
html = renderer(content)
# mistune 3.3 types `Markdown.__call__` as `str | list[...]` (le
# renderer HTML renvoie toujours `str` à l'exécution).
html = cast(str, renderer(content))
return generate_pdf(build_pdf_html(html, title), title)
except Exception as e:
# WeasyPrint loads GTK lazily: a missing native library can surface at
+4
View File
@@ -52,6 +52,10 @@ _STEP_LABELS: dict[str, tuple[str, str | None]] = {
"git_search_issues": ("git_issues", "query"),
"git_get_file": ("git_file", "path"),
"create_xlsx": ("xlsx_create", "path"),
"list_xlsx_sheets": ("xlsx_sheets", "path"),
"xlsx_to_markdown": ("xlsx_read", "path"),
"update_xlsx_cells": ("xlsx_update", "path"),
"append_xlsx_rows": ("xlsx_append", "path"),
"create_docx": ("docx_create", "path"),
"create_csv": ("csv_create", "path"),
"create_pdf": ("pdf_create", "path"),
+55
View File
@@ -315,6 +315,61 @@ class DocxInput(BaseModel):
overwrite: bool = Field(True, description="Replace an existing file (with backup)")
class ListXlsxSheetsInput(BaseModel):
"""List the sheets of an existing .xlsx workbook (#153 A6)."""
vault: str = Field(..., description="Vault name")
path: str = Field(..., description="Vault-relative path of the .xlsx file")
class XlsxToMarkdownInput(BaseModel):
"""Read one sheet of an existing .xlsx workbook as markdown (#153 A6)."""
vault: str = Field(..., description="Vault name")
path: str = Field(..., description="Vault-relative path of the .xlsx file")
sheet: str = Field(
"", description="Sheet name (empty = the first/active sheet)"
)
class UpdateXlsxCellsInput(BaseModel):
"""Batch-edit cells of an existing .xlsx workbook (#153 A6)."""
vault: str = Field(..., description="Vault name")
path: str = Field(..., description="Vault-relative path of the .xlsx file")
sheet: str = Field(..., description="Worksheet title to edit")
cells: dict[str, str | int | float | bool | None] = Field(
..., description="A1 reference -> new value (max 500 per call)"
)
allow_formula: bool = Field(
False,
description="Store '='/'@' values as real formulas (off by default, DDE guard)",
)
force: bool = Field(
False,
description="Write even when features openpyxl cannot rewrite would be dropped",
)
class AppendXlsxRowsInput(BaseModel):
"""Append rows at the end of a sheet of an existing .xlsx (#153 A6)."""
vault: str = Field(..., description="Vault name")
path: str = Field(..., description="Vault-relative path of the .xlsx file")
sheet: str = Field(..., description="Worksheet title to extend")
rows: list[list[str | int | float | bool | None]] = Field(
..., description="Rows of cell values, appended below the last used row (max 500)"
)
allow_formula: bool = Field(
False,
description="Store '='/'@' values as real formulas (off by default, DDE guard)",
)
force: bool = Field(
False,
description="Write even when features openpyxl cannot rewrite would be dropped",
)
class CsvInput(BaseModel):
"""Create a .csv file in a vault from rows of cells."""
+17 -11
View File
@@ -17,6 +17,7 @@ from __future__ import annotations
import json
import logging
import os
import threading
from pathlib import Path
logger = logging.getLogger("obsigate.tools.secrets")
@@ -34,6 +35,9 @@ TOOL_KEY_NAMES: tuple[str, ...] = (
_SECRET_MARKERS = ("API_KEY", "TOKEN")
# ROADMAP #85 T10a — verrou autour des read-modify-write du store de clés.
_lock = threading.RLock()
def _keys_file() -> Path:
base = os.environ.get("OBSIGATE_DATA_DIR", "data")
@@ -89,21 +93,23 @@ def set_tool_key(name: str, value: str) -> None:
if name not in TOOL_KEY_NAMES:
raise ValueError(f"Clé non prise en charge: {name}")
value = (value or "").strip()
keys = _read_keys()
if value:
keys[name] = value
else:
keys.pop(name, None)
_write_keys(keys)
with _lock:
keys = _read_keys()
if value:
keys[name] = value
else:
keys.pop(name, None)
_write_keys(keys)
def delete_tool_key(name: str) -> bool:
"""Remove one key from the store; return True when it existed."""
if name not in TOOL_KEY_NAMES:
raise ValueError(f"Clé non prise en charge: {name}")
keys = _read_keys()
if name in keys:
del keys[name]
_write_keys(keys)
return True
with _lock:
keys = _read_keys()
if name in keys:
del keys[name]
_write_keys(keys)
return True
return False
+286
View File
@@ -0,0 +1,286 @@
"""Spreadsheet tools (#153 A6) — read and mutate existing ``.xlsx`` workbooks.
Complements :mod:`backend.tools.documents` (``create_xlsx`` creates a *new*
file; here the assistant can read and edit one that already exists):
* ``list_xlsx_sheets`` — READ, sheet names + dimensions;
* ``xlsx_to_markdown`` — READ, bounded markdown table for the LLM context;
* ``update_xlsx_cells`` — WRITE, batch cell edits (wraps the guarded service);
* ``append_xlsx_rows`` — WRITE, append whole rows at the end of a sheet.
Mutation tools go through :func:`backend.services.mutations.edit_xlsx_cells`,
which already carries the #153 P0 guards: per-file lock, atomic replace,
formula neutralisation (``allow_formula`` opt-in) and the lossy-write 409.
"""
from __future__ import annotations
import logging
from pathlib import Path
from typing import Any
from backend.services.errors import ServiceError
from backend.services.paths import resolve_safe_path
from backend.services.vaults import get_vault_root
from backend.tools.context import ToolContext, ToolError, ToolRisk
from backend.tools.registry import tool
from backend.tools.schemas import (
AppendXlsxRowsInput,
ListXlsxSheetsInput,
UpdateXlsxCellsInput,
XlsxToMarkdownInput,
)
logger = logging.getLogger("obsigate.tools.spreadsheets")
# xlsx_to_markdown ceiling: a workbook is a data dump, not prose. The table is
# for the LLM context, so both axes are bounded (same spirit as A5's index cap).
MAX_MD_ROWS = 100
MAX_MD_COLS = 20
MAX_MD_CHARS = 20_000
def _workbook_path(vault: str, path: str) -> Path:
"""Resolve and validate a vault-relative ``.xlsx`` path."""
path = (path or "").strip()
if not path.lower().endswith(".xlsx"):
raise ToolError("Extension attendue : .xlsx", code="invalid_arguments")
try:
root = get_vault_root(vault)
except ServiceError as e:
raise ToolError(e.message, code=e.code, details=e.details) from e
return resolve_safe_path(root, path)
def _map_service_error(e: ServiceError) -> ToolError:
return ToolError(e.message, code=e.code, details=e.details)
@tool(
name="list_xlsx_sheets",
description=(
"List the sheets of an .xlsx workbook with their dimensions "
"(rows x columns) and whether the display caps truncate them. "
"Use before editing to pick the right sheet name."
),
input_model=ListXlsxSheetsInput,
risk=ToolRisk.READ,
requires_vault=True,
)
def list_xlsx_sheets(ctx: ToolContext, params: ListXlsxSheetsInput) -> dict[str, Any]:
"""Return sheet names and extents of the workbook."""
from backend.xlsx_reader import MAX_COLS, MAX_ROWS, _sheet_extent
file_path = _workbook_path(params.vault, params.path)
try:
from openpyxl import load_workbook
wb = load_workbook(str(file_path), read_only=True, data_only=True)
except ServiceError as e:
raise _map_service_error(e) from e
except Exception as e:
raise ToolError(f"Classeur illisible: {e}", code="invalid") from e
try:
sheets = []
for ws in wb.worksheets:
total_rows, total_cols = _sheet_extent(ws)
sheets.append(
{
"name": ws.title,
"total_rows": total_rows,
"total_cols": total_cols,
"truncated": total_rows > MAX_ROWS or total_cols > MAX_COLS,
}
)
return {"vault": params.vault, "path": params.path, "sheets": sheets}
finally:
wb.close()
@tool(
name="xlsx_to_markdown",
description=(
"Read a sheet of an .xlsx workbook as a bounded markdown table "
"(up to 100 rows x 20 columns). Use to inspect spreadsheet data "
"before answering or editing."
),
input_model=XlsxToMarkdownInput,
risk=ToolRisk.READ,
requires_vault=True,
)
def xlsx_to_markdown(ctx: ToolContext, params: XlsxToMarkdownInput) -> dict[str, Any]:
"""Render one sheet as a markdown table for the LLM context."""
from openpyxl import load_workbook
from backend.xlsx_reader import _fmt
file_path = _workbook_path(params.vault, params.path)
try:
wb = load_workbook(str(file_path), read_only=True, data_only=True)
except ServiceError as e:
raise _map_service_error(e) from e
except Exception as e:
raise ToolError(f"Classeur illisible: {e}", code="invalid") from e
try:
if params.sheet:
if params.sheet not in wb.sheetnames:
raise ToolError(
f"Feuille introuvable: {params.sheet}", code="not_found"
)
ws = wb[params.sheet]
else:
ws = wb.active
title = ws.title
rows: list[list[str]] = []
truncated = False
for row in ws.iter_rows(
min_row=1, max_row=MAX_MD_ROWS, max_col=MAX_MD_COLS, values_only=True
):
cells = [_fmt(v) for v in row]
if not any(c.strip() for c in cells):
continue
rows.append(cells)
# Real tail beyond the caps? Probe one row further.
probe = list(
ws.iter_rows(
min_row=MAX_MD_ROWS + 1,
max_row=MAX_MD_ROWS + 1,
max_col=MAX_MD_COLS,
values_only=True,
)
)
if any(any(str(v or "").strip() for v in r) for r in probe):
truncated = True
finally:
wb.close()
lines: list[str] = []
if rows:
header = rows[0]
lines.append("| " + " | ".join(header) + " |")
lines.append("|" + "|".join("---" for _ in header) + "|")
for row in rows[1:]:
lines.append("| " + " | ".join(row) + " |")
table = "\n".join(lines)[:MAX_MD_CHARS]
return {
"vault": params.vault,
"path": params.path,
"sheet": title,
"rows": len(rows),
"cols": max((len(r) for r in rows), default=0),
"truncated": truncated,
"markdown": table,
}
@tool(
name="update_xlsx_cells",
description=(
"Edit cells of an existing .xlsx workbook. ``cells`` maps A1 "
"references to new values (max 500). A value starting with '=' or "
"'@' is stored as TEXT unless allow_formula is set (DDE guard). "
"Editing a workbook carrying features openpyxl cannot rewrite "
"requires force=true (cached formula results, slicers…)."
),
input_model=UpdateXlsxCellsInput,
risk=ToolRisk.WRITE,
requires_vault=True,
)
def update_xlsx_cells(ctx: ToolContext, params: UpdateXlsxCellsInput) -> dict[str, Any]:
"""Wrap the guarded cell-edit service."""
from backend.services.mutations import edit_xlsx_cells
if not params.cells:
raise ToolError("Aucune cellule fournie", code="invalid_arguments")
try:
result = edit_xlsx_cells(
params.vault,
params.path,
params.sheet,
dict(params.cells),
allow_formula=params.allow_formula,
force=params.force,
)
except ServiceError as e:
raise _map_service_error(e) from e
return {
"status": "ok",
"vault": result["vault"],
"path": result["path"],
"sheet": params.sheet,
"cells": len(params.cells),
}
@tool(
name="append_xlsx_rows",
description=(
"Append rows at the end of a sheet of an existing .xlsx workbook. "
"Values are typed like in the viewer (numbers, TRUE/FALSE, FR dates "
"JJ/MM/AAAA). The workbook is rewritten atomically with a backup."
),
input_model=AppendXlsxRowsInput,
risk=ToolRisk.WRITE,
requires_vault=True,
)
def append_xlsx_rows(ctx: ToolContext, params: AppendXlsxRowsInput) -> dict[str, Any]:
"""Append whole rows below the last used row of the sheet."""
from openpyxl import load_workbook
from openpyxl.utils import get_column_letter
from backend.services.mutations import _coerce_xlsx_value, edit_xlsx_cells
if not params.rows:
raise ToolError("Aucune ligne fournie", code="invalid_arguments")
if len(params.rows) > 500:
raise ToolError("Trop de lignes (max 500)", code="invalid_arguments")
file_path = _workbook_path(params.vault, params.path)
try:
wb = load_workbook(str(file_path), read_only=True, data_only=True)
try:
if params.sheet not in wb.sheetnames:
raise ToolError(
f"Feuille introuvable: {params.sheet}", code="not_found"
)
ws = wb[params.sheet]
first_free = (ws.max_row or 0) + 1
finally:
wb.close()
except ServiceError as e:
raise _map_service_error(e) from e
except ToolError:
raise
except Exception as e:
raise ToolError(f"Classeur illisible: {e}", code="invalid") from e
cells: dict[str, Any] = {}
for i, row in enumerate(params.rows):
for j, value in enumerate(row):
if value is None or (isinstance(value, str) and not value.strip()):
continue
ref = f"{get_column_letter(j + 1)}{first_free + i}"
cells[ref] = _coerce_xlsx_value(value)
if not cells:
raise ToolError("Aucune valeur fournie", code="invalid_arguments")
try:
result = edit_xlsx_cells(
params.vault,
params.path,
params.sheet,
cells,
allow_formula=params.allow_formula,
force=params.force,
)
except ServiceError as e:
raise _map_service_error(e) from e
return {
"status": "ok",
"vault": result["vault"],
"path": result["path"],
"sheet": params.sheet,
"rows": len(params.rows),
"first_row": first_free,
}
+3 -2
View File
@@ -22,7 +22,7 @@ Exemples :
from __future__ import annotations
import os
import subprocess
import subprocess # nosec B404
from pathlib import Path
_ROOT = Path(__file__).resolve().parent.parent # racine du dépôt ObsiGate
@@ -34,7 +34,8 @@ _ENV_VAR = "OBSIGATE_VERSION"
def _run_git(args: list[str]) -> str:
"""Run a git command in the repo root; return stdout (stripped) or ''."""
try:
result = subprocess.run(
# argv fixe (git + args internes), sans shell : pas d'injection.
result = subprocess.run( # nosec B404 B603 B607
["git", *args],
cwd=str(_ROOT),
capture_output=True,
+2 -1
View File
@@ -280,7 +280,8 @@ class VaultWatcher:
for observer in self.observers.values():
try:
observer.join(timeout=5)
except Exception: # nosec B110 — best-effort shutdown, ignore failures
# best-effort shutdown, ignore failures (B110) :
except Exception: # nosec B110
pass
self.observers.clear()
logger.info("VaultWatcher stopped")
+28
View File
@@ -0,0 +1,28 @@
"""Shared VaultWatcher handle (ROADMAP #85, tranche 8).
Holder extrait de :mod:`backend.main` sans changement de comportement : le
lifespan de ``main`` y dépose l'instance (``set_watcher``) et l'y reprend à
l'extinction ; le router ``vaults`` la consulte via :func:`get_watcher`
(démarrage/arrêt de surveillance à l'ajout/retrait dynamique de vault,
état dans ``/api/vaults/status``).
"""
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from backend.watcher import VaultWatcher
_watcher: VaultWatcher | None = None
def get_watcher() -> VaultWatcher | None:
"""Return the shared VaultWatcher instance (``None`` if disabled)."""
return _watcher
def set_watcher(watcher: VaultWatcher | None) -> None:
"""Store (or clear) the shared VaultWatcher instance."""
global _watcher
_watcher = watcher
+54 -43
View File
@@ -26,6 +26,7 @@ import json
import logging
import os
import socket
import threading
import uuid
from datetime import datetime, timezone
from pathlib import Path
@@ -144,6 +145,12 @@ def _read_secrets() -> dict:
return {}
# ROADMAP #85 T10a — verrou autour des read-modify-write des deux stores
# (webhooks + secrets) : perte de mises à jour en cas de mutations
# concurrentes.
_lock = threading.RLock()
def _write_secrets(secrets: dict):
WEBHOOK_SECRETS_FILE.parent.mkdir(parents=True, exist_ok=True)
tmp = WEBHOOK_SECRETS_FILE.with_suffix(".tmp")
@@ -156,12 +163,13 @@ def _write_secrets(secrets: dict):
def _store_secret(wh_id: str, secret: str | None) -> None:
secrets = _read_secrets()
if secret:
secrets[wh_id] = secret
else:
secrets.pop(wh_id, None)
_write_secrets(secrets)
with _lock:
secrets = _read_secrets()
if secret:
secrets[wh_id] = secret
else:
secrets.pop(wh_id, None)
_write_secrets(secrets)
def _get_secret(wh: dict) -> str | None:
@@ -189,52 +197,55 @@ def get_webhooks() -> list:
def create_webhook(name: str, url: str, events: list[str], secret: str | None = None) -> dict:
validate_webhook_url(url)
webhooks = _read()
wh_id = str(uuid.uuid4())
wh = {
"id": wh_id,
"name": name,
"url": url,
"events": [e for e in events if e in VALID_EVENTS],
"enabled": True,
"created_at": datetime.now(timezone.utc).isoformat(),
"last_fired_at": None,
}
webhooks.append(wh)
_write(webhooks)
if secret:
_store_secret(wh_id, secret)
with _lock:
webhooks = _read()
wh_id = str(uuid.uuid4())
wh = {
"id": wh_id,
"name": name,
"url": url,
"events": [e for e in events if e in VALID_EVENTS],
"enabled": True,
"created_at": datetime.now(timezone.utc).isoformat(),
"last_fired_at": None,
}
webhooks.append(wh)
_write(webhooks)
if secret:
_store_secret(wh_id, secret)
logger.info(f"Created webhook '{name}' → {url}")
return _public_view(wh)
def update_webhook(wh_id: str, updates: dict) -> dict | None:
webhooks = _read()
for wh in webhooks:
if wh["id"] == wh_id:
if updates.get("url"):
validate_webhook_url(updates["url"])
if "secret" in updates:
_store_secret(wh_id, updates["secret"])
safe_updates = {
k: v for k, v in updates.items()
if k not in ("id", "secret")
}
wh.update(safe_updates)
_write(webhooks)
return _public_view(wh)
with _lock:
webhooks = _read()
for wh in webhooks:
if wh["id"] == wh_id:
if updates.get("url"):
validate_webhook_url(updates["url"])
if "secret" in updates:
_store_secret(wh_id, updates["secret"])
safe_updates = {
k: v for k, v in updates.items()
if k not in ("id", "secret")
}
wh.update(safe_updates)
_write(webhooks)
return _public_view(wh)
return None
def delete_webhook(wh_id: str) -> bool:
webhooks = _read()
new_list = [wh for wh in webhooks if wh["id"] != wh_id]
if len(new_list) == len(webhooks):
return False
_write(new_list)
secrets = _read_secrets()
if secrets.pop(wh_id, None) is not None:
_write_secrets(secrets)
with _lock:
webhooks = _read()
new_list = [wh for wh in webhooks if wh["id"] != wh_id]
if len(new_list) == len(webhooks):
return False
_write(new_list)
secrets = _read_secrets()
if secrets.pop(wh_id, None) is not None:
_write_secrets(secrets)
return True
+570 -15
View File
@@ -3,11 +3,17 @@
Read-only: formulas are shown as their text (``data_only=False``) so a
round-trip through the viewer never depends on Excel's cached values.
Write-side lives in ``backend.services.mutations.edit_xlsx_cells``.
:func:`inspect_workbook` lists the workbook features that an openpyxl
round-trip would drop (#153 A1) so the UI can warn before saving.
"""
from __future__ import annotations
import html
import logging
import re
import zipfile
from datetime import date, datetime
from pathlib import Path
from typing import Any
@@ -15,11 +21,174 @@ from typing import Any
from openpyxl import load_workbook
from openpyxl.utils import get_column_letter
logger = logging.getLogger("obsigate.xlsx_reader")
# ponytail: hard caps bound the rendered grid (500 rows x 40 cols per sheet).
# Raise them, or paginate per sheet, if a real workbook needs more.
MAX_ROWS = 500
MAX_COLS = 40
# #153 A9 — window size served by ``read_sheet_window()`` (lazy per-sheet
# loading). The endpoint is bounded so a single request can never ask for the
# whole workbook back in one JSON payload; the UI pages through the rest.
MAX_WINDOW_ROWS = 1_000
DEFAULT_WINDOW_ROWS = 200
# #153 A1 — workbook parts openpyxl does not re-serialize on load+save.
# Verified against openpyxl 3.1.5: charts, images, drawings and pivot tables
# DO survive the round-trip, so they are deliberately absent from this map.
LOSSY_PARTS: dict[str, tuple[str, ...]] = {
"slicers": ("xl/slicers/", "xl/slicerCaches/", "xl/timelines/"),
"form_controls": ("xl/ctrlProps/", "xl/activeX/"),
"connections": ("xl/queryTables/", "xl/connections.xml"),
"custom_xml": ("customXml/",),
"signature": ("_xmlsignatures/",),
"rich_comments": ("xl/threadedComments/", "xl/persons/"),
"macros": ("xl/vbaProject.bin",),
}
# A formula cell carrying its last computed result: ``<f>…</f><v>…</v>``.
# openpyxl writes an EMPTY ``<v></v>`` itself, hence the ``[^<]`` guard: only a
# non-empty value counts. openpyxl keeps the formula but drops the cached result,
# so any reader using ``data_only=True`` (pandas, converters) sees ``None`` until
# Excel recalculates.
_CACHED_FORMULA_RE = re.compile(rb"<f[ >][^<]*</f>\s*<v>[^<]")
# Sheet XML scanned by the cached-formula probe (CPU guard, like MAX_REPLACE_FILE_BYTES).
_MAX_PROBE_BYTES = 8_000_000
# #153 A5 — ceiling on the text handed to the TF-IDF / semantic index. A workbook
# is a data dump, not prose: indexing every cell would flood the inverted index
# and bury the notes. Sheet names + the first rows are enough to make a
# spreadsheet findable by its headers.
MAX_INDEX_CHARS = 5_000
_INDEX_ROWS_PER_SHEET = 20
MAX_INDEX_SHEETS = 20
# #153 A15 — reading styles is a second (non-read_only) pass on the sheet XML.
# Bounded like everything else: a cell must be INSIDE the rendered window to
# deserve an inline style, so a huge workbook never triggers a huge payload.
# Only data-driven fragments are emitted: the hex values come from the file,
# never from a hardcoded color table.
def _cell_fragments(cell: Any) -> tuple[list[str], str | None]:
"""Inline CSS fragments of one cell plus its horizontal alignment.
Fixed, color-first order: the API contract documents ``color:...`` as the
first fragment of a styled cell. Only data-driven values are emitted —
every hex comes from the workbook itself, never a hardcoded table.
"""
fragments: list[str] = []
font = cell.font
if font and font.color is not None and isinstance(font.color.rgb, str):
# ARGB from the workbook itself — never a hardcoded table.
rgb = font.color.rgb
if len(rgb) == 8 and rgb != "FF000000":
fragments.append(f"color:#{rgb[2:].lower()}")
fill = cell.fill
if fill and fill.fgColor is not None and isinstance(fill.fgColor.rgb, str):
rgb = fill.fgColor.rgb
if len(rgb) == 8 and rgb not in ("00000000", "FFFFFFFF"):
fragments.append(f"background:#{rgb[2:].lower()}")
if font and font.bold:
fragments.append("font-weight:600")
if font and font.italic:
fragments.append("font-style:italic")
fmt = cell.number_format
if fmt and fmt not in ("General", "@"):
# A custom number format is signalled typographically (mono font)
# rather than rendered: the displayed value already carries the
# formatting from _fmt(). Single quotes: the fragment lands inside a
# double-quoted HTML attribute.
fragments.append("font-family:'JetBrains Mono',monospace")
alignment = cell.alignment
align = alignment.horizontal if alignment else None
return fragments, (align if align in ("left", "right", "center") else None)
def _sheet_style_maps(ws: Any) -> tuple[dict[str, str], dict[str, str]]:
"""Flat ``{ref: css}`` and ``{ref: align}`` maps of one worksheet.
The flat string is what the API serves and what the viewer applies
verbatim to ``td.style``; a plain cell is simply absent from the map.
``left`` is the table default and never included. Bounded by
``MAX_ROWS x MAX_COLS`` like the render itself.
"""
styles: dict[str, str] = {}
aligns: dict[str, str] = {}
for row in ws.iter_rows(min_row=1, max_row=MAX_ROWS, max_col=MAX_COLS):
for cell in row:
if cell.value is None and cell.number_format == "General":
continue
fragments, align = _cell_fragments(cell)
if fragments:
styles[cell.coordinate] = ";".join(fragments)
if align and align != "left":
aligns[cell.coordinate] = align
return styles, aligns
def read_sheet_styles(file_path: Path, sheet: str) -> dict[str, dict[str, Any]]:
"""Return ``{ref: {style, align}}`` for the styled cells of one sheet.
``style`` is the flat CSS fragment the viewer applies verbatim and
``align`` the horizontal text-align when it is not the table default.
Normal (non-streaming) load — styles are unavailable in read_only mode;
a failure yields ``{}`` so the viewer falls back to the plain rendering.
"""
meta = read_workbook_meta(file_path).get(sheet, {})
styles_map = meta.get("styles", {})
aligns = meta.get("aligns", {})
out: dict[str, dict[str, Any]] = {}
for ref, css in styles_map.items():
entry: dict[str, Any] = {"style": css}
if ref in aligns:
entry["align"] = aligns[ref]
out[ref] = entry
return out
def read_sheet_merges(file_path: Path, sheet: str) -> list[str]:
"""Return the merged ranges of one sheet as ``A1:C3`` strings."""
try:
# Styles and merges are only fully materialised in normal mode
# (read_only=True leaves merged_cells empty).
wb = load_workbook(str(file_path))
except Exception:
return []
try:
if sheet not in wb.sheetnames:
return []
merged = getattr(wb[sheet], "merged_cells", None)
ranges = getattr(merged, "ranges", None) or []
return [str(r) for r in ranges]
except Exception:
logger.debug("xlsx merges unavailable", exc_info=True)
return []
finally:
wb.close()
def read_sheet_freeze(file_path: Path, sheet: str) -> str:
"""Return the freeze-panes anchor of one sheet ('' when not frozen).
Normal (non-streaming) load: `freeze_panes` is NOT materialised on
ReadOnlyWorksheet in openpyxl 3.1.x — read_only=True always yields ''.
"""
try:
wb = load_workbook(str(file_path))
except Exception:
return ""
try:
if sheet not in wb.sheetnames:
return ""
return str(getattr(wb[sheet], "freeze_panes", None) or "")
except Exception:
return ""
finally:
wb.close()
def _fmt(value: Any) -> str:
if value is None:
@@ -46,7 +215,43 @@ def _trim(grid: list[list[str]]) -> list[list[str]]:
return [row[:width] for row in grid]
def _table(grid: list[list[str]]) -> str:
def _cell_cached(cached: list[list[str]] | None, r: int, c: int) -> str:
"""Return the cached result for a 0-based cell, or ``""``.
The shadow grid is read positionally and may be narrower than the formula
grid (``_trim`` collapses the trailing empty columns of each grid
independently), so every lookup is bounds-checked rather than assumed.
"""
if not cached or r >= len(cached):
return ""
row = cached[r]
return row[c] if c < len(row) else ""
def _table(
grid: list[list[str]],
cached: list[list[str]] | None = None,
row_offset: int = 0,
styles: dict[str, dict[str, Any]] | None = None,
) -> str:
"""Render a grid as an HTML table.
``cached`` is the same grid read with ``data_only=True`` (#153 A12): where a
formula cell still carries its last computed result, it is shown as a
discreet second line (``<span class="xlsx-cached">``) so the user sees the
number Excel last calculated instead of only the formula text. The span
carries ``data-cached-value`` and is titled client-side from
``xlsx.cached_value_title`` — the backend never emits UI text.
``row_offset`` is the number of rows skipped before this grid (#153 A9): the
row numbers and the ``data-cell`` references must stay the real A1
coordinates of the sheet, not of the window.
``styles`` maps A1 references to ``{style, align}`` fragments (#153 A15:
bold, italic, background, alignment) — the backend only reads the
workbook, the fragments are built from it and always data-driven, never
hardcoded colors. A plain ``str`` value is tolerated (legacy callers).
"""
if not grid:
return "<p><em>Feuille vide</em></p>"
n_cols = max(len(row) for row in grid)
@@ -58,29 +263,379 @@ def _table(grid: list[list[str]]) -> str:
]
out += [f"<th>{get_column_letter(c)}</th>" for c in range(1, n_cols + 1)]
out.append("</tr></thead><tbody>")
for r, row in enumerate(grid, start=1):
for r, row in enumerate(grid, start=row_offset + 1):
out.append(f'<tr><th class="xlsx-rownum">{r}</th>')
for c, val in enumerate(row, start=1):
ref = f"{get_column_letter(c)}{r}"
out.append(f'<td data-cell="{ref}">{html.escape(val)}</td>')
meta = (styles or {}).get(ref)
if meta is None:
style_attr = ""
else:
# Legacy callers may still pass a bare CSS string.
if isinstance(meta, str):
meta = {"style": meta}
fragment = meta.get("style", "")
align = meta.get("align")
if align and align not in ("left",):
# left is the table default; only non-default alignments
# need an explicit declaration.
fragment = f"{fragment};text-align:{align}" if fragment else f"text-align:{align}"
style_attr = f' style="{fragment}"' if fragment else ""
# The cached result only makes sense for a formula cell: on a plain
# value cell the two reads are identical and showing both would
# duplicate the text.
shadow = ""
if cached is not None and val.startswith("="):
# `c` is 1-based (A1 notation) and `r` too, while the grid is
# 0-based: translate both.
cval = _cell_cached(cached, r - 1, c - 1)
if cval and cval != val:
# The tooltip is translated client-side from
# `xlsx.cached_value_title`; never hardcode UI text here.
shadow = (
f'<span class="xlsx-cached" data-cached-value="1">'
f"{html.escape(cval)}</span>"
)
out.append(
f'<td data-cell="{ref}"{style_attr}>{html.escape(val)}{shadow}</td>'
)
out.append("</tr>")
out.append("</tbody></table></div>")
return "".join(out)
def render_sheets(file_path: Path) -> list[dict[str, str]]:
"""Return ``[{"name": sheet_title, "html": table_html}, ...]``."""
wb = load_workbook(str(file_path), read_only=True, data_only=False)
def _has_cached_formulas(zf: zipfile.ZipFile) -> bool:
"""True when at least one formula cell still carries its computed value."""
budget = _MAX_PROBE_BYTES
for name in zf.namelist():
if not name.startswith("xl/worksheets/sheet") or not name.endswith(".xml"):
continue
try:
with zf.open(name) as fh:
while budget > 0:
chunk = fh.read(65536)
if not chunk:
break
budget -= len(chunk)
if _CACHED_FORMULA_RE.search(chunk):
return True
except (KeyError, OSError, zipfile.BadZipFile):
continue
return False
def inspect_workbook(file_path: Path) -> list[str]:
"""Return the sorted keys of :data:`LOSSY_PARTS` present in *file_path*.
Read-only inspection of the OPC package (central directory + a bounded scan
of the sheet XML). Never raises: an unreadable or encrypted workbook simply
yields ``[]`` and the save path keeps its current behaviour.
``cached_values`` is a synthetic key: openpyxl keeps the formula but drops
the cached result, so the workbook stays correct once Excel recalculates it.
"""
try:
with zipfile.ZipFile(file_path) as zf:
names = set(zf.namelist())
found = {
key
for key, prefixes in LOSSY_PARTS.items()
if any(name.startswith(prefix) for name in names for prefix in prefixes)
}
if _has_cached_formulas(zf):
found.add("cached_values")
return sorted(found)
except (OSError, zipfile.BadZipFile):
return []
def read_workbook_meta(file_path: Path) -> dict[str, dict[str, Any]]:
"""Return ``{sheet: {styles, aligns, merges, freeze}}`` for every sheet.
One normal (non-streaming) load serves the three A15 metadata maps: the
fragments are the workbook's own values, a failure yields ``{}`` per sheet
so the viewer keeps its plain rendering. Styles are read with
``data_only=False`` — the edited value is the formula, not its result.
"""
try:
wb = load_workbook(str(file_path), data_only=False)
except Exception:
return {}
out: dict[str, dict[str, Any]] = {}
try:
sheets = []
for ws in wb.worksheets:
grid = [
[_fmt(v) for v in row]
for row in ws.iter_rows(
min_row=1, max_row=MAX_ROWS, max_col=MAX_COLS, values_only=True
)
]
sheets.append({"name": ws.title, "html": _table(_trim(grid))})
return sheets
styles, aligns = _sheet_style_maps(ws)
merged = getattr(ws, "merged_cells", None)
ranges = getattr(merged, "ranges", None) or []
out[ws.title] = {
"styles": styles,
"aligns": aligns,
"merges": [str(r) for r in ranges],
"freeze": str(getattr(ws, "freeze_panes", None) or ""),
}
return out
except Exception:
logger.debug("xlsx meta unavailable", exc_info=True)
for t in wb.sheetnames:
out.setdefault(t, {"styles": {}, "aligns": {}, "merges": [], "freeze": ""})
return out
finally:
wb.close()
def render_sheets(file_path: Path) -> list[dict[str, Any]]:
"""Return one dict per sheet: ``{name, html, rows, cols, total_*, truncated}``.
Reads the workbook twice: once with ``data_only=False`` for the formulas
(what the user must edit) and, when any formula carries a cached result
(#153 A12), once with ``data_only=True`` to show what Excel last computed.
The second pass is skipped entirely when the archive holds no cached value,
so the common case still costs a single load.
``total_rows``/``total_cols`` are the dimensions the sheet declares and
``truncated`` says whether the hard caps actually cut it (#153 A8) — the
viewer needs both to stop silently hiding the tail of a sheet.
"""
wb = load_workbook(str(file_path), read_only=True, data_only=False)
try:
formulas = [_sheet_grid(ws) for ws in wb.worksheets]
titles = [ws.title for ws in wb.worksheets]
extents = [_sheet_extent(ws) for ws in wb.worksheets]
finally:
wb.close()
cached: list[list[list[str]]] | None = None
if _has_cached_values(file_path):
cached = _read_cached_grids(file_path, titles)
# #153 A15 — one extra normal-mode load serves the styles/merges/freeze
# metadata of every sheet; the HTML then carries the fragments itself.
meta = read_workbook_meta(file_path)
sheets = []
for i, title in enumerate(titles):
grid = _trim(formulas[i])
# The shadow grid is NOT trimmed independently: _trim drops the
# trailing empty columns of each grid on its own width, which would
# shift every cached value left of its formula. Indexing it
# positionally against the untrimmed grid keeps the two aligned.
shadow = cached[i] if cached is not None and i < len(cached) else None
total_rows, total_cols = extents[i]
sheet_meta = meta.get(title, {})
sheets.append(
{
"name": title,
"html": _table(grid, shadow, styles=sheet_meta.get("styles")),
"rows": len(grid),
"cols": max((len(r) for r in grid), default=0),
"total_rows": total_rows,
"total_cols": total_cols,
# Coverage, not display size: `rows`/`cols` are post-trim (a
# sheet of 3 filled cells in a 500-row block renders 1x1), and
# the client must announce the cap it stopped at, not how many
# cells happen to be non-empty.
"max_rows": MAX_ROWS,
"max_cols": MAX_COLS,
# A sheet is truncated when the caps, not the trailing blanks,
# decided its shape: comparing against the *rendered* size would
# flag every sheet carrying a few empty formatted rows.
"truncated": total_rows > MAX_ROWS or total_cols > MAX_COLS,
"styles": sheet_meta.get("styles", {}),
"aligns": sheet_meta.get("aligns", {}),
"merges": sheet_meta.get("merges", []),
"freeze": sheet_meta.get("freeze", ""),
}
)
return sheets
def read_sheet_window(
file_path: Path,
sheet: str,
offset: int = 0,
limit: int = DEFAULT_WINDOW_ROWS,
) -> dict[str, Any] | None:
"""Return a window of rows of one sheet, or ``None`` if the sheet is unknown.
Backs the lazy per-sheet loading of #153 A9: the viewer asks for the rows
it is about to display instead of shipping every sheet in the initial file
payload. ``offset`` is 0-based; the row numbers and the ``data-cell``
references in the returned ``html`` are the real A1 coordinates of the
sheet, so a window is indistinguishable from a full render.
``limit`` is clamped to :data:`MAX_WINDOW_ROWS`. Raises nothing: an unknown
sheet yields ``None`` and a broken workbook propagates the caller's usual
500.
"""
offset = max(int(offset), 0)
limit = min(max(int(limit), 1), MAX_WINDOW_ROWS)
wb = load_workbook(str(file_path), read_only=True, data_only=False)
try:
if sheet not in wb.sheetnames:
return None
ws = wb[sheet]
total_rows, total_cols = _sheet_extent(ws)
grid = _trim(
_sheet_grid(ws, min_row=offset + 1, max_row=offset + limit)
)
finally:
wb.close()
shadow: list[list[str]] | None = None
# Same A12 rule as the full render: the second read only happens when the
# archive really holds cached results.
if _has_cached_values(file_path):
shadow = _read_cached_window(file_path, sheet, offset, limit)
# #153 A15 — same metadata as the full render, so a lazy window is
# indistinguishable from it (styles in the HTML, merges/freeze for the
# client-side spanning).
meta = read_workbook_meta(file_path).get(sheet, {})
return {
"sheet": sheet,
"offset": offset,
"limit": limit,
"rows": len(grid),
"cols": max((len(r) for r in grid), default=0),
"total_rows": total_rows,
"total_cols": total_cols,
"max_rows": MAX_ROWS,
"max_cols": MAX_COLS,
"truncated": total_rows > MAX_ROWS or total_cols > MAX_COLS,
"has_more": offset + len(grid) < total_rows,
"html": _table(grid, shadow, row_offset=offset, styles=meta.get("styles")),
"styles": meta.get("styles", {}),
"aligns": meta.get("aligns", {}),
"merges": meta.get("merges", []),
"freeze": meta.get("freeze", ""),
}
def _read_cached_window(
file_path: Path, sheet: str, offset: int, limit: int
) -> list[list[str]] | None:
"""``data_only=True`` grid for one window, or ``None`` if unavailable.
Best effort like :func:`_read_cached_grids`: a workbook Excel opens but
openpyxl cannot re-read must still display (formulas only).
"""
try:
wb = load_workbook(str(file_path), read_only=True, data_only=True)
except Exception:
return None
try:
if sheet not in wb.sheetnames:
return None
return _sheet_grid(
wb[sheet], min_row=offset + 1, max_row=offset + limit
)
except Exception:
logger.debug("xlsx cached window unavailable", exc_info=True)
return None
finally:
wb.close()
def _sheet_extent(ws: Any) -> tuple[int, int]:
"""Rows and columns the worksheet declares, never negative.
``max_row``/``max_column`` come from the sheet's dimension record; a
hand-edited file may omit it, hence the defensive coercion.
"""
try:
rows = max(int(getattr(ws, "max_row", 0) or 0), 0)
except (TypeError, ValueError):
rows = 0
try:
cols = max(int(getattr(ws, "max_column", 0) or 0), 0)
except (TypeError, ValueError):
cols = 0
return rows, cols
def _sheet_grid(
ws: Any, min_row: int = 1, max_row: int = MAX_ROWS, max_col: int = MAX_COLS
) -> list[list[str]]:
"""Read a worksheet window into a grid of formatted strings, bounded by the caps."""
return [
[_fmt(v) for v in row]
for row in ws.iter_rows(
min_row=min_row, max_row=max_row, max_col=max_col, values_only=True
)
]
def _has_cached_values(file_path: Path) -> bool:
"""True when the archive holds at least one ``<f>…</f><v>…</v>``."""
try:
with zipfile.ZipFile(file_path) as zf:
return _has_cached_formulas(zf)
except (OSError, zipfile.BadZipFile):
return False
def _read_cached_grids(
file_path: Path, titles: list[str]
) -> list[list[list[str]]] | None:
"""Read every sheet with ``data_only=True`` (what Excel last computed).
Best effort: returns ``None`` on any failure so the viewer falls back to the
formula-only rendering. A workbook Excel opens but openpyxl cannot re-read
must still display.
"""
try:
wb = load_workbook(str(file_path), read_only=True, data_only=True)
except Exception:
return None
try:
grids = [_sheet_grid(ws) for ws in wb.worksheets]
if [ws.title for ws in wb.worksheets] != titles:
return None
return grids
except Exception:
logger.debug("xlsx cached values unavailable", exc_info=True)
return None
finally:
wb.close()
def extract_indexable_text(file_path: Path) -> str:
"""Return searchable text for the TF-IDF / semantic index (#153 A5).
Sheet names plus the first :data:`_INDEX_ROWS_PER_SHEET` rows of each
sheet, capped at :data:`MAX_INDEX_CHARS`. Rows are tab-joined so a search
for a header matches the sheet it belongs to.
Never raises: a corrupt, encrypted or unsupported workbook yields ``""`` so
the file still gets indexed by name (same contract as :func:`inspect_workbook`).
"""
chunks: list[str] = []
budget = MAX_INDEX_CHARS
try:
wb = load_workbook(str(file_path), read_only=True, data_only=True)
except Exception:
# Encrypted (BadZipFile) or not a real workbook: name-only indexing.
return ""
try:
for ws in wb.worksheets[:MAX_INDEX_SHEETS]:
if budget <= 0:
break
# The sheet title alone is a strong signal ("Recettes", "Budget").
block = [ws.title]
for row in ws.iter_rows(
min_row=1, max_row=_INDEX_ROWS_PER_SHEET, max_col=MAX_COLS, values_only=True
):
cells = [_fmt(v) for v in row]
# Skip blank rows instead of emitting runs of tabs.
if not any(c.strip() for c in cells):
continue
block.append("\t".join(cells).rstrip())
text = "\n".join(block)
chunks.append(text[:budget])
budget -= len(text)
except Exception:
# Truncated but still useful: keep whatever was collected.
pass
finally:
wb.close()
return "\n".join(c for c in chunks if c).strip()
+1 -1
View File
@@ -2626,7 +2626,7 @@ dependencies = [
[[package]]
name = "obsigate-desktop"
version = "2.27.9"
version = "2.37.0"
dependencies = [
"chrono",
"env_logger",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "obsigate-desktop"
version = "2.27.9"
version = "2.37.0"
description = "ObsiGate Desktop — Porte d'entrée native pour vos vaults Obsidian"
authors = ["Bruno Charest"]
edition = "2021"
+1 -1
View File
@@ -1,7 +1,7 @@
{
"$schema": "https://raw.githubusercontent.com/nicedoc/obsigate/main/desktop/tauri.conf.schema.json",
"productName": "ObsiGate",
"version": "2.27.9",
"version": "2.37.0",
"identifier": "com.obsigate.desktop",
"build": {
"frontendDist": "../frontend",
+6 -1
View File
@@ -53,7 +53,12 @@ services:
- OBSIGATE_AUTH_ENABLED=true
- OBSIGATE_ADMIN_USER=admin
# OBSIGATE_ADMIN_PASSWORD → .env
# OBSIGATE_SECURE_COOKIES=true # si derrière reverse proxy HTTPS
# OBSIGATE_SECURE_COOKIES : auto par défaut (Secure si https, sinon
# pas de flag) — forcer à true uniquement si le proxy termine TLS
# sans X-Forwarded-Proto (avec TRUST_PROXY, l'auto suffit).
# Reverse proxy devant l'app : IPs d'audit réelles (BUG-030) et
# X-Forwarded-Proto honoré pour les cookies Secure (auto).
- OBSIGATE_TRUST_PROXY=true
- OLLAMA_BASE_URL=http://ollama:11434/v1
- OLLAMA_MODEL=qwen2.5-coder:1.5b
env_file:
+3 -3
View File
@@ -6,7 +6,7 @@ vaults Obsidian et raccourcis essentiels.
> **Public :** tous les utilisateurs · **Durée de lecture :** ~10 min
> **Voir aussi :** [Déploiement Docker](./DEPLOIEMENT_DOCKER.md) ·
> [Recherche, PDF & Excalidraw](./RECHERCHE_PDF_EXCALIDRAW.md) ·
> [Recherche, PDF, Excel & Excalidraw](./RECHERCHE_PDF_EXCALIDRAW.md) ·
> [API REST](./API_REST.md)
---
@@ -185,7 +185,7 @@ des **onglets** (avec possibilité de vue multi-panneaux / split view).
La recherche est un point fort d'ObsiGate : index inversé TF-IDF, stemming
français, normalisation des accents, facettes et pagination. La syntaxe complète
(`tag:`, `#`, `vault:`, `title:`, `path:`, `ext:`, phrases exactes) est décrite
dans le [Guide Recherche, PDF & Excalidraw](./RECHERCHE_PDF_EXCALIDRAW.md).
dans le [Guide Recherche, PDF, Excel & Excalidraw](./RECHERCHE_PDF_EXCALIDRAW.md).
Démarrage rapide :
@@ -234,7 +234,7 @@ Voir [PWA & mode hors-ligne](./PWA_HORS_LIGNE.md).
| Objectif | Guide |
|---|---|
| Mieux chercher, lire PDF et Excalidraw | [Recherche, PDF & Excalidraw](./RECHERCHE_PDF_EXCALIDRAW.md) |
| Mieux chercher, lire PDF/Excel et Excalidraw | [Recherche, PDF, Excel & Excalidraw](./RECHERCHE_PDF_EXCALIDRAW.md) |
| Utiliser l'IA intégrée | [Assistant IA & Forge](./ASSISTANT_IA_FORGE.md) |
| Éditer à plusieurs | [Édition & collaboration](./COLLABORATION.md) |
| Sécuriser l'accès | [Authentification & sécurité](./AUTHENTIFICATION_SECURITE.md) |
+1 -1
View File
@@ -15,7 +15,7 @@ captures conceptuelles).
| Guide | Public | Contenu |
|---|---|---|
| 🚀 [Prise en main](./PRISE_EN_MAIN.md) | Tous | Premier lancement, interface, navigation, vaults, raccourcis |
| 🔍 [Recherche, PDF & Excalidraw](./RECHERCHE_PDF_EXCALIDRAW.md) | Tous | Syntaxe de requête, recherche sémantique, lecteur PDF, diagrammes |
| 🔍 [Recherche, PDF, Excel & Excalidraw](./RECHERCHE_PDF_EXCALIDRAW.md) | Tous | Syntaxe de requête, recherche sémantique, lecteurs PDF/Excel, diagrammes |
| 🤖 [Assistant IA & Forge](./ASSISTANT_IA_FORGE.md) | Tous | Fournisseurs, éditeur IA, BooksLM, Forge, commandes `@` / `/` |
| 📝 [Édition & collaboration](./COLLABORATION.md) | Tous | Édition simultanée, curseurs distants, persistance |
| 📱 [PWA & mode hors-ligne](./PWA_HORS_LIGNE.md) | Tous | Installation PWA, cache, file de synchronisation, notifications |
+80 -4
View File
@@ -1,4 +1,4 @@
# 🔍 Guide Recherche, PDF & Excalidraw
# 🔍 Guide Recherche, PDF, Excel & Excalidraw
ObsiGate va au-delà de la simple lecture : recherche puissante, rendu des
documents riches (PDF, diagrammes) et indexation de leur contenu pour que tout
@@ -131,7 +131,83 @@ curl "http://localhost:2020/api/file/Recettes/pdf/info?path=menu.pdf"
---
## 6. Diagrammes Excalidraw
## 6. Tableurs Excel (XLSX)
### Affichage et édition
Un fichier `.xlsx` s'ouvre dans une visionneuse dédiée : un tableau par
feuille, des onglets pour naviguer entre elles, les en-têtes A1/B1 et les
numéros de ligne. Chaque cellule est modifiable directement (clic), `Entrée`
valide, `Échap` annule la saisie. **Enregistrer** envoie les cellules
modifiées à `PUT /api/file/{vault}/xlsx/save` : une sauvegarde par feuille,
avec **backup automatique** du fichier avant écriture, et une écriture
**atomique** (le classeur n'est jamais laissé à moitié écrit).
### Avertissement avant enregistrement
Certains classeurs contiennent des éléments qu'ObsiGate ne sait pas
réécrire : **valeurs calculées** mises en cache par Excel, segments
(slicers), chronologies, contrôles de formulaire, connexions/requêtes,
XML personnalisé, signature numérique, commentaires enrichis, macros.
L'ouverture affiche alors un bandeau qui les liste, et la première
sauvegarde demande confirmation. Si vous refusez, rien n'est écrit.
> Les **graphiques, images et tableaux croisés** sont, eux, bien conservés.
### Formules
Par sécurité, une valeur saisie commençant par `=` ou `@` est **stockée comme
texte** (une formule injectée s'exécuterait à l'ouverture du fichier dans
Excel). Le bouton `f(x)` de la barre d'outils active les vraies formules pour
la session en cours.
```bash
curl -X PUT "http://localhost:2020/api/file/Recettes/xlsx/save?path=budget.xlsx" -H "Content-Type: application/json" -d '{"sheet": "Budget", "cells": {"B1": "250"}, "allow_formula": false, "force": false}'
```
- `allow_formula` : `true` pour écrire une vraie formule (`=B1*2`).
- `force` : `true` pour enregistrer malgré les éléments non préservés
(sinon l'API répond **409** `xlsx_lossy_content`).
- Deux sauvegardes simultanées sur le même fichier : la seconde reçoit
**409** `conflict` au lieu d'écraser la première.
### Feuilles volumineuses et lecture par fenêtres
Le rendu est plafonné à **500 lignes × 40 colonnes** par feuille. Quand
une feuille dépasse ce plafond, un bandeau **« Feuille tronquée »**
l'annonce explicitement (par exemple « 500 lignes affichées sur 520 »)
au lieu de présenter une table courte comme complète — le classeur,
lui, n'est jamais modifié. La ligne d'en-têtes de colonnes reste
visible pendant le défilement vertical.
Côté API, `GET /api/file/{vault}/xlsx/sheet` sert une feuille **par
fenêtres de lignes**, y compris au-delà du plafond d'affichage — les
coordonnées A1 renvoyées sont celles de la feuille réelle :
```bash
curl "http://localhost:2020/api/file/Recettes/xlsx/sheet?path=budget.xlsx&sheet=Budget&offset=500&limit=200"
```
- `offset` : première ligne renvoyée (0-based) ; `limit` : nombre de
lignes (1 à 1 000 par requête).
- La réponse porte `total_rows`, `truncated` et `has_more` pour paginer.
- Erreurs : **404** si la feuille n'existe pas, **415** si le fichier
n'est pas un `.xlsx`.
### Limites
- L'affichage intégré démarre à **500 lignes × 40 colonnes** par feuille ;
sous une feuille plus grande, le bouton **« Charger la suite »** (ou le
défilement vers le bas du tableau) ajoute les lignes suivantes par
fenêtres de 500 — elles deviennent aussitôt éditables et
sauvegardables.
- Styles, formats de nombre, cellules fusionnées et volets figés ne sont pas
rendus.
- Formats non gérés : `.xls`, `.xlsm` (macros), `.ods`.
---
## 7. Diagrammes Excalidraw
Les fichiers `.excalidraw` et `.excalidraw.md` (dont le format compressé du
**plugin Obsidian Excalidraw**) s'ouvrent dans un **éditeur visuel Excalidraw
@@ -147,7 +223,7 @@ Fiche technique : [`features/excalidraw.md`](../features/excalidraw.md).
---
## 7. Autres contenus riches
## 8. Autres contenus riches
### Mermaid
@@ -182,7 +258,7 @@ curl -X POST "http://localhost:2020/api/attachments/rescan/Recettes"
---
## 8. Dépannage
## 9. Dépannage
| Symptôme | Piste |
|---|---|
+20 -3
View File
@@ -14,7 +14,7 @@
- **Projet** : ObsiGate — Porte d'entrée web pour vaults Obsidian
- **Stack** : Python 3.11+ (backend FastAPI) · JavaScript/Vanilla (frontend) · Tauri/Rust (desktop)
- **Dernière mise à jour** : 2026-09-24
- **Dernière mise à jour** : 2026-09-27
---
@@ -188,13 +188,20 @@ Avant de corriger quoi que ce soit, un agent IA doit :
| *BUG-076* | [🟡 IMPORTANT] Assistant IA : après une action de l'agent, l'arborescence et le document ouvert ne sont pas rafraîchis dynamiquement | 🟢 corrigé | P1 | 📱 frontend | IA | `frontend/js/bookslm.js` | Mode agent : créer/supprimer un fichier ou dossier, modifier le document ouvert → l'UI ne bouge pas | `frontend/js/bookslm.js` : `MUTATING_TOOLS`/`FILE_WRITE_TOOLS`, refresh d'arborescence débouncé sur event `tool`, `_notifyFileWritten` étendu (xlsx/docx/csv/pdf). Tests : `tests/frontend/ai.test.mjs`, `tests/frontend/editor-inline.test.mjs` | Aucun refresh explicite sur les événements `tool` mutateurs (repose uniquement sur le watcher SSE) ; `_notifyFileWritten` ignore les créations de documents (xlsx/docx/csv/pdf) |
| *BUG-077* | [🟡 IMPORTANT] Assistant IA : aucun bouton « Stop » pour arrêter l'exécution de l'agent à tout moment | 🟢 corrigé | P1 | 📱 frontend | IA | `frontend/js/bookslm.js` | Mode agent : lancer une longue tâche → le bouton Envoyer est désactivé, impossible d'arrêter (seule la fermeture du panneau abort) | `frontend/js/bookslm.js` + `frontend/style.css` : bouton d'envoi → Stop (`_syncSendButton`/`_stopGeneration`/`_markStopped`), i18n `ai.stop`/`ai.stopped`. Tests : `tests/frontend/ai.test.mjs` (+2) | `_abortCtrl` n'est déclenché que par `close()` ; aucun signal d'arrêt côté client pendant le stream |
| *BUG-078* | [🟡 IMPORTANT] Fichiers de code : la coloration syntaxique (highlight.js) disparaît — les feuilles de thème sont basculées à partir de la **clé** de thème au lieu du **mode** | 🟢 corrigé | P1 | 📱 frontend | IA | `frontend/js/themes.js`, `frontend/js/ui.js`, `tests/frontend/unit.test.mjs` | Ouvrir un fichier `.py`/`.sh`/`.ps1`/`.yml` : le code s'affiche en texte brut, sans couleurs | `frontend/js/themes.js` : `applyTheme` bascule `hljs-theme-dark`/`hljs-theme-light` selon le **mode** (`isDark`). `frontend/js/ui.js` : `initTheme`/`applyTheme` résolvent le mode persisté (`obsigate-theme-mode`) au lieu de traiter la clé (`defaut-obsigate`) comme un mode. Test : `unit.test.mjs` (+1). | Les deux feuilles étaient désactivées car `defaut-obsigate !== "dark"` et `!== "light"` ; résultat **non déterministe** selon l'ordre `UI.initTheme()` (clé) / `Sync.init()` → `themes.initThemes()` (mode). Vérifié Playwright : 5/5 chargements colorés (`.py`), sépia/contraste élevé sur la palette claire |
| *BUG-081* | `GET /api/auth/mfa/status` → 500 quand l'auth est désactivée (`user` None, `AttributeError` sur `user.get`) | 🟢 corrigé | P1 | ⚙️ backend | IA | `backend/auth/router.py::mfa_status`, `tests/test_mfa.py` | Auth désactivée : `curl http://127.0.0.1:2029/api/auth/mfa/status` → 500 (reproduit live 2026-09-27) | Garde `user is None` → payload MFA désactivé (`mfa_enabled: false`, `totp_enabled: false`, `webauthn_credentials: 0`) ; test `TestMfaStatusAuthDisabled` (échoue en 500 sans le correctif). Vérifié : `test_mfa.py` 32 passed, ruff/mypy 0 | `require_auth` laisse passer le pseudo-user anonymous, `get_user(username)` → None non gardé. Trouvé via les logs E2E pendant BUG-080 |
| *BUG-079* | `GET /api/diagnostics` → 500 « dictionary changed size during iteration » (stats d'index) | 🟢 corrigé | P1 | ⚙️ backend | IA | `backend/main.py` | Charger la page de diagnostic pendant une indexation : `GET /api/diagnostics` → 500 | `backend/main.py` (`api_diagnostics`) : snapshot avant itération — `list(index.items())` et `inv.word_index.copy()` (copie C atomique sous le GIL) ; test de non-régression `tests/test_api_main.py::TestConfig::test_diagnostics_concurrent_index_writes` | Le handler itérait les dicts en direct alors que l'indexeur les modifiait depuis un autre thread (rebuild initial dans `_search_executor`, hooks incrémentaux `add_document`/`remove_document`) → `RuntimeError` dans le générateur → 500. Test déterministe (`RaceDict` fait grossir le dict en cours d'itération) : échoue sans le correctif, passe avec. Vérifié : pytest 1305 passed / 6 skipped, ruff 0, mypy 0 |
| *BUG-084* | Index inversé : la suppression d'une vault y laisse des documents fantômes (résultats pour une vault inexistante) | 🟢 corrigé | P1 | ⚙️ backend | IA | `backend/indexer.py::remove_vault_from_index`, `backend/search.py::_remove_doc_internals` | Supprimer une vault configurée, puis chercher un terme contenu dans ses fichiers → les résultats la concernent encore | `remove_vault_from_index()` déclenche `_on_index_change('remove', …)` pour chaque fichier de la vault ; `_remove_doc_internals()` supprime la clé `vault_docs` dont le set devient vide (`defaultdict` : une lecture la recréait). Test `tests/test_search_advanced.py::TestVaultRemovalPurgesInvertedIndex` (contre-preuve : échoue sans le correctif) | Trouvé pendant la relecture de `plan.md` (étape 6 déjà livrée). Mesuré : 8 documents fantômes sur 8 après suppression de la vault de test (`postings`, `doc_info`, `doc_vault`, `vault_docs`) ; seul un reindex manuel les effaçait. Vérifié : `test_search_advanced.py` 27 passed, ruff/mypy 0, suite complète 1374 passed / 6 skipped |
| *BUG-085* | Édition d'un `.xlsx` : les valeurs calculées en cache disparaissent du classeur (et tout lecteur `data_only=True` voit `None`) | 🟢 corrigé | P1 | tableur Excel | IA | `backend/xlsx_reader.py::inspect_workbook`, `backend/services/mutations.py::edit_xlsx_cells`, `backend/routers/files_read.py`, `backend/routers/files_write.py`, `frontend/js/viewer.js::renderXlsxViewer` | Ouvrir un classeur contenant `=B1*2` (avec sa valeur calculée) → éditer une cellule → le `<v>` disparaît du XML de la feuille | `LOSSY_PARTS` + sonde `<f>…</f><v>[^<]` ; la lecture renvoie `xlsx_lossy_features` ; `PUT xlsx/save` refuse sans `force` (**409** `xlsx_lossy_content`) ; bandeau + confirmation UI puis reprise `force: true`. Tests : `TestXlsxLossyGuard` (5) + `xlsx-viewer.test.mjs` (10) + `tests/e2e/xlsx-viewer.spec.js` (3) | #153 A1. Périmètre réel vérifié sur openpyxl 3.1.5 : graphiques, images, dessins **et** TCD survivent au round-trip ; les pertes sont valeurs en cache, slicers/chronologies, contrôles de formulaire, connexions/requêtes, custom XML, signature, commentaires enrichis, macros. Vérifié : `test_xlsx_viewer.py` 31 passed, suite 1390 passed / 6 skipped, ruff/mypy 0, E2E 3/3 |
| *BUG-086* | Édition d'un `.xlsx` : `wb.save()` écrit en place, un plantage laisse un classeur corrompu | 🟢 corrigé | P1 | tableur Excel | IA | `backend/services/mutations.py::edit_xlsx_cells` | Simuler un `OSError` pendant `Workbook.save` → le fichier d'origine est tronqué | Écriture atomique : `wb.save(<nom>.<pid>.tmp)` puis `os.replace()` ; `.tmp` supprimé sur échec ; le backup `.bak` reste inchangé. Test : `TestXlsxAtomicWrite::test_failed_save_keeps_the_original` (octets identiques après échec) + `test_no_tmp_left_after_a_successful_save` | #153 A2. Le fichier temporaire a un suffixe `.tmp` → ignoré par le watcher (`_is_relevant` ne retient que les extensions supportées). Vérifié : cf. BUG-085 |
| *BUG-087* | Édition d'un `.xlsx` concurrente (deux onglets, agent IA + viewer) : read-modify-write sans verrou, le dernier écrivain gagne silencieusement | 🟢 corrigé | P1 | tableur Excel | IA | `backend/services/mutations.py::_xlsx_write_lock` | Deux `PUT xlsx/save` simultanés sur le même fichier → une écriture est écrasée sans trace | Verrou par chemin (registre + garde, timeout 15 s) autour du cycle load → edit → `os.replace` ; attente dépassée → **409** `conflict`. L'endpoint est devenu `def` (sync) pour que l'attente s'exécute dans le threadpool et ne bloque pas la boucle d'événements. Test : `TestXlsxWriteLock` (2) | #153 A3. Verrou en mémoire, par processus : protège les cas d'un même serveur (le cas desktop/Tauri). Vérifié : cf. BUG-085 |
| *BUG-088* | Injection de formule dans un `.xlsx` : une saisie `=cmd\|'/c calc'!A1` est stockée comme formule et s'exécute à l'ouverture dans Excel (DDE) | 🟢 corrigé | P0 | tableur Excel / sécurité | IA | `backend/services/mutations.py::_write_cell`, `backend/routers/files_write.py`, `frontend/js/viewer.js::renderXlsxViewer` | `PUT /api/file/V/xlsx/save` avec `{"sheet": "S", "cells": {"A1": "=1+1"}}` → la cellule sort en `data_type == "f"` | `cell.data_type = "s"` après affectation : le texte est stocké comme chaîne, aucun `<f>` n'est écrit. Opt-in via `allow_formula: true` (endpoint) et le bouton `f(x)` de la visionneuse (session, jamais persisté). Test : `TestXlsxFormulaGuard` (4) + `xlsx-viewer.test.mjs` (toggle) | #153 A4. `+`/`-` ne sont pas neutralisés : ils sont déjà convertis en nombre par `_coerce_xlsx_value`. Le handler global `ServiceError` expose désormais `code` + `details` (le client en a besoin pour le 409), et `api()` (frontend) les propage sur l'Error. Vérifié : cf. BUG-085 |
| *BUG-089* | Un reindex manuel ne reconstruisait pas l'index inversé : la recherche TF-IDF continuait de servir un index périmé | 🟢 corrigé | P1 | ⚙️ backend / recherche | IA | `backend/indexer.py::reload_index`, `backend/indexer.py::reload_single_vault`, `backend/search.py` | Modifier le contenu d'un fichier, puis `GET /api/index/reload` → la recherche renvoie encore l'ancien contenu (ou rien pour un fichier nouveau) | `reload_index()` / `reload_single_vault()` appellent `init_inverted_index()` après le rebuild (le remplacement wholesale d'une entrée de vault n'émet pas les notifications incrémentales). En prime, `backend/search.py` lisait l'index via `from backend.indexer import index` (liaison **par valeur** du dict) : un `importlib.reload(backend.indexer)` recréait le dict côté indexer tandis que la recherche écrivait encore dans l'ancien — l'index inversé n'indexait alors plus rien. Tous les accès passent désormais par `_indexer.index`. Contre-preuve : `TestXlsxSearchable::test_search_finds_a_word_stored_in_a_cell` échoue sans le correctif | #153 A5. Trouvé en écrivant le test de recherche d'A5 : il passait isolément et échouait en suite complète selon l'ordre. Le reload incrémental par fichier (watcher, edition) n'est pas concerné : il passe par le hook `_on_index_change`. Vérifié : suite 1402 passed / 6 skipped, ruff/mypy 0 |
| *BUG-090* | Troncature silencieuse d'une feuille `.xlsx` au-delà de 500 lignes × 40 colonnes : l'utilisateur voit une table courte sans aucun indice que la suite existe | 🟢 corrigé | P1 | tableur Excel / UX | IA | `backend/xlsx_reader.py::render_sheets`, `backend/routers/files_read.py`, `frontend/js/viewer.js::renderXlsxViewer`, `frontend/style.css` | Ouvrir `test_vault/sample-xlsx-large.xlsx` (520 lignes) → la feuille s'arrête à la ligne 500 sans aucun message | `render_sheets()` renvoie désormais `total_rows`/`total_cols` (dimensions déclarées par la feuille), `max_rows`/`max_cols` (plafonds du moteur) et `truncated` ; la visionneuse affiche un bandeau « Feuille tronquée — 500 lignes affichées sur 520 » (i18n `xlsx.truncated_*` FR/EN, axe des colonnes inclus). Contre-preuve : neutraliser `truncated` → `TestXlsxTruncationNotice` (2 tests) échoue | #153 A8/R5. La ligne d'en-têtes est aussi `sticky` au défilement vertical (`thead th { top: 0 }` + `top: auto` sur les numéros de ligne pour éviter l'empilement en haut à gauche). L'endpoint `GET …/xlsx/sheet` (#153 A9) sert les fenêtres au-delà du plafond, mais le chargement paresseux complet (défilement virtuel, « charger tout ») reste à faire — le bandeau dit la vérité en attendant. Vérifié : `test_xlsx_viewer.py` 58 passed, E2E 7/7 (dont 3 nouveaux), suite 1417 passed / 6 skipped, ruff/mypy 0, i18n parity |
### TODOs techniques (améliorations / nouvelles tâches)
| # | Titre | Statut | Priorité | Scope | Assigné | Zone (fichier) | Cmd de repro | Correctif / Commit | Notes |
|---|---|---|---|---|---|---|---|---|---|
| *(exemple)* TODO-002 | Rendre l'index inversé incrémental (40k+ fichiers) | 🔴 ouvert | P1 | ⚙️ backend | IA | `backend/indexer.py`, `backend/search.py` | Recherche sur très gros vault | — | Exemple à remplacer. Cf. plan.md |
| *(À remplir)* | | | | | | | | | |
---
@@ -207,6 +214,9 @@ Avant de corriger quoi que ce soit, un agent IA doit :
| Date | ID(s) traité(s) | Action | Fichiers modifiés | Résumé | Statut après |
|---|---|---|---|---|---|
| 2026-09-28 | BUG-090 (#153 A8 + A9) | Correction + feature | `backend/xlsx_reader.py`, `backend/routers/files_read.py`, `backend/schemas.py`, `backend/openapi_docs.py`, `frontend/js/viewer.js`, `frontend/style.css`, `frontend/locales/{fr,en}.json`, `tests/test_xlsx_viewer.py`, `tests/frontend/xlsx-viewer.test.mjs`, `tests/e2e/xlsx-viewer.spec.js`, `test_vault/sample-xlsx-large.xlsx` | **La troncature d'une feuille est annoncée et les lignes cachées restent accessibles** : (BUG-090/A8) `render_sheets()` renvoie `total_rows`/`total_cols`/`max_rows`/`max_cols`/`truncated`, la visionneuse affiche un bandeau « Feuille tronquée » (i18n FR/EN, axes lignes et colonnes) et la ligne d'en-têtes devient `sticky` (`top: auto` sur les numéros de ligne pour éviter l'empilement) ; (A9) `GET /api/file/{vault}/xlsx/sheet?sheet=&offset=&limit=` (`XlsxSheetWindowResponse`, plafond 1 000 lignes/requête, 404 feuille inconnue, 415 non-xlsx) sert une fenêtre avec les **vraies** coordonnées A1 et le `has_more` de pagination. Contre-preuves : neutraliser `truncated` → 2 tests échouent ; neutraliser l'offset → 3 tests échouent. Vérifié : `test_xlsx_viewer.py` 58 passed, xlsx-viewer.test.mjs 14/14, E2E 7/7 (3 nouveaux + fixture `sample-xlsx-large.xlsx` 520 lignes), suite 1417 passed / 6 skipped, ruff 0, mypy 0, i18n parity, validate-imports 40 modules | 🟢 corrigé (en attente vérif utilisateur) |
| 2026-09-28 | BUG-089 (#153 A5, A10, A12) | Correction | `backend/xlsx_reader.py`, `backend/indexer.py`, `backend/search.py`, `backend/services/mutations.py`, `frontend/js/viewer.js`, `frontend/style.css`, `frontend/locales/{fr,en}.json`, `tests/test_xlsx_viewer.py` | **Les tableurs deviennent visibles ettypés** : (A5) `extract_indexable_text()` indexe noms de feuilles + 20 premières lignes (plafond 5 k caractères) dans le TF-IDF et la recherche sémantique — un mot tapé dans une cellule rend le fichier trouvable ; (A10) `_coerce_xlsx_value()` reconnaît désormais les booléens (`TRUE`/`FAUX`/`OUI`/`NON`) et les dates FR `JJ/MM/AAAA` (jour-first : `01/02/2026` = 1er février), symétrique avec l'affichage ; (A12) la valeur calculée en cache s'affiche sous la formule (`<span class="xlsx-cached">`, 2ᵉ lecture `data_only=True` uniquement si l'archive contient un `<v>`), info-bulle traduite via `xlsx.cached_value_title` FR/EN. (BUG-089) un reindex manuel reconstruisait mal l'index inversé et `backend/search.py` lisait l'index par valeur. Contre-preuves vérifiées pour A5, A10 et A12. Vérifié : `test_xlsx_viewer.py` 43 passed, suite 1402 passed / 6 skipped, ruff 0, mypy 0, i18n parity, validate-imports 40 modules, xlsx-viewer.test.mjs 10/10 | 🟢 corrigé (en attente vérif utilisateur) |
| 2026-09-27 | BUG-085 → BUG-088 (#153 A1-A4) | Correction | `backend/xlsx_reader.py`, `backend/services/mutations.py`, `backend/routers/files_read.py`, `backend/routers/files_write.py`, `backend/schemas.py`, `backend/main.py`, `frontend/js/viewer.js`, `frontend/js/auth.js`, `frontend/style.css`, `frontend/locales/{fr,en}.json`, `frontend/sw.js`, `tests/test_xlsx_viewer.py`, `tests/frontend/xlsx-viewer.test.mjs`, `tests/e2e/xlsx-viewer.spec.js`, `test_vault/sample-xlsx-lossy.xlsx`, `.gitea/workflows/ci.yml` | **Garde-fous d'écriture des classeurs Excel** : (BUG-085) `inspect_workbook()` détecte ce qu'un round-trip openpyxl perd (valeurs calculées, slicers, contrôles, connexions, custom XML, signature) → la lecture expose `xlsx_lossy_features`, la visionneuse affiche une bannière et `PUT xlsx/save` refuse sans `force` (**409** `xlsx_lossy_content`, confirmation explicite puis reprise) ; (BUG-086) écriture atomique `.tmp` + `os.replace` ; (BUG-087) verrou par fichier (409 `conflict`, endpoint sync pour le threadpool) ; (BUG-088) une saisie `=`/`@` est stockée en texte (`data_type = "s"`), sauf opt-in `allow_formula` / bouton `f(x)`. Le handler `ServiceError` expose désormais `code` + `details` et `api()` les propage. Périmètre de perte revalidé empiriquement sur openpyxl 3.1.5 (graphiques, images et TCD sont préservés). Vérifié : `test_xlsx_viewer.py` 31 passed, suite 1390 passed / 6 skipped, ruff/mypy 0, validate-imports 40 modules, xlsx-viewer.test.mjs 10/10, E2E 3/3 | 🟢 corrigé (en attente vérif utilisateur) |
| *(exemple)* 2026-06-15 | BUG-001 | Correction | `frontend/app.js` | Réécriture de `renderFile()` pour préserver le DOM dashboard | 🟢 corrigé (en attente vérif) |
| 2026-09-09 | BUG-001, BUG-002 | Correction | `backend/main.py`, `frontend/excalidraw-editor.html`, `tests/test_pdf_stream.py` | BUG-001: Content-Disposition RFC 5987 (nom PDF accentué ne casse plus l'en-tête → plus de 500). BUG-002: suppression alias esm.sh (408 jotai) + React 19 cohérent + prop `excalidrawAPI` → Loading masqué, save OK. Vérifié: 534 tests backend verts + E2E navigateur. | 🟢 corrigé (en attente vérif utilisateur) |
| 2026-09-11 | BUG-003, BUG-004 | Correction | `backend/{main,indexer,export,pdf_reader,bookslm_routes}.py`, `backend/auth/router.py`, `.gitea/workflows/ci.yml`, `README.md`, `README.fr.md` | BUG-003: 33 erreurs mypy corrigées (annotations, gardes `None`, import `PROVIDERS` manquant → bug latent) + étape CI mypy rendue bloquante. BUG-004: lien `README.md` → `docs/CONTRIBUTING.md`. Vérifié: mypy 0 erreur, ruff OK, pytest 728 passed, frontend OK. | 🟢 corrigé (en attente vérif utilisateur) |
@@ -273,6 +283,11 @@ Avant de corriger quoi que ce soit, un agent IA doit :
| 2026-09-24 | #115, #117, BUG-078 | Feature + correction | `frontend/js/themes.js`, `frontend/js/ui.js`, `frontend/js/viewer.js`, `frontend/js/config.js`, `frontend/index.html`, `frontend/style.css`, `frontend/popout.html`, `frontend/locales/{fr,en}.json`, `frontend/icons/avatar/*` (nouveau), `tests/frontend/unit.test.mjs`, `tests/frontend/toolbar-order.test.mjs`, `tests/frontend/settings-order-avatar.test.mjs`, `docs/features/viewer-toolbar-highlight-avatars.md` (nouvelle), `docs/ROADMAP.md`, `CHANGELOG.md` | **#115** barre d'outils de lecture épinglée : `viewer.js`/`popout.html` sortent `.file-actions` de `.file-header` dans un `.file-toolbar` enfant direct de `.content-area` (`position: sticky; top: 0`), masqué en mode lecture. **BUG-078** coloration syntaxique : le basculement des feuilles highlight.js suit le **mode** (`themes.applyTheme` + `ui.initTheme/applyTheme` lisent `obsigate-theme-mode`) au lieu de la clé de thème qui désactivait les deux feuilles. **#117** avatars prédéfinis : galerie de 12 images (`frontend/icons/avatar/`) dans `#cfg-profile`, clic → recadrage 256 px (pipeline import) + `PATCH /api/auth/me`, avatars actifs surlignés (`obsigate-avatar-preset`), import personnalisé et suppression conservés. Vérifié : Playwright (coloration 5/5 déterministe, toolbar épinglée à `barTop` constant au défilement), `unit.test.mjs` 12/12, `toolbar-order` 13/13, `settings-order-avatar` 12/12, JSDOM editor-inline/pane-manager/mobile-editor/image-viewer/pdf-viewer/config-mobile/media-viewer/excalidraw verts, pytest 1304 passed / 6 skipped, ruff/mypy 0, validate-imports 40 modules. | 🟢 corrigé (en attente vérif utilisateur) |
| 2026-09-24 | BUG-079 | Correction | `backend/main.py`, `tests/test_api_main.py`, `CHANGELOG.md`, `docs/ISSUES_TODOLIST.md` | **BUG-079** : `GET /api/diagnostics` renvoyait 500 « dictionary changed size during iteration ». Le handler itérait `inv.word_index.values()` et `index.items()` en direct alors que l'indexeur les modifiait depuis un autre thread (rebuild initial dans `_search_executor`, hooks incrémentaux `add_document`/`remove_document`) → `RuntimeError` dans le générateur. Correctif : **snapshot avant itération** (`list(index.items())`, `inv.word_index.copy()`) — copie C atomique sous le GIL, pas de verrou ajouté. Test de non-régression déterministe (`RaceDict` fait grossir le dict pendant l'itération ; échoue sans le correctif, passe avec). Vérifié : pytest 1305 passed / 6 skipped, ruff 0, mypy 0 (80 fichiers), validate-imports 40 modules, unit 12/12. | 🟢 corrigé (en attente vérif utilisateur) |
| 2026-09-27 | BUG-080, BUG-081 | Correction + enregistrement | `scripts/run-e2e-local.ps1`, `scripts/run-e2e-local.sh`, `scripts/e2e-server.ps1`, `playwright.config.ts`, `tests/test_e2e_harness.py` (nouveau), `CHANGELOG.md`, `docs/ISSUES_TODOLIST.md` | **BUG-080** : run E2E local pendu toute la nuit → harnais anti-blocage : `npx --yes` (plus de prompt interactif), install Chromium sautée si présent (`E2E_INSTALL_BROWSERS=1`), timeouts `E2E_TIMEOUT_SEC` (900)/`E2E_BROWSER_INSTALL_TIMEOUT_SEC` (600, exit 124), `globalTimeout` Playwright (15 min local / 30 min CI, `E2E_GLOBAL_TIMEOUT_MS`), pidfile resynchronisé sur le vrai owner du port + `stop` qui tue l'arbre complet (orphelins 81180/81936 nettoyés, port 2029 libéré). Diagnostic : double processus systématique (parent `.venv` parqué + enfant qui sert — environnemental, aussi sur flowdeck/3.13). **BUG-081** (ouvert, non traité) : `GET /api/auth/mfa/status` → 500 auth désactivée (`user` None, `router.py:827`, reproduit live). Vérifié : `test_e2e_harness.py` 8/8, cycle start/stop live (pidfile cohérent, port libéré). | 🟢 corrigé (en attente vérif utilisateur) ; BUG-081 🔴 ouvert |
| 2026-09-27 | BUG-082 | Correction CI | `.gitea/workflows/ci.yml`, `tests/test_ci_workflow.py`, `CHANGELOG.md`, `docs/ISSUES_TODOLIST.md` | **BUG-082** : `lint` rouge (`ERR_MODULE_NOT_FOUND: jsdom`, rouge depuis `7bee4a2`) — les fichiers de l'étape frontend racine à import statique `jsdom` (`upload.test.mjs`, puis `config-ai-keys.test.mjs` révélé par le CI après le 1er fix), alors que `jsdom` n'est installé que dans `tests/frontend/node_modules` (étape JSDOM). Les deux déplacés dans l'étape JSDOM (les deux branches) ; garde-fou `test_ci_workflow.py` généralisé (aucun fichier racine à import statique jsdom + suites verrouillées en JSDOM, contre-preuve OK). Vérifié : étape racine verte (11 suites) + `upload` et `config-ai-keys` verts depuis `tests/frontend/`. | 🟢 corrigé (en attente vérif utilisateur) |
| 2026-09-27 | BUG-083 | Correction CI | `.gitea/workflows/ci.yml`, `tests/test_ci_workflow.py` (nouveau), `CHANGELOG.md`, `docs/ISSUES_TODOLIST.md` | **BUG-083** : job `security` rouge — le runner Gitea Act tronque naïvement au premier `#` (même entre guillemets) : `echo "... see #87)"` devenait une citation non fermée (`unexpected EOF while looking for matching '"'"`, `/var/run/act/workflow/4` ligne 2). Seul `run:` du workflow avec un `#` (les `#` des noms d'étapes Bandit/Npm audit sont inoffensifs, ces étapes passent). Correctif : echo sans `#` (réf `#87` en commentaire YAML). Garde-fou `test_ci_workflow.py` (aucun `#` dans le code des `run:`, `upload.test.mjs` verrouillé en étape JSDOM — BUG-082) + contre-preuve sur l'ancien `ci.yml`. Vérifié : 56 passed. | 🟢 corrigé (en attente vérif utilisateur) |
| 2026-09-27 | BUG-081 | Correction | `backend/auth/router.py`, `tests/test_mfa.py`, `CHANGELOG.md`, `docs/ISSUES_TODOLIST.md` | **BUG-081** : `GET /api/auth/mfa/status` répondait 500 quand l'auth est désactivée — le pseudo-user `anonymous` n'a aucune entrée en store (`get_user` → `None`, `AttributeError` sur `user.get`). Garde `user is None` → payload « MFA désactivé ». Test `TestMfaStatusAuthDisabled` (échoue en 500 sans le correctif). Vérifié : `test_mfa.py` 32 passed, ruff/mypy 0. | 🟢 corrigé (en attente vérif utilisateur) |
| 2026-09-27 | #87 T6, T7, T8 | Sécurité (fin #87) | `backend/requirements.txt`, `backend/{render,export}.py`, `backend/tools/documents.py`, `backend/auth/router.py`, `backend/main.py`, `semgrep-rules/` (nouveau), `.gitea/workflows/ci.yml`, `tests/test_i18n_parity.py` (nouveau), `tests/test_auth_api.py`, `tests/test_security_headers.py`, `docker-compose.yml`, `.env.example`, `CHANGELOG.md`, `docs/ROADMAP.md`, `docs/ISSUES_TODOLIST.md` | **T6** : dépendances qualifiées (mistune 3.3.3, multipart 0.0.31, weasyprint 70, mcp 1.28.1, fastapi 0.141.1 + starlette 1.7.0, setuptools 84 ; `cast` mistune 3 sites) — suite 1359 passed, ruff/mypy 0, **`pip-audit` bloquant 0 vuln** (exception ecdsa/Minerva documentée : sans fix, HS256 only). **T7** : **semgrep bloquant** local 8 règles, 0 finding (trivy écarté : réseau). **T8** : Secure auto + `X-Forwarded-Proto` (`TRUST_PROXY`), warning affiné, CORS same-origin explicite, `style-src` résiduel assumé (189+343 sites) ; TODO exemple purgé, locales FR/EN 2213 parité testée, `npm audit` 0. | 🟢 corrigé (en attente vérif utilisateur) |
---
@@ -283,7 +298,9 @@ Avant de corriger quoi que ce soit, un agent IA doit :
| # | Titre | Date résolution | Résolu par | Correctif / Commit | Notes |
|---|---|---|---|---|---|
| *(aucun pour l'instant)* | | | | | |
| *BUG-083* | Job CI `security` rouge : le runner Gitea Act tronque le script `pip-audit` au premier `#` (citation de l'echo non fermée → `unexpected EOF while looking for matching '"'`) | 2026-09-27 | Utilisateur | `run:` assaini (echo sans `#`, réf `#87` en commentaire YAML) ; `tests/test_ci_workflow.py` (2 tests : aucun `#` dans le code des `run:`, `upload.test.mjs` verrouillé en étape JSDOM) ; vérifié : 56 passed (ci_workflow + e2e_harness + version), contre-preuve OK sur l'ancien `ci.yml` | Seul `run:` du workflow contenant un `#` (`see #87` dans l'echo). Les `#` des noms d'étapes (Bandit, Npm audit) sont inoffensifs (ces étapes passent). Correctif : echo sans `#`, réf `#87` en commentaire YAML |
| *BUG-082* | CI `lint` rouge : suites frontend à import statique `jsdom` exécutées dans l'étape racine où `jsdom` n'est jamais installé | 2026-09-27 | Utilisateur | `upload.test.mjs` + `config-ai-keys.test.mjs` déplacés dans l'étape JSDOM (les deux branches) ; garde-fou `test_ci_workflow.py` (aucun fichier racine à import statique jsdom + suites verrouillées en JSDOM) ; vérifié : étape racine verte + `upload` et `config-ai-keys` verts depuis `tests/frontend/` | `jsdom` ne vit que dans `tests/frontend/node_modules` (installé par l'étape JSDOM). Correctif : déplacer les suites concernées dans l'étape JSDOM |
| *BUG-080* | [🔴 BLOQUANT] E2E locaux bloqués toute la nuit : `npm run test:e2e:ps` ne termine jamais (serveurs orphelins sur le port 2029, `npx playwright install` sans `--yes` ni garde-fou, suite ~130 tests sans timeout global) | 2026-09-27 | Utilisateur | `run-e2e-local` : `npx --yes`, skip install Chromium si présent (`E2E_INSTALL_BROWSERS=1`), timeouts `E2E_TIMEOUT_SEC` (900)/`E2E_BROWSER_INSTALL_TIMEOUT_SEC` (600, exit 124) ; `playwright.config.ts` : `globalTimeout` 15 min local / 30 min CI (`E2E_GLOBAL_TIMEOUT_MS`) ; `e2e-server.ps1` : pidfile = vrai owner du port, `stop` tue l'arbre complet. Tests : `tests/test_e2e_harness.py` (8/8), cycle start/stop live (pidfile cohérent, port libéré) | Constat 2026-09-27 : `e2e-server.ps1 start` OK (READY 12 s) mais run suivant pendu toute la nuit ; 2 python orphelins (PID 81180 parent + 81936 sur le port, pidfile périmé). Double processus systématique (parent `.venv` parqué + enfant qui sert — aussi sur flowdeck/3.13 : environnemental, sans impact après correctif). Trouvé au passage : BUG-081 (`/api/auth/mfa/status` → 500 auth désactivée) |
---
+66 -20
View File
@@ -1,6 +1,6 @@
# ObsiGate — Roadmap
> **Version :** 2.27.9 | **Dernière mise à jour :** 2026-09-26
> **Version :** 2.37.0 | **Dernière mise à jour :** 2026-09-28
> **Ce fichier ne contient que le travail à venir** (🔵 En cours + ⚪ Backlog) et un index compact
> vers les fonctionnalités livrées.
> - **Méthode de livraison à appliquer pour toute tâche : [DELIVERY_WORKFLOW.md](./DELIVERY_WORKFLOW.md)**
@@ -42,6 +42,56 @@
---
## 🔵 En cours — Visionneuse & édition Excel (P0/P1/P2)
### 153. Visionneuse & édition XLSX — complétude (fidélité, recherche, IA, UX, formats)
- **Effort :** 8-13 jours (P0 ✅ 2-3 j · P1 : 4-6 j · P2 : 2-4 j) | **Impact :** 🟡
- **Statut :** 🔵 en cours — **P0 livré le 2026-09-27** (BUG-085 → BUG-088), **A5/A10/A12 livrés le 2026-09-28** (avec BUG-089), **A8/A9/A9bis livrés le 2026-09-28** (avec BUG-090), reste A6-A7 puis A13-A17
- **Analyse, risques et critères d'acceptation :** [features/xlsx-viewer.md](./features/xlsx-viewer.md)
- **Description :** #152 (visionneuse XLSX, 2.27.0) lit et édite correctement la **grille de
valeurs** d'un `.xlsx`, mais l'ensemble supporté est étroit : valeurs seulement (ni structure,
ni styles en écriture, ni formule recalculée), **écriture destructive** d'une partie du classeur,
tableurs **invisibles à la recherche** et **inutilisables par l'IA** au-delà de la création. Ce
lot suit ces ajouts ; les cases ci-dessous sont le **suivi de référence**, la fiche feature porte
le détail.
- **Constat (points de départ) :** `MAX_ROWS = 500` / `MAX_COLS = 40` sans indicateur (troncature
silencieuse) · `wb.save()` non atomique et sans verrou (concurrence) · saisie `=…` stockée comme
formule par openpyxl (injection DDE) · `content=""` à l'indexation (recherche TF-IDF et sémantique
aveugles) · aucun outil IA de lecture/édition d'un classeur existant · aucun test frontend ni
E2E sur le viewer.
- **Périmètre réel des pertes au round-trip (mesuré sur openpyxl 3.1.5, 2026-09-27) :** graphiques,
images, dessins **et** tableaux croisés sont préservés ; sont perdus les **valeurs calculées en
cache**, slicers/chronologies, contrôles de formulaire, connexions/requêtes, custom XML,
signature numérique, commentaires enrichis et macros.
- **Sous-tâches :**
- **P0 — garde-fous d'écriture (🔴, 2-3 j) — 🟢 livré**
- [x] **A1** Alerte de fidélité avant écriture : `inspect_workbook()` → `xlsx_lossy_features` + bandeau FR/EN + **409** `xlsx_lossy_content` sans `force` (confirmation explicite puis reprise) — BUG-085
- [x] **A2** Écriture atomique (`wb.save(.tmp)` + `os.replace()`, backup inchangé) — BUG-086
- [x] **A3** Verrou par fichier autour du read-modify-write (timeout 15 s + **409** `conflict`) — BUG-087
- [x] **A4** Neutralisation de l'injection de formule (`=`/`@` stockés en texte, opt-in `allow_formula` + bouton `f(x)`) — BUG-088
- **P1 — recherche, IA, UX (🟡, 4-6 j) — 🔵 en cours**
- [x] **A5** Indexation du contenu des feuilles (noms de feuilles + 20 premières lignes, plafond 5 k caractères) — les mots tapés dans une cellule rendent le fichier trouvable ; au passage **BUG-089** (reindex manuel ne reconstruisait pas l'index inversé)
- [ ] **A6** Outils IA `update_xlsx_cells` / `append_xlsx_rows` / `xlsx_to_markdown` / `list_xlsx_sheets`
- [ ] **A7** Navigation clavier + barre de formule + nom de cellule (Tab/Entrée/flèches, `Maj+Entrée`, copie de plage)
- [x] **A8** `thead` sticky + bandeau « feuille tronquée » (lève la troncature silencieuse) — BUG-090
- [x] **A9** Chargement paresseux par feuille (`GET …/xlsx/sheet?offset&limit`, défilement virtuel)
- [x] **A10** Types & formats de saisie (nombre/texte, booléens `TRUE`/`FAUX`, dates FR `JJ/MM/AAAA` jour-first)
- [ ] **A11** Tests frontend (`tests/frontend/xlsx-viewer.test.mjs`) + E2E (`tests/e2e/xlsx-viewer.spec.js`) au CI
- [x] **A12** Valeur calculée affichée sous la formule (2ᵉ lecture `data_only=True` seulement si l'archive contient un `<v>`, info-bulle FR/EN)
- **P2 — étendu (🟢, 2-4 j) — ⚪ à faire**
- [ ] **A13** Tri / filtre / recherche dans la feuille + export CSV de la sélection
- [ ] **A14** CRUD de feuilles, lignes et colonnes (renommer, insérer, supprimer, dupliquer)
- [ ] **A15** Styles minimaux en écriture + lecture fidèle (gras, fond, formats, fusions, volets figés)
- [ ] **A16** Formats additionnels (`.xlsm` avec `keep_vba`, `.xls`, `.ods`, `.csv` éditable)
- [ ] **A17** Vue « tableau de bord » (plages nommées, TCD, KPI par feuille, actions IA)
- **Convention de suivi :** chaque sous-tâche démarre par son ID stable (`#153-A<n>` dans cette
Roadmap) ; celles qui sont des **défauts** sont aussi ouvertes comme `BUG-NNN` dans
[ISSUES_TODOLIST.md](./ISSUES_TODOLIST.md) (A1→BUG-085, A2→BUG-086, A3→BUG-087, A4→BUG-088 ;
A8 le sera à son tour).
---
## ⚪ Backlog — Priorité 4 (P4)
### 73. Synchronisation multi-appareils — Obsidian Sync compatible
@@ -63,28 +113,20 @@
## ⚪ Backlog — Sécurité, architecture & performance (P0/P1)
### 85. Refonte architecturale — découpage du monolithe & persistance d'état (phase 2)
- **Effort :** 8-12 jours | **Impact :** 🟡 | **Zone :** backend
- **Décision 2026-09-26 : prioritaire (axe Dette & sécurité).**
- **Statut :** 🔵 en cours depuis 2026-09-26 — découpe par tranches à impact minimal (comportement inchangé, un domaine par commit). **T1 livrée (v2.27.2) :** `health` (`/api/health`, `/api/health/detailed` → `backend/routers/health.py`, `HealthResponse` → `schemas.py`). **T2 livrée (v2.27.3) :** `webhooks` (CRUD `/api/webhooks` → `backend/routers/webhooks.py`, logique déjà dans `backend/webhooks.py`). **T3 livrée (v2.27.4) :** `sharing` (`/api/share/*`, `/api/shares`, `/s/{token}*` → `backend/routers/sharing.py`, logique déjà dans `backend/share.py`). **T4 livrée (v2.27.5) :** `backups` (9 routes `/api/file/{vault}/backups|diff|restore` + `/api/backups*` → `backend/routers/backups.py`, `Diff/Restore*` → `schemas.py`, singleton SSE → `backend/sse.py`). **T5 livrée (v2.27.6) :** `search` (11 routes search/tags/suggest/graph/reload → `backend/routers/search.py`, modèles search → `schemas.py`, pool threads → `backend/search_executor.py`). **T6a livrée (v2.27.7) :** lecture fichiers (`/api/browse`, `/raw`, `/download`, `/backlinks`, `GET /api/file` → `backend/routers/files_read.py`, modèles + `EXT_TO_LANG` déplacés, helpers `_content_disposition`/`_media_max_inline_bytes` → `backend/routers/helpers.py`). **T6b livrée (v2.27.8) :** mutations fichiers/dossiers (save, xlsx/save, delete, create, rename, move, directories ×3, batch-upload → `backend/routers/files_write.py`, 15 modèles → `schemas.py`). **T6c livrée (v2.27.9) :** media/pdf/export/guide (file/pdf, exports ×3, guide, pdf/stream|info, image, media+thumb, attachments ×2, vault settings ×3, vault files → `backend/routers/files_media.py`, Range helper → `helpers.py`).
- **Description :** extraire le monolithe `backend/main.py` (~4 827 lignes au 2026-09-26, ~17 % du backend) en routers FastAPI par domaine et rendre persistant l'état qui ne l'est pas (index de recherche, JTI révoqués, compteurs de rate-limit) pour préparer le multi-nœuds. L'état mémoire actuel (index, inverted index, vecteurs sémantiques, `SSEManager`, collab) rend le multi-workers unsafe.
- **Sous-tâches :**
- [ ] Routers par domaine : files, search, share, webhooks, plugins, collab, admin, ai — `main.py` conservé comme assemblage (< 500 lignes) ; dédupliquer les modèles Pydantic vers `schemas.py`. **Avancement :** `health` ✅ (T1, `backend/routers/health.py`), `webhooks` ✅ (T2, `backend/routers/webhooks.py`), `sharing` ✅ (T3, `backend/routers/sharing.py`), `backups` ✅ (T4, `backend/routers/backups.py` + `backend/sse.py`), `search` ✅ (T5, `backend/routers/search.py` + `backend/search_executor.py`), `files-read` ✅ (T6a, `backend/routers/files_read.py` + `helpers.py`), `files-write` ✅ (T6b, `backend/routers/files_write.py`), `files-media` ✅ (T6c, `backend/routers/files_media.py`) ; `tools/registry.py` existe déjà (permissions/quotas/redaction — à compléter, pas à créer)
- [ ] Compléter `tools/registry.py` (existant : permissions/quotas/redaction) comme contrat central des outils IA si des manques sont constatés
- [ ] Persister index, JTI révoqués et compteurs de rate-limit (SQLite par défaut, Redis en option multi-nœuds ; le rate-limit actuel est in-memory mono-process)
- [ ] Verrous asyncio autour de l'index global et des stores JSON ; auditer les `except Exception` larges (> 100 occurrences) : best-effort (backup/audit) vs masquage d'erreur (erreurs typées 4xx/5xx + test)
- [ ] Extraire le service de partage public (expiration, révocation, quotas)
### 87. Amélioration continue — tests, CI/CD, revues de sécurité (phase 4)
- **Effort :** 3-5 jours | **Impact :** 🟡 | **Zone :** `.gitea/workflows/`, `tests/`
- **Décision 2026-09-26 : prioritaire (axe Dette & sécurité).**
- **Statut :** 🔵 en cours depuis 2026-09-26 — par tranches. **T1 livrée (v2.28.1) :** bandit bloquant (`nosec` justifiés B324/B404/B603/B607/B406, B105 exclu comme `pyproject`), `npm audit` bloquant (0 vulnérabilité), 5 suites frontend intégrées au CI (`upload`, `pretty`, `media-viewer`, `mfa-settings`, `config-ai-keys`). pip-audit reste consultatif (montées starlette/weasyprint à qualifier).
- **T6 livrée (v2.28.15) :** dépendances qualifiées — mistune 3.3.3, python-multipart 0.0.31, weasyprint 70, mcp 1.28.1, fastapi 0.141.1 + starlette 1.7.0, setuptools 84 (`cast` mistune 3 sites) — suite 1359 passed, ruff/mypy 0, **`pip-audit` bloquant, 0 vulnérabilité** (seule exception documentée : PYSEC-2026-1325 ecdsa, sans correctif upstream, JWT HS256 uniquement).
- **T7 livrée (v2.28.15) :** **semgrep bloquant** sur ruleset 100 % local `semgrep-rules/` (8 règles, 0 finding, contrôle négatif OK) ; trivy écarté (binaire + DB réseau, couche Python couverte).
- **T8 livrée (v2.28.15, fin BUG-034) :** cookies `Secure` auto (`true|false|auto`, `X-Forwarded-Proto` sous `TRUST_PROXY`, warning affiné, `TRUST_PROXY=true` en prod) ; `CORSMiddleware` same-origin explicite ; `style-src 'unsafe-inline'` conservé assumé (189 `style=` + 343 `el.style`, T5c ayant verrouillé `script-src`).
- **Description :** renforcer le pipeline (`.gitea/workflows/ci.yml`, `desktop-build.yml`) pour le rendre bloquant par défaut et accompagner les phases 1 → 3. Constat 2026-09-26 : job `security` non bloquant (`bandit`/`pip-audit` en `|| echo`, ni semgrep ni trivy), E2E limité à `chromium-desktop`, 5 suites frontend hors CI.
- **Sous-tâches :**
- [ ] Jobs CI sécurité **bloquants** (bandit/semgrep/trivy, audits pip/npm) + tests E2E XSS (page de partage + lecteur markdown)
- [ ] Tests de concurrence (`users.json`), fuzzing de timing regex, couverture des composants critiques ; intégrer au CI les 5 suites frontend hors CI (`upload`, `pretty`, `media-viewer`, `mfa-settings`, `config-ai-keys`)
- [ ] Finir BUG-034 (migration CSP **nonce**, conversion des handlers inline), `Secure` cookies à `true` par défaut, politique CORS same-origin explicite ; confirmer la rotation de la clé DeepSeek (BUG-006, clé dans l'historique Git)
- [ ] Jobs CI sécurité **bloquants** (bandit/semgrep/trivy, audits pip/npm) + tests E2E XSS (page de partage + lecteur markdown) — **T4 livrée :** `tests/e2e/xss.spec.js` (BUG-021/022, 2/2 vert) + `scripts/e2e-server.ps1` (cycle de vie serveur E2E avec progression `start|stop|status|logs`) + validation locale projet `chromium-desktop` : **108/108 verts** (obsigate 44, split 37, viewers 24, xss/header 3), mobiles ciblés 10/10
- [ ] Tests de concurrence (`users.json`), fuzzing de timing regex, couverture des composants critiques ; intégrer au CI les 5 suites frontend hors CI (`upload`, `pretty`, `media-viewer`, `mfa-settings`, `config-ai-keys`) — **T2 livrée (v2.28.2) :** `tests/test_hardening_concurrency.py` (users.json concurrent + budget temps regex) ; 5 suites au CI (T1)
- [ ] Finir BUG-034 (migration CSP **nonce**, conversion des handlers inline), `Secure` cookies à `true` par défaut, politique CORS same-origin explicite ; confirmer la rotation de la clé DeepSeek (BUG-006, clé dans l'historique Git) — **T3 livrée (v2.28.3)** (helper + avertissement + CORS attesté) ; **T5a livrée (v2.28.6)** (16 handlers inline → listeners, CSP inchangée) ; **T5b livrée :** nonce frais par réponse (`backend/csp.py`, `script-src`), injection dans les 6 pages HTML (dont nouvelle route `/excalidraw-editor.html`), `unsafe-inline` conservé (inerte) ; **T5c livrée (v2.28.13)** (`script-src` sans `unsafe-inline`) ; **T8 livrée (v2.28.15)** (fin BUG-034 : Secure auto + CORS explicite ; `style-src` résiduel assumé ; rotation DeepSeek BUG-006 toujours côté utilisateur)
- [ ] Revue périodique des dépendances ; documentation utilisateur FR/EN synchronisée ; contrôle automatisé de la conformité au DoD — **T6/T9 livrées (v2.28.15)** (`pip-audit` 0, `npm audit` 0, locales FR/EN 2213 clés parité testée `test_i18n_parity.py`, gardes `test_version.py` + `test_ci_workflow.py`)
- [ ] Revue périodique des dépendances ; documentation utilisateur FR/EN synchronisée ; contrôle automatisé de la conformité au DoD
---
@@ -164,6 +206,7 @@
| BUG-078 | Fichiers de code — coloration syntaxique restaurée (feuilles highlight.js basculées sur le mode de thème et non la clé) | 2.25.0 | [features/viewer-toolbar-highlight-avatars.md](./features/viewer-toolbar-highlight-avatars.md) |
| 115 | Viewer — barre d'outils de lecture épinglée au défilement | 2.25.0 | [features/viewer-toolbar-highlight-avatars.md](./features/viewer-toolbar-highlight-avatars.md) |
| 117 | Configuration — avatars prédéfinis dans le profil utilisateur (12 images) | 2.25.0 | [features/viewer-toolbar-highlight-avatars.md](./features/viewer-toolbar-highlight-avatars.md) |
| 85 | Refonte architecturale — découpage du monolithe (14 routers, `main.py` 4 827 → ~750 lignes), stores JSON verrouillés, rate-limit SQLite optionnel | 2.27.2→2.27.13 | [features/archi-refonte-85.md](./features/archi-refonte-85.md) |
---
@@ -171,17 +214,20 @@
| Priorité | Items | Effort total estimé |
|---|---|---|
| ✅ Complété | #1 → #59, #61–72, #74–76, #78–84, #86, #88–93, #94–100, #102–115, #117, #92 | ~133 jours réalisés |
| ✅ Complété | #1 → #59, #61–72, #74–76, #78–86, #88–93, #94–100, #102–115, #117, #92 | ~141 jours réalisés |
| 🔵 Finitions | #77 Desktop : 6 tests E2E **manuels** ([protocole](./DESKTOP_E2E_CHECKLIST.md)) — signature Windows non retenue (décision 2026-09-26) | ~0,5-1 jour |
| ⚪ P4 reporté | #73 Sync — **reporté (décision 2026-09-26)**, hors chemin critique | 6-8 jours si réactivé |
| ⚪ P0/P1 prioritaire | #85, #87 Refonte architecturale, CI/CD (BUG-035 → BUG-040 corrigés, #86 livré) | ~11-17 jours |
| **Total chemin critique** | **#77 fin + #85 + #87** | **~12-18 jours** |
| ⚪ P0/P1 prioritaire | #87 CI/CD (BUG-035 → BUG-040 corrigés, #86 livré) | ~3-5 jours |
| ⚪ P0/P1/P2 backlog | #153 Visionneuse & édition XLSX — complétude (P0 ✅ A1-A4 ; P1 ✅ A5, A8-A10, A12, A9bis — reste A6-A7 ; A13-A17 2-4 j) | 2-4 jours restants |
| **Total chemin critique** | **#77 fin + #87** | **~4-6 jours** |
---
## Notes
- **Décisions 2026-09-26 :** axe prioritaire = dette & sécurité (#85/#87) ; #73 Sync reporté (P4, hors chemin critique) ; desktop livré non signé + doc SmartScreen.
- **Ajout 2026-09-27 :** #153 ouvert à la suite de l'audit de la visionneuse XLSX (limitations, risques de perte de données, périmètre IA/recherche) — détail et critères dans [features/xlsx-viewer.md](./features/xlsx-viewer.md).
- **Clôture #85 (v2.27.13) :** monolithe découpé (T1→T9), stores verrouillés + rate-limit SQLite (T10), fiche `docs/features/archi-refonte-85.md`.
- Les items P3/P4 ne sont pas ordonnés par priorité interne — à raffiner selon les retours utilisateurs.
- L'effort inclut le développement + tests unitaires + intégration CI, mais pas la documentation utilisateur.
- Les items marqués 🟢 (nice-to-have) sont de bons candidats pour des contributions externes.
+77
View File
@@ -0,0 +1,77 @@
# #85 — Refonte architecturale : découpage du monolithe & persistance d'état (phase 2)
> **Statut :** livré (T1→T10) — `backend/main.py` 4 827 → ~750 lignes, 14 routers,
> persistance partielle (stores verrouillés + rate-limit SQLite optionnel).
> Méthode : tranches à impact minimal, comportement inchangé, un domaine par
> commit, suite complète verte à chaque commit (1320 passed / 6 skipped).
## 1. Découpage du monolithe (T1→T9, comportement inchangé)
Chaque tranche déplace un domaine vers `backend/routers/` (handlers verbatim,
mêmes chemins/modèles/auth/tags OpenAPI), les modèles vers `backend/schemas.py`,
et ne committe que sur suite verte + `test_version` vert.
| Tranche | Domaine | Nouveau module | Version |
|---|---|---|---|
| T1 | health (`/api/health*`) | `routers/health.py` (+ `HealthResponse` → schemas) | 2.27.2 |
| T2 | webhooks CRUD | `routers/webhooks.py` | 2.27.3 |
| T3 | sharing (`/api/share*`, `/s/*`) | `routers/sharing.py` | 2.27.4 |
| T4 | backups (9 routes) | `routers/backups.py` (+ `Diff/Restore*` → schemas, `backend/sse.py`) | 2.27.5 |
| T5 | search (11 routes) | `routers/search.py` (+ modèles → schemas, `backend/search_executor.py`) | 2.27.6 |
| T6a | lecture fichiers | `routers/files_read.py` (+ modèles, `routers/helpers.py`) | 2.27.7 |
| T6b | mutations fichiers/dossiers | `routers/files_write.py` (+ 15 modèles → schemas) | 2.27.8 |
| T6c | media/pdf/export/guide | `routers/files_media.py` (Range helper → `helpers.py`) | 2.27.9 |
| T7 | config (12 routes) | `routers/config.py` (`_FALLBACK_MODELS` déplacé) | 2.27.10 |
| T8 | vaults + history + conflicts (13 routes) | `routers/vaults.py`, `history.py`, `conflicts.py` (+ `backend/watcher_state.py`) | 2.27.11 |
| T9 | realtime + render | `routers/realtime.py` (SSE + collab WS), `backend/render.py` | 2.27.12 |
`main.py` ne contient plus que l'assemblage : lifespan, middlewares, montage
des routers, racine `/api`, statique/SPA, 4 cales de compatibilité testées
(`_resolve_safe_path`, `_backup_file`, `_check_vault_writable`, `_get_backup_dir`).
Correctifs au passage : décorateur orphelin `/s/{token}` (double-enregistrement
de `/api/conflicts`), tag OpenAPI `media` inexistant (assignation par chemin
conservée), tests statiques frontend réalignés (`image-viewer`, `media-viewer`),
tests repointés vers les modules canoniques (`test_ai_models`, `test_api_main`).
## 2. Persistance d'état (T10)
| État | Avant | Après |
|---|---|---|
| JTI révoqués (`revoked_tokens.json`) | persisté, **sans verrou** | `RLock` (load/save/revoke/check) |
| `shares.json` | persisté, **sans verrou** | `RLock` (4 mutateurs) |
| `webhooks.json` + secrets | persistés, **sans verrou** | `RLock` (create/update/delete/secrets) |
| `api_keys.json` (tool-secrets) | persisté, **sans verrou** | `RLock` (set/delete) |
| Rate-limit auth | mémoire, mono-process | **inchangé par défaut** + option `OBSIGATE_RATELIMIT_DB` (SQLite WAL : mêmes fenêtres/budgets, partagé multi-workers, survit au redémarrage) |
| Index de recherche | mémoire, rebuild au démarrage | **conservé** (voir §3) |
| `users.json`, `api_tokens.json`, `vault_settings.json` | déjà verrouillés (BUG-029, #107) | inchangé |
Tests : `tests/test_store_locks.py` (4 — concurrence threads, pertes prouvées
sans verrou : 25/200 partages), `tests/test_ratelimit_store.py` (7 —
sémantique SQLite identique, persistance, concurrence 200/200).
Déjà existants et vérifiés (pas de code) : verrous `threading` + `asyncio`
de l'indexeur (`_index_lock`, `_async_index_lock`), contrat central des
outils IA — `backend/tools/registry.py` couvre déjà permissions
(`requires_vault`, `require_destructive_allowed`), quotas
(`check_and_record` par outil) et redaction (`redact_payload`) pour les
35 outils enregistrés via `@tool(`.
## 3. Décisions assumées (non fait, et pourquoi)
- **Index non persisté sur disque.** Le rebuild différentiel (#86 : réutilise
les entrées inchangées `size` + `mtime`) rend le démarrage rapide ; un
snapshot introduirait des risques de staleness/drift de format sans gain
mesuré. Réévaluer si le démarrage devient lent (vaults 50k+ fichiers).
- **Redis exclu.** SQLite WAL couvre le multi-workers mono-hôte sans nouvelle
infra ; Redis reste l'option multi-nœuds documentée (cf. `ratelimit.py`).
- **`.gitignore` (`_*.py` ignore les `__init__.py`).** Contourné par
`git add -f` comme les packages existants ; assainir la règle à part.
- Noms en `_` conservés (`backend/render.py`, stores) : déplacement verbatim,
zéro churn d'appels.
## 4. Reste connu (hors #85)
- CSP `unsafe-inline` (migration nonce, BUG-034 partiel) et `Secure` cookies → #87.
- `main.py` (~750 lignes) : lifespan, middlewares, statique/SPA — cible
d'extraction ultérieure si besoin, non bloquant.
+235
View File
@@ -0,0 +1,235 @@
# #153 — Visionneuse & édition XLSX — état des lieux et backlog
> **Item de roadmap :** [#153 — Visionneuse & édition XLSX — complétude](../ROADMAP.md)
> **Origine :** #152 (visionneuse XLSX, livrée en 2.27.0 — voir
> [archive/COMPLETED_v1-v2.md](../archive/COMPLETED_v1-v2.md))
> **Statut :** 🔵 En cours — **P0 livré le 2026-09-27** (BUG-085 → BUG-088), **A5/A10/A12 livrés le 2026-09-28** (avec BUG-089), **A8/A9/A9bis livrés le 2026-09-28** (avec BUG-090), reste A6-A7 puis A13-A17
> **Effort estimé :** 8-13 jours au total (P0 ✅ 2-3 j · P1 4-6 j · P2 2-4 j)
> **Règle de maintenance :** la Roadmap porte les cases à cocher (suivi), cette fiche porte
> l'analyse, les risques et les critères d'acceptation. **Ne pas dupliquer le détail.**
---
## 1. Périmètre et architecture
| Couche | Fichier | Rôle |
|---|---|---|
| Lecture | `backend/xlsx_reader.py` | `render_sheets()` → un tableau HTML par feuille (openpyxl `read_only=True`, `data_only=False`) |
| Endpoint lecture | `backend/routers/files_read.py:241-265` | `GET /api/file/{vault}?path=…` → `is_xlsx: true` + `xlsx_sheets: [{name, html, rows, cols, total_*, max_*, truncated}]` |
| Endpoint fenêtre | `backend/routers/files_read.py` | `GET /api/file/{vault}/xlsx/sheet?path=&sheet=&offset=&limit=` (#153 A9) — une fenêtre de lignes, vraies coordonnées A1 |
| Schéma API | `backend/schemas.py:286-290` | `is_xlsx`, `xlsx_sheets`, `XlsxSheetWindowResponse` |
| Écriture | `backend/services/mutations.py:227-320` | `edit_xlsx_cells()` (backup, refs A1 validées, coercion `str`→`int`/`float`) |
| Endpoint écriture | `backend/routers/files_write.py:116-148` | `PUT /api/file/{vault}/xlsx/save` (1 à 500 cellules / requête) |
| Documentation API | `backend/openapi_docs.py:184-187` | exemple d'appel `xlsx/save` |
| UI | `frontend/js/viewer.js:998-1100` | `renderXlsxViewer()` (onglets, cellules sales, Entrée/Échap, collage monoligne) |
| CSS | `frontend/style.css:10927-10988` | `.xlsx-*` (variables CSS, colonne A `sticky`) |
| Indexation | `backend/indexer.py:68, 563-568, 957-960` | `.xlsx` supporté, **métadonnées seules** (`content=""`) |
| Outils IA | `backend/tools/documents.py:66-89` + `schemas.py:296-305` | `create_xlsx` (WRITE + confirmation) — **création seule** |
| Tests | `tests/test_xlsx_viewer.py` | 11 tests backend (affichage, index, save, backup, 400) |
## 2. Ce qui est supporté aujourd'hui (livré, non concerné par #153 sauf mention)
**Lecture** — multi-feuilles avec onglets ; en-têtes A1/A2/B1 et numéros de ligne ; valeurs
`_fmt()` (dates `YYYY-MM-DD` / `YYYY-MM-DD HH:MM`) ; lignes et colonnes de fin élaguées
(`_trim`) ; feuille vide affichée ; `html.escape()` sur chaque valeur.
**Édition** — `contentEditable` par `<td>`, classe `xlsx-dirty`, bouton Save actif seulement si
modification ; `Entrée` → blur, `Échap` → restauration, collage forcé en monoligne ; un `PUT` par
feuille sale ; coercion automatique des nombres (`"250"` → int `250`) ; chaîne vide → cellule
vidée ; backup `.bak` avant écriture ; garde-fou vault read-only (403) ; `resolve_safe_path()`
(anti path-traversal) ; `check_vault_access()` + `require_auth` ; journalisation d'audit
(`log_file_save`).
**Divers** — téléchargement de l'original ; refresh de l'arborescence via le watcher après
écriture ; rafraîchissement de la visionneuse après une action IA (`create_xlsx` →
`obsigate:file-written`, BUG-076).
## 3. Limites connues (par couche)
### 3.1 Fidélité du round-trip — risque n°1
`load_workbook()` → `wb.save()` : ce qui est **réellement** perdu a été mesuré sur
openpyxl 3.1.5 (2026-09-27), et non repris de la documentation :
| Élément | Round-trip openpyxl 3.1.5 |
|---|---|
| Graphiques, images, dessins | ✅ **préservés** (mesuré : `xl/charts/`, `xl/drawings/`, `xl/media/` intacts) |
| Tableaux croisés (pivot) + caches | ✅ **préservés** (`reader/excel.py` relit les `TableDefinition`, `workbook/_writer.py` les réécrit) |
| Styles, formats, fusions, validation de données, mise en forme conditionnelle, commentaires | ✅ préservés |
| **Valeurs calculées en cache** (`<f>…</f><v>…</v>`) | ❌ **perdues** → tout lecteur `data_only=True` (pandas, script tiers, convertisseur) renvoie `None` tant qu'Excel n'a pas recalculé |
| Slicers / chronologies, contrôles de formulaire (`ctrlProps`/`activeX`), connexions & requêtes, custom XML, signature numérique, commentaires enrichis, macros | ❌ **perdus** (parties absentes de l'archive après écriture) |
La liste fait foi dans le code : [`LOSSY_PARTS`](../backend/xlsx_reader.py) + la sonde
`<f>…</f><v>[^<]` pour les valeurs en cache (openpyxl écrivant lui-même un `<v></v>` vide).
**Ce qui reste ouvert** (non mesuré, prudence) : types de graphiques exotiques (treemap,
sunburst, funnel…), `sparklines`, `xl/queryTables` en lecture Excel. Un classeur qui en contient
peut sortir dégradé, voire échouer au chargement — d'où le refus par défaut (A1).
### 3.2 Lecture
- Aucun style, format de nombre, devise, pourcentage, largeur de colonne, ligne figée, cellule
fusionnée, commentaire, lien hypertexte, validation de données, mise en forme conditionnelle.
- Plafonds durs `MAX_ROWS = 500`, `MAX_COLS = 40` par feuille, **sans indicateur dans l'UI** : au-delà,
contenu silencieusement tronqué et **non éditable**.
- Pas de pagination ni de chargement à la demande : toutes les feuilles sont rendues d'un bloc
dans le JSON (20 feuilles × 20 000 cellules = payload énorme, UI gelée).
- Formules affichées **en texte** (`=B1*2`), jamais recalculées ; après édition, les cellules
dépendantes ne se mettent pas à jour à l'écran.
### 3.3 UI (`viewer.js`)
Navigation clavier (Tab/flèches) absente ; pas de barre de formule, pas de nom de cellule actif,
pas d'undo/redo global, pas de recherche dans la feuille, pas de tri/filtre, pas d'export CSV,
pas d'ajout/renommage/suppression de feuille, pas d'insertion/suppression de ligne ou colonne,
pas de sélection de plage, pas de copie d'une plage, pas de retour ligne dans une cellule
(`Maj+Entrée`) ; seul le retour de l'API est signalé (plafond 500 cellules) ; seule la
**colonne A** est `sticky` (le `thead` ne l'est pas → les en-têtes de colonnes disparaissent au
défilement vertical). **Couverture de test** : `tests/frontend/xlsx-viewer.test.mjs` (10) et
`tests/e2e/xlsx-viewer.spec.js` (3) depuis #153 P0 — la navigation clavier et la barre de formule
restent à faire (A7).
### 3.4 Recherche, IA et knowledge base
- **Indexation** : `content=""` → un `.xlsx` est totalement **invisible** à la recherche TF-IDF, à
la recherche sémantique, au remplacement global, aux tags et aux statistiques de contenu.
- **Outils IA** : seul `create_xlsx` existe (crée un fichier neuf, une seule feuille,
`overwrite=True` par défaut) ; `read_file` fait un `read_text()` sur l'archive ZIP → **bruit
binaire** envoyé au LLM ; pas de `update_xlsx_cells` pourtant le service existe déjà, pas
d'ajout de lignes, pas de `xlsx → markdown` pour le contexte.
## 4. Risques de sécurité / robustesse
| # | Risque | Où | Traitement | État |
|---|---|---|---|---|
| R1 | Perte silencieuse (valeurs calculées, slicers, contrôles, connexions, custom XML, signature) | `mutations.edit_xlsx_cells` | **A1** — bandeau + **409** `xlsx_lossy_content` sans `force` | 🟢 livré (BUG-085) |
| R2 | Écriture non atomique (`wb.save()` en place) → classeur corrompu si crash | `mutations.edit_xlsx_cells` | **A2** — `.tmp` + `os.replace` | 🟢 livré (BUG-086) |
| R3 | Concurrence : deux éditions (onglets, watcher + IA) → dernier écrivain gagne | `mutations.edit_xlsx_cells` | **A3** — verrou par chemin, **409** `conflict` | 🟢 livré (BUG-087) |
| R4 | **Injection de formule** : une saisie `=cmd\|…`, `=HYPERLINK(…)` est stockée comme formule par openpyxl → DDE à l'ouverture dans Excel | `mutations._write_cell` | **A4** — forçage texte (`data_type="s"`), opt-in `allow_formula` | 🟢 livré (BUG-088) |
| R5 | Troncature silencieuse au-delà de 500×40 | `xlsx_reader.MAX_ROWS/MAX_COLS` | A8 / A9 | 🟢 bandeau + dimensions exposées (BUG-090) ; le chargement paresseux par fenêtres sert les lignes au-delà du plafond |
## 5. Backlog #153 — sous-tâches
Légende : 🔴 P0 (sécurité / perte de données) · 🟡 P1 (valeur immédiate) · 🟢 P2 (confort /
couverture) · effort en jours-homme de développement + tests.
### P0 — Garde-fous d'écriture (2-3 j) — 🟢 livré le 2026-09-27
- [x] **A1 — Alerte de fidélité avant écriture (R1).** `inspect_workbook()` liste ce qu'un
round-trip perd (`LOSSY_PARTS` + sonde valeurs en cache) ; la lecture renvoie
`xlsx_lossy_features` ; la visionneuse affiche un bandeau listant les éléments ; `PUT
…/xlsx/save` répond **409** `xlsx_lossy_content` (avec `details.features`) tant que `force` n'est
pas passé, le client demande confirmation puis réémet avec `force: true` (une seule fois par
session). *Vérifié :* `TestXlsxLossyGuard` (5), `xlsx-viewer.test.mjs` (10), E2E (3).
- [x] **A2 — Écriture atomique (R2).** `wb.save(<nom>.<pid>.tmp)` puis `os.replace()` ; `.tmp`
supprimé sur échec ; backup `.bak` inchangé. Le `.tmp` est ignoré par le watcher. *Vérifié :*
`TestXlsxAtomicWrite` (2) — les octets d'origine sont intacts après un `save` en échec.
- [x] **A3 — Verrou par fichier (R3).** Verrou `threading.Lock` par chemin (registre + garde,
timeout 15 s) autour du cycle load → edit → replace ; **409** `conflict` si le délai est dépassé.
L'endpoint est passé en `def` (sync) pour que l'attente s'exécute dans le threadpool. *Vérifié :*
`TestXlsxWriteLock` (2). *Limite :* verrou en mémoire, par processus (suffisant pour un serveur
ObsiGate, y compris desktop).
- [x] **A4 — Neutralisation de l'injection de formule (R4).** `cell.data_type = "s"` après
affectation : une saisie `=`/`@` est stockée en texte. Opt-in `allow_formula: true` côté API et
bouton `f(x)` dans la visionneuse (état de session, jamais persisté). `+`/`-` restent des
nombres. Au passage : le handler `ServiceError` expose `code` + `details` et `api()` les
propage sur l'Error. *Vérifié :* `TestXlsxFormulaGuard` (4) + test du toggle côté UI.
### P1 — Recherche, IA, UX (4-6 j) — 🟢 A5, A10, A12 livrés le 2026-09-28
- [x] **A5 — Indexation du contenu des feuilles.** `extract_indexable_text()` (noms de feuilles +
20 premières lignes, `MAX_INDEX_CHARS = 5 000`, 20 feuilles max) alimente le TF-IDF et la
recherche sémantique ; la lecture binaire reste inchangée pour l'affichage. Un classeur
chiffré/corrompu s'indexe par son seul nom (jamais d'exception). Au passage : **BUG-089**,
un reindex manuel ne reconstruisait pas l'index inversé. *Vérifié :* `TestXlsxSearchable` (4)
+ `TestXlsxIndexing`, **contre-preuve** (neutraliser l'extraction → 3 tests échouent).
- [ ] **A6 — Outils IA sur classeur.** `update_xlsx_cells` (enveloppe du service existant),
`append_xlsx_rows`, `xlsx_to_markdown` (contexte LLM, plafonné), `list_xlsx_sheets` — risque
WRITE + confirmation pour les mutations, libellés i18n dans `backend/tools/labels.py`,
refresh viewer via `obsigate:file-written`.
- [ ] **A7 — Navigation clavier & barre de formule.** `Tab`/`Maj+Tab`/`Entrée`/flèches, cellule
active affichée (nom A1), `Maj+Entrée` pour le multiligne, copier une plage, focus visible
et compatible mobile (≥ 44 px, `tests/e2e/mobile-editor.spec.js`).
- [x] **A8 — `thead` sticky + indicateur de troncature (R5) — livré 2026-09-28 (BUG-090).**
Ligne d'en-têtes figlée au défilement vertical (`thead th { top: 0 }` ; `top: auto` sur les
numéros de ligne, sans quoi ils s'empilent en haut à gauche) ; `render_sheets()` expose
`total_rows`/`total_cols` (dimensions déclarées), `max_rows`/`max_cols` (plafonds) et
`truncated` — le bandeau « feuille tronquée » annonce le **plafond atteint** et non la
taille élaguée (une feuille creuse rend 1×1 tout en couvrant 500 lignes) ; libellés
`xlsx.truncated_*` FR/EN. *Vérifié :* `TestXlsxTruncationNotice` (4), `xlsx-viewer.test.mjs`
(4 nouveaux), E2E sur `test_vault/sample-xlsx-large.xlsx` (520 lignes).
- [x] **A9 — Chargement paresseux par feuille (côté API).** Endpoint
`GET /api/file/{vault}/xlsx/sheet?sheet=&offset=&limit=` (`XlsxSheetWindowResponse`,
exemple dans `backend/openapi_docs.py`) : une fenêtre de 1 à 1 000 lignes (plafond
`MAX_WINDOW_ROWS`, `limit>1000` → 422), `has_more` pour paginer, valeurs calculées A12
incluses. Les numéros de ligne et `data-cell` restent les coordonnées A1 réelles de la
feuille (`_table(..., row_offset=offset)`) : une fenêtre est indistinguishable d'un rendu
complet et une édition dans la fenêtre cible la bonne cellule. Erreurs : 404 feuille
inconnue / fichier absent, 415 non-`.xlsx`. *Vérifié :* `TestXlsxSheetWindow` (11),
**contre-preuve** (neutraliser l'offset → 3 tests échouent), E2E « l'endpoint de fenêtre
sert les lignes au-delà du plafond ».
- [x] **A9bis — Chargement à la demande côté UI.** Sous une feuille tronquée, un pied de page
« N lignes affichées sur M · Charger la suite » apparaît : cliquer — ou approcher du bas
du tableau (sentinelle de défilement, marge 120 px) — fetch la fenêtre suivante
(`limit=500`) et l'insère dans la table. Les lignes ajoutées passent par le **même**
pipeline d'édition que le rendu initial (`setupCell` factorisé : contenteditable, dirty,
Échap, collage monoligne, info-bulle valeurs calculées) et sont donc sauvegardables
immédiatement. Un fetch échoué restore le libellé du pied de page (retry possible) et
toast l'erreur ; feuille complète → pied de page masqué (`class="done"`).
*Vérifié :* `xlsx-viewer.test.mjs` 19/19 (5 nouveaux), **contre-preuve** (désactiver
`wireLazyRows` → 5 tests échouent), E2E « le bouton charger la suite ajoute les lignes
cachées » sur `sample-xlsx-large.xlsx` (A520 visible et éditable après clic).
- [x] **A10 — Types et formats de saisie.** `_coerce_xlsx_value()` reconnait les booléens
(`true`/`vrai`/`oui`/`yes` et leurs négatifs) et les dates FR `JJ/MM/AAAA` (+ `HH:MM`),
jour-first comme Excel en locale française : `01/02/2026` = 1ᵉʳ février. Une saisie
ressemblant à une formule n'est jamais convertie (BUG-088 préservé) ; un code postal
numérique ou une version restent ce qu'ils sont. *Vérifié :* `TestXlsxValueCoercion` (5),
**contre-preuve** (neutraliser la coercion → 2 tests échouent).
- [ ] **A11 — Tests frontend + E2E.** `tests/frontend/xlsx-viewer.test.mjs` (dirty, Échap,
collage, 1 PUT par feuille, bouton désactivé) et `tests/e2e/xlsx-viewer.spec.js`
(ouverture, onglets, édition, sauvegarde, rechargement) ; intégration au CI.
- [x] **A12 — Valeurs calculées.** La valeur en cache s'affiche sous la formule dans un
`<span class="xlsx-cached">`. La 2ᵉ lecture `data_only=True` n'a lieu que si l'archive
contient réellement un `<f>…</f><v>…</v>` (sonde déjà présente pour A1) : le cas courant
reste à un seul chargement, et toute erreur retombe sur l'affichage formules seul.
L'info-bulle est traduite côté client (`xlsx.cached_value_title` FR/EN) — aucun texte
d'interface n'est émis par le backend. *Vérifié :* `TestXlsxCachedValues` (3),
**contre-preuve** (neutraliser la 2ᵉ lecture → 2 tests échouent).
### P2 — Étendu (2-4 j)
- [ ] **A13 — Tri / filtre / recherche dans la feuille + export CSV de la sélection.**
- [ ] **A14 — CRUD de feuilles et de lignes/colonnes** (renommer, insérer, supprimer, dupliquer).
- [ ] **A15 — Styles minimaux en écriture et lecture fidèle** (gras, fond, format
devise/pourcentage/date, cellules fusionnées, volets figés) ; conserver `csv-table` comme
socle de rendu.
- [ ] **A16 — Formats additionnels.** `.xlsm` (`keep_vba=True`), `.xls`, `.ods`, `.csv` éditable
comme tableur — dépendances à qualifier (`xlrd`/`odfpy`) ou conversion.
- [ ] **A17 — Vue « tableau de bord ».** Détection des plages nommées, TCD et graphiques ; vue
résumée (KPI par feuille) et proposal d'actions IA sur ces plages.
## 6. Règles de livraison (rappel `AGENTS.md` / `DELIVERY_WORKFLOW.md`)
- Chaque sous-tâche démarre par un **ID stable** : nouvelle feature = `#153-A<n>` dans la
Roadmap ; si la sous-tâche est un **défaut** (A1, A2, A3, A4, A8), l'ouvrir aussi comme
`BUG-NNN` dans `docs/ISSUES_TODOLIST.md` au moment du démarrage.
- Backend : docstrings, `response_model` pour tout endpoint ajouté, exemple dans
`backend/openapi_docs.py`, chemin utilisateur via `resolve_safe_path()`.
- Frontend : vanilla JS sans build, `safeCreateIcons()`, **variables CSS** (jamais de couleur
hardcodée), **i18n FR + EN** pour chaque nouveau texte (`test_i18n_parity.py` vert).
- Tests : `pytest tests/test_xlsx_viewer.py`, `ruff`, `mypy`, `validate-imports`, suite frontend
ciblée, E2E si l'UI change — puis CI verte.
- Documentation : `CHANGELOG.md` `[Unreleased]`, Roadmap (case cochée), cette fiche (résultat),
guide utilisateur i18n + README si impact utilisateur.
## 7. Historique
| Date | Événement |
|---|---|
| 2.27.0 | #152 livré : affichage multi-feuilles, édition des cellules, téléchargement (`docs/archive/COMPLETED_v1-v2.md`) |
| 2026-09-27 | Audit complet → création de #153 : limites, risques R1-R5, backlog A1-A17 |
| 2026-09-27 | Périmètre de perte **remesuré** sur openpyxl 3.1.5 : graphiques / images / TCD sont préservés, seules les valeurs en cache et quelques parties exotiques sont perdues |
| 2026-09-27 | **P0 livré** (BUG-085 → BUG-088) : `xlsx_lossy_features` + 409 `xlsx_lossy_content`, écriture atomique, verrou par fichier, formules stockées en texte par défaut |
| 2026-09-28 | **A5 + A10 + A12 livrés** : le contenu des cellules est indexé (recherche), la saisie est typée (booléens, dates FR), la valeur calculée s'affiche sous la formule. **BUG-089** corrigé au passage (reindex manuel ≠ reconstruction de l'index inversé ; `backend/search.py` lisait l'index par valeur) |
| 2026-09-28 | **A8 + A9 livrés** (BUG-090) : la troncature d'une feuille est annoncée (bandeau + dimensions dans la réponse de lecture), les en-têtes restent visibles au défilement, et `GET …/xlsx/sheet` sert une fenêtre de lignes avec les vraies coordonnées A1 — les lignes au-delà du plafond redeviennent accessibles aux clients API. Défilement virtuel côté UI à suivre |
| 2026-09-28 | **A9bis livré** : « Charger la suite » + sentinelle de défilement sous une feuille tronquée ; les lignes ajoutées sont éditables et sauvegardables immédiatement (même pipeline que le rendu initial) |
+1 -31
View File
@@ -1576,17 +1576,6 @@
class="help-search-clear"
id="config-search-clear"
title="Effacer"
onclick="
var s =
document.getElementById(
'config-nav-search',
);
if (s) {
s.value = '';
s.dispatchEvent(new Event('input'));
s.focus();
}
"
>
X
</button>
@@ -1698,7 +1687,7 @@
<input type="text" id="profile-name" class="config-input" placeholder="Votre nom" maxlength="60">
</div>
<button class="config-save-btn" id="profile-save" data-i18n="config.save">Enregistrer</button>
<button class="config-save-btn" id="profile-logout" style="background:var(--danger-bg);color:var(--danger);border-color:var(--danger);margin-left:8px" data-i18n="config.logout" onclick="if(window.handleLogout)window.handleLogout();else{doLogoutFallback()}">Déconnexion</button>
<button class="config-save-btn" id="profile-logout" style="background:var(--danger-bg);color:var(--danger);border-color:var(--danger);margin-left:8px" data-i18n="config.logout">Déconnexion</button>
<span class="profile-saved" id="profile-saved" style="display:none" data-i18n="config.saved">✓ Sauvegardé</span>
</div>
</section>
@@ -3005,14 +2994,6 @@
id="help-hamburger"
title="Sommaire"
aria-label="Afficher le sommaire"
onclick="
var n = document.getElementById('help-nav');
if (n) {
var d = n.style.display;
n.style.display =
d === 'none' || d === '' ? 'flex' : 'none';
}
"
>
<i
data-lucide="menu"
@@ -3077,17 +3058,6 @@
id="help-search-clear"
title="Effacer la recherche"
aria-label="Effacer"
onclick="
var s =
document.getElementById(
'help-nav-search',
);
if (s) {
s.value = '';
s.dispatchEvent(new Event('input'));
s.focus();
}
"
>
✕
</button>
+24 -3
View File
@@ -72,14 +72,25 @@ async function api(path, opts) {
}
if (!res.ok) {
var detail = "";
var code = "";
var details = null;
try {
var body = await res.json();
detail = body.detail || "";
// #153 A1 : the service layer exposes a stable code + details so callers
// can branch on the failure (e.g. confirm a lossy .xlsx write) instead of
// matching on the message.
code = body.code || "";
details = body.details || null;
} catch (_) {
/* no json body */
}
showToast(detail || "Erreur API : " + res.status, "error");
throw new Error(detail || "API error: " + res.status);
var apiError = new Error(detail || "API error: " + res.status);
apiError.status = res.status;
apiError.code = code;
apiError.details = details;
throw apiError;
}
return res.json();
}
@@ -1305,8 +1316,7 @@ async function _startMfaSetup() {
// secret when the backend has no QR generator available.
const qrImg = data.qr_data_url
? `<img id="mfa-qr-img" alt="QR Code" class="mfa-qr-code-img"
src="${data.qr_data_url}"
onerror="this.style.display='none';document.getElementById('mfa-qr-fallback').style.display='block';">`
src="${data.qr_data_url}">`
: "";
const fallbackStyle = data.qr_data_url ? "display:none" : "";
flowArea.innerHTML = `
@@ -1335,6 +1345,17 @@ async function _startMfaSetup() {
codeInput.value = codeInput.value.replace(/[^0-9]/g, "");
});
// QR fallback (#87, ex-onerror inline) : si l'image ne charge pas,
// afficher la saisie manuelle du secret.
const qrImgEl = document.getElementById("mfa-qr-img");
if (qrImgEl) {
qrImgEl.addEventListener("error", () => {
qrImgEl.style.display = "none";
const fallback = document.getElementById("mfa-qr-fallback");
if (fallback) fallback.style.display = "block";
});
}
document.getElementById("mfa-confirm-btn").addEventListener("click", async () => {
const code = codeInput.value.trim();
if (code.length !== 6) return;
+2
View File
@@ -60,12 +60,14 @@ const MUTATING_TOOLS = new Set([
'rename_file', 'rename_directory', 'move_path', 'replace_in_files',
'delete_file', 'delete_directory', 'restore_backup',
'create_xlsx', 'create_docx', 'create_csv', 'create_pdf',
'update_xlsx_cells', 'append_xlsx_rows',
]);
// Subset carrying a concrete `vault` + `path`: the displayed document is
// reloaded from disk so an open viewer/editor reflects the agent's write.
const FILE_WRITE_TOOLS = new Set([
'edit_file', 'append_to_file', 'create_file', 'restore_backup',
'create_xlsx', 'create_docx', 'create_csv', 'create_pdf',
'update_xlsx_cells', 'append_xlsx_rows',
]);
/**
+40 -4
View File
@@ -363,6 +363,27 @@ function initHelpModal() {
}
});
// Help TOC hamburger + search clear (#87, ex-onclick inline in index.html).
var helpHamburger = document.getElementById("help-hamburger");
if (helpHamburger) {
helpHamburger.addEventListener("click", function() {
var n = document.getElementById("help-nav");
if (n) {
var d = n.style.display;
n.style.display = d === "none" || d === "" ? "flex" : "none";
}
});
}
var helpSearch = document.getElementById("help-nav-search");
var helpSearchClear = document.getElementById("help-search-clear");
if (helpSearchClear && helpSearch) {
helpSearchClear.addEventListener("click", function() {
helpSearch.value = "";
helpSearch.dispatchEvent(new Event("input"));
helpSearch.focus();
});
}
document.addEventListener("keydown", (e) => {
if (e.key === "Escape" && modal.classList.contains("active")) {
closeHelpModal();
@@ -883,7 +904,7 @@ function initConfigModal() {
});
}
// Logout button — handled inline in index.html (onclick)
// Logout button — wired here with addEventListener (#87, no inline onclick)
// Config nav search
var cfgSearch = document.getElementById("config-nav-search");
@@ -901,6 +922,16 @@ function initConfigModal() {
});
}
// Config search clear button (#87, ex-onclick inline).
var cfgSearchClear = document.getElementById("config-search-clear");
if (cfgSearchClear && cfgSearch) {
cfgSearchClear.addEventListener("click", function() {
cfgSearch.value = "";
cfgSearch.dispatchEvent(new Event("input"));
cfgSearch.focus();
});
}
// BUG-071/#114: mobile table of contents. #config-nav shares the .help-nav
// rule that hides it below 768px, but — unlike the help modal — the config
// modal had no toggle to reveal it, leaving mobile users with no way to
@@ -1191,7 +1222,7 @@ function renderDiagnostics(container, data) {
["Postings total", data.inverted_index.total_postings.toLocaleString()],
["Documents", data.inverted_index.documents],
["Mémoire estimée", data.inverted_index.memory_estimate_mb + " MB"],
["Stale", data.inverted_index.is_stale ? "Oui" : "Non"],
["Index prêt", data.inverted_index.is_ready ? "Oui" : "Non"],
],
},
{
@@ -1575,13 +1606,15 @@ export async function openShareDialog(vault, path) {
<p style="font-size:0.85rem;color:var(--text-muted);margin-bottom:4px">${escapeHtml(vault)}/${escapeHtml(path)}</p>
${expiresInfo}
<p style="font-size:0.75rem;color:var(--text-muted);margin-bottom:8px">${existingShare.access_count} vue(s)</p>
<input type="text" class="share-url-input" value="${url}" readonly onclick="this.select()">
<input type="text" class="share-url-input" value="${url}" readonly>
<div class="share-dialog-actions">
<button class="share-copy-btn">📋 Copier le lien</button>
<button class="share-revoke-btn">🗑 Révoquer</button>
<button class="share-close-btn">Fermer</button>
</div>
</div>`;
const shareUrlInput = div.querySelector(".share-url-input");
if (shareUrlInput) shareUrlInput.addEventListener("click", function() { shareUrlInput.select(); });
div.querySelector(".share-copy-btn").addEventListener("click", async () => {
try {
await navigator.clipboard.writeText(url);
@@ -2525,7 +2558,10 @@ function initProfile() {
} catch(e) {}
});
// Logout button — handled inline in index.html (onclick)
// Logout button (#87, ex-onclick inline in index.html).
if (logoutBtn && window.handleLogout) {
logoutBtn.addEventListener('click', function() { window.handleLogout(); });
}
// ── Avatar (#113) ────────────────────────────────────────────────
var avatarField = document.getElementById('profile-avatar-field');
+1 -1
View File
@@ -35,7 +35,7 @@ export function renderExcalidraw(container, data, vaultName, filePath, opts = {}
// Build the iframe
const iframe = document.createElement('iframe');
iframe.id = editorId;
iframe.src = '/static/excalidraw-editor.html?v=' + Date.now();
iframe.src = '/excalidraw-editor.html?v=' + Date.now();
iframe.sandbox.add('allow-scripts');
iframe.sandbox.add('allow-same-origin');
// Let the editor's own Fullscreen button work (native Fullscreen API inside
+7 -6
View File
@@ -478,8 +478,8 @@ function openTemplateModal() {
'</div>' +
'</div>' +
'<div class="modal-footer">' +
'<button class="btn btn-primary" data-i18n="common.close" onclick="this.closest(\'.modal-overlay\').remove()">Close</button>' +
'<button class="btn btn-secondary" onclick="copyTemplate()">Copy to Clipboard</button>' +
'<button class="btn btn-primary btn-modal-close" data-i18n="common.close">Close</button>' +
'<button class="btn btn-secondary btn-copy-template">Copy to Clipboard</button>' +
'</div>' +
'</div>';
@@ -487,11 +487,11 @@ function openTemplateModal() {
modal.querySelector('.modal-close')?.addEventListener('click', () => modal.remove());
modal.addEventListener('click', (e) => { if (e.target === modal) modal.remove(); });
window.copyTemplate = () => {
modal.querySelector('.btn-modal-close')?.addEventListener('click', () => modal.remove());
modal.querySelector('.btn-copy-template')?.addEventListener('click', () => {
navigator.clipboard.writeText(JSON.stringify(template, null, 2));
showToast('Template copied to clipboard', 'success');
};
});
});
}
@@ -508,12 +508,13 @@ function openCodeModal(name, code) {
'<pre class="code-block" style="max-height: 500px; overflow: auto;">' + escapeHtml(code) + '</pre>' +
'</div>' +
'<div class="modal-footer">' +
'<button class="btn btn-primary" data-i18n="common.close" onclick="this.closest(\'.modal-overlay\').remove()">Close</button>' +
'<button class="btn btn-primary btn-modal-close" data-i18n="common.close">Close</button>' +
'</div>' +
'</div>';
document.body.appendChild(modal);
modal.querySelector('.modal-close')?.addEventListener('click', () => modal.remove());
modal.querySelector('.btn-modal-close')?.addEventListener('click', () => modal.remove());
modal.addEventListener('click', (e) => { if (e.target === modal) modal.remove(); });
}
+8 -2
View File
@@ -543,10 +543,16 @@ function showUpdateNotification() {
message.innerHTML = `
<div class="pwa-update-content">
<span>Une nouvelle version d'ObsiGate est disponible !</span>
<button class="pwa-update-btn" onclick="window.location.reload()">Mettre à jour</button>
<button class="pwa-update-dismiss" onclick="this.parentElement.parentElement.remove()">×</button>
<button class="pwa-update-btn">Mettre à jour</button>
<button class="pwa-update-dismiss">×</button>
</div>
`;
message.querySelector(".pwa-update-btn").addEventListener("click", function() {
window.location.reload();
});
message.querySelector(".pwa-update-dismiss").addEventListener("click", function() {
message.remove();
});
document.body.appendChild(message);
// Auto-dismiss after 30 seconds
+681 -14
View File
@@ -998,27 +998,159 @@ export function renderVideoViewer(area, data) {
// ── Excel .xlsx — sheet tabs + editable cells ─────────────────────────────
// Cells are contenteditable; edits are collected per sheet and sent to
// PUT /api/file/{vault}/xlsx/save. Formula cells show their text and are
// saved back as formulas (no client-side recalculation — ceiling accepted).
function renderXlsxViewer(area, data) {
// never recalculated here.
//
// #153 A1/A4 — the read response carries `xlsx_lossy_features` (parts openpyxl
// drops on save): a banner lists them and the first save asks for an explicit
// confirmation before retrying with `force: true`. A value starting with "=" or
// "@" is stored as text unless the user turns the formula toggle on, so a typed
// `=cmd|…` cannot execute when the file is later opened in Excel.
// #153 A8 — a sheet bigger than the render caps used to be silently cut: the
// user saw a short table and no way to tell the rest of the workbook still
// existed. The backend now reports the real dimensions of every sheet, so the
// note states exactly what is hidden (and that those cells are not editable
// here — the workbook itself is untouched). A payload without those fields
// (older cache) simply shows no note.
function truncationNote(sheet) {
// The cap is the real "shown" figure, not `rows`/`cols`: those are post-trim
// (a sparse sheet renders 1x1) while the note must say how far the view
// reaches.
const cap = { rows: Number(sheet.max_rows) || 0, cols: Number(sheet.max_cols) || 0 };
const reasons = [];
if (Number(sheet.total_rows) > cap.rows) {
reasons.push(t("xlsx.truncated_rows", { shown: cap.rows, total: Number(sheet.total_rows) }));
}
if (Number(sheet.total_cols) > cap.cols) {
reasons.push(t("xlsx.truncated_cols", { shown: cap.cols, total: Number(sheet.total_cols) }));
}
if (!reasons.length) return "";
return `<div class="xlsx-truncated" role="note">
<i data-lucide="scissors" class="xlsx-truncated-icon"></i>
<div class="xlsx-warning-body">
<strong>${escapeHtml(t("xlsx.truncated_title"))}</strong>
<span>${escapeHtml(reasons.join(" "))}</span>
<span class="xlsx-warning-hint">${escapeHtml(t("xlsx.truncated_hint"))}</span>
</div>
</div>`;
}
export function renderXlsxViewer(area, data) {
const sheets = data.xlsx_sheets || [];
const lossy = data.xlsx_lossy_features || [];
// Session-scoped state: once the lossy write is confirmed, the rest of the
// session saves without asking again (never persisted — a confirmation is
// per workbook, not a global preference).
let lossyConfirmed = false;
let allowFormula = false;
const tabs = sheets.length > 1
? `<div class="xlsx-tabs">${sheets.map((s, i) =>
`<button class="xlsx-tab${i === 0 ? " active" : ""}" data-sheet="${i}">${escapeHtml(s.name)}</button>`
).join("")}</div>`
: "";
// #153 A15 — inline styles (bold/italic/colors from the file itself), merged
// ranges and the freeze anchor are applied AFTER the panels are in the DOM
// (the HTML string alone cannot carry them: col/row spans need the table).
const applySheetMeta = (panel) => {
const meta = sheets[Number(panel.dataset.sheet)] || {};
const styleMap = meta.styles || {};
const alignMap = meta.aligns || {};
panel.querySelectorAll(".xlsx-table td[data-cell]").forEach((td) => {
const ref = td.dataset.cell;
const style = styleMap[ref];
if (style) td.setAttribute("style", style);
const align = alignMap[ref];
if (align) td.style.textAlign = align;
});
// Merged ranges: span the anchor cell over the range and hide the covered
// cells (rowSpan/colSpan survive a re-render; hidden cells are skipped by
// the save collector because their text was already blank in openpyxl).
(meta.merges || []).forEach((range) => {
const [from, to] = range.split(":");
const anchor = panel.querySelector(`td[data-cell="${from}"]`);
const last = panel.querySelector(`td[data-cell="${to}"]`);
if (!anchor || !last || anchor === last) return;
const a = parseRef(from);
const b = parseRef(to);
if (!a || !b) return;
anchor.rowSpan = Math.max(1, b.row - a.row + 1);
anchor.colSpan = Math.max(1, b.col - a.col + 1);
// Hide every covered cell of the block.
for (let r = a.row; r <= b.row; r++) {
for (let c = a.col; c <= b.col; c++) {
if (r === a.row && c === a.col) continue;
const covered = panel.querySelector(`td[data-cell="${columnName(c)}${r}"]`);
if (covered) covered.style.display = "none";
}
}
});
// Freeze panes: sticky the rows/cols left of and above the anchor.
const freeze = meta.freeze;
if (freeze && freeze !== "A1") {
const f = parseRef(freeze);
if (f) {
panel.querySelectorAll(".xlsx-table tbody tr").forEach((tr) => {
const ref = tr.querySelector("td")?.dataset.cell || "";
const p = parseRef(ref);
if (p && p.row < f.row) tr.classList.add("xlsx-frozen-row");
});
panel.querySelectorAll(".xlsx-table td[data-cell]").forEach((td) => {
const p = parseRef(td.dataset.cell);
if (p && p.col < f.col) td.classList.add("xlsx-frozen-col");
});
}
}
};
const panels = sheets.map((s, i) =>
`<div class="xlsx-panel" data-sheet="${i}"${i === 0 ? "" : ' style="display:none"'}>${s.html}</div>`
`<div class="xlsx-panel" data-sheet="${i}"${i === 0 ? "" : ' style="display:none"'}>${truncationNote(s)}${s.html}</div>`
).join("");
const lossWarning = lossy.length
? `<div class="xlsx-warning" role="note">
<i data-lucide="alert-triangle" class="xlsx-warning-icon"></i>
<div class="xlsx-warning-body">
<strong>${escapeHtml(t("xlsx.lossy_title"))}</strong>
<span class="xlsx-warning-list">${lossy
.map((f) => `<span class="xlsx-warning-tag">${escapeHtml(t("xlsx.feature_" + f))}</span>`)
.join("")}</span>
<span class="xlsx-warning-hint">${escapeHtml(t("xlsx.lossy_hint"))}</span>
</div>
</div>`
: "";
area.innerHTML = `
<div class="xlsx-viewer">
<div class="xlsx-toolbar">
${tabs}
<span class="xlsx-toolbar-actions">
<button class="btn-action xlsx-formula-toggle" id="xlsx-formula-btn" type="button"
aria-pressed="false" title="${escapeHtml(t("xlsx.formula_toggle_title"))}">f(x)</button>
<button class="btn-action" id="xlsx-save-btn" disabled>${t("common.save")}</button>
<button class="btn-action" id="xlsx-download-btn">
<i data-lucide="download" style="width:14px;height:14px"></i> ${t("viewer.download")}
</button>
<button class="btn-action" id="xlsx-csv-btn" title="${escapeHtml(t("xlsx.csv_export"))}">
<i data-lucide="file-spreadsheet" style="width:14px;height:14px"></i> CSV
</button>
<button class="btn-action" id="xlsx-structure-btn" title="${escapeHtml(t("xlsx.structure_btn"))}">
<i data-lucide="table-properties" style="width:14px;height:14px"></i>
</button>
</span>
</div>
${lossWarning}
<div class="xlsx-formula-bar">
<span class="xlsx-active-cell" id="xlsx-active-cell">A1</span>
<i data-lucide="chevron-right" class="xlsx-formula-sep"></i>
<input type="text" class="xlsx-formula-input" id="xlsx-formula-input"
spellcheck="false" placeholder="${escapeHtml(t("xlsx.formula_bar_placeholder"))}" />
<span class="xlsx-find-group">
<input type="text" class="xlsx-find-input" id="xlsx-find-input"
spellcheck="false" placeholder="${escapeHtml(t("xlsx.find_placeholder"))}" />
<span class="xlsx-find-count" id="xlsx-find-count"></span>
<button class="btn-action xlsx-find-btn" id="xlsx-find-prev" title="${escapeHtml(t("xlsx.find_prev"))}">↑</button>
<button class="btn-action xlsx-find-btn" id="xlsx-find-next" title="${escapeHtml(t("xlsx.find_next"))}">↓</button>
<button class="btn-action xlsx-find-case" id="xlsx-find-case" aria-pressed="false" title="${escapeHtml(t("xlsx.find_case"))}">Aa</button>
<button class="btn-action xlsx-sort-reset" id="xlsx-sort-reset" title="${escapeHtml(t("xlsx.sort_reset"))}" style="display:none">
<i data-lucide="rotate-ccw" style="width:14px;height:14px"></i>
</button>
</span>
</div>
<div class="xlsx-panels">${panels}</div>
@@ -1029,30 +1161,243 @@ function renderXlsxViewer(area, data) {
const dirtyCount = () => area.querySelectorAll("td.xlsx-dirty").length;
const refreshSaveState = () => { saveBtn.disabled = dirtyCount() === 0; };
// Editable cells: Enter blurs, Escape reverts, paste stays single-line.
area.querySelectorAll(".xlsx-table td").forEach((td) => {
// #153 A9bis — the first render stops at MAX_ROWS/MAX_COLS; the tail is
// fetched window by window from GET …/xlsx/sheet when the user reaches the
// end of a truncated sheet (scroll sentinel) or clicks « Charger la suite ».
// Appended rows reuse the exact same edit pipeline as the initial render.
const wireLazyRows = (panel) => {
const meta = sheets[Number(panel.dataset.sheet)] || {};
if (!meta.truncated) return;
const wrapper = panel.querySelector(".csv-table-wrapper");
const table = panel.querySelector(".xlsx-table tbody");
if (!wrapper || !table) return;
let offset = Number(meta.rows) || 0;
const total = Number(meta.total_rows) || 0;
let loading = false;
let done = offset >= total;
const foot = document.createElement("div");
foot.className = "xlsx-load-more";
const refreshFoot = () => {
foot.textContent = done
? ""
: `${t("xlsx.truncated_rows", { shown: offset, total })} · ${t("xlsx.load_more")}`;
foot.classList.toggle("done", done);
};
refreshFoot();
foot.addEventListener("click", () => { if (!done) loadMore(); });
wrapper.insertAdjacentElement("afterend", foot);
const appendWindow = (win) => {
const doc = new DOMParser().parseFromString(`<table>${win.html}</table>`, "text/html");
// No `tbody` selector: the fragment embeds its own wrapper div, so the
// parse yields bare `<tr>` inside `<table>` (the rows we want) — while
// any `tbody` in the fragment belongs to the *embedded* wrapper table.
doc.querySelectorAll("tr").forEach((tr) => table.appendChild(tr));
offset = win.offset + win.rows;
done = !win.has_more;
refreshFoot();
// The new rows must behave like the initial ones: contenteditable,
// dirty tracking, cached-value tooltip, icons in the fresh footnote.
panel.querySelectorAll("tbody tr:not([data-wired]) td").forEach(setupCell);
panel.querySelectorAll("tbody tr").forEach((tr) => tr.setAttribute("data-wired", "1"));
const cachedEls = panel.querySelectorAll(".xlsx-cached[data-cached-value]");
cachedEls.forEach((el) => { if (!el.title) el.title = t("xlsx.cached_value_title"); });
// #153 A15 — the window carries its own style/merge metadata: fold it
// into the sheet meta and re-apply, so lazy rows look like the initial
// render (applySheetMeta is idempotent over already-processed cells).
const sheetMeta = sheets[Number(panel.dataset.sheet)];
if (sheetMeta) {
Object.assign(sheetMeta.styles || (sheetMeta.styles = {}), win.styles || {});
Object.assign(sheetMeta.aligns || (sheetMeta.aligns = {}), win.aligns || {});
if (!sheetMeta.merges) sheetMeta.merges = [];
(win.merges || []).forEach((m) => {
if (!sheetMeta.merges.includes(m)) sheetMeta.merges.push(m);
});
if (win.freeze && !sheetMeta.freeze) sheetMeta.freeze = win.freeze;
applySheetMeta(panel);
}
safeCreateIcons();
};
const loadMore = async () => {
if (loading || done) return;
loading = true;
const label = foot.textContent;
foot.textContent = t("xlsx.loading_more");
try {
const win = await api(
`/api/file/${encodeURIComponent(data.vault)}/xlsx/sheet?path=${encodeURIComponent(data.path)}` +
`&sheet=${encodeURIComponent(sheets[Number(panel.dataset.sheet)].name)}` +
`&offset=${offset}&limit=500`,
);
appendWindow(win);
} catch (err) {
foot.textContent = label; // restore: a failed fetch must not eat the button
showToast(`${t("xlsx.load_error")}: ${err.message || err}`, "error");
} finally {
loading = false;
}
};
// Scroll sentinel: near the bottom of the wrapper, pull the next window.
wrapper.addEventListener("scroll", () => {
if (loading || done) return;
if (wrapper.scrollTop + wrapper.clientHeight >= wrapper.scrollHeight - 120) loadMore();
});
};
// ── #153 A7 — keyboard navigation & formula bar ────────────────────────
// One active cell per viewer: clicking or arrowing into a cell shows its
// A1 name in the bar; the input mirrors the cell text and typing there
// edits the cell live (Enter commits, Escape reverts, then re-focuses).
let activeTd = null;
const activeCellEl = area.querySelector("#xlsx-active-cell");
const formulaInput = area.querySelector("#xlsx-formula-input");
const syncing = { value: false }; // guard against input-event feedback loops
const cellName = (td) => td?.dataset.cell || "";
const setActiveCell = (td) => {
activeTd = td || null;
if (area.querySelector("td.xlsx-active")) area.querySelector("td.xlsx-active").classList.remove("xlsx-active");
if (!td) {
activeCellEl.textContent = "—";
formulaInput.value = "";
formulaInput.disabled = true;
return;
}
td.classList.add("xlsx-active");
activeCellEl.textContent = cellName(td);
formulaInput.disabled = false;
formulaInput.value = td.textContent;
};
const syncFormulaBar = (td) => {
if (td !== activeTd) return;
if (!syncing.value) formulaInput.value = td.textContent;
};
// Parse an A1 reference into its (row, col) parts.
const parseRef = (ref) => {
const m = /^([A-Z]+)(\d+)$/.exec(ref || "");
if (!m) return null;
let col = 0;
for (const ch of m[1]) col = col * 26 + (ch.charCodeAt(0) - 64);
return { row: Number(m[2]), col };
};
const findTd = (panel, row, col) =>
panel.querySelector(`td[data-cell="${columnName(col)}${row}"]`);
const columnName = (col) => {
let name = "";
while (col > 0) {
const rem = (col - 1) % 26;
name = String.fromCharCode(65 + rem) + name;
col = Math.floor((col - 1) / 26);
}
return name;
};
const moveActive = (td, key, forward = true) => {
const ref = parseRef(cellName(td));
if (!ref) return;
let { row, col } = ref;
if (key === "Tab") col += forward ? 1 : -1;
else if (key === "ArrowRight") col += 1;
else if (key === "ArrowLeft") col -= 1;
else if (key === "ArrowDown") row += 1;
else if (key === "ArrowUp") row -= 1;
if (row < 1 || col < 1) return;
const panel = td.closest(".xlsx-panel");
const next = findTd(panel, row, col);
if (!next) return; // edge of the rendered window: no wrap
td.blur();
next.focus();
setActiveCell(next);
};
formulaInput.addEventListener("input", () => {
if (!activeTd) return;
syncing.value = true;
activeTd.textContent = formulaInput.value;
syncing.value = false;
activeTd.classList.add("xlsx-dirty");
refreshSaveState();
});
formulaInput.addEventListener("keydown", (e) => {
if (!activeTd) return;
if (e.key === "Enter") {
e.preventDefault();
activeTd.blur();
activeTd.focus();
} else if (e.key === "Escape") {
e.preventDefault();
activeTd.textContent = activeTd.dataset.orig;
activeTd.classList.remove("xlsx-dirty");
syncFormulaBar(activeTd);
refreshSaveState();
activeTd.focus();
}
});
const setupCell = (td) => {
td.contentEditable = "true";
td.spellcheck = false;
// Focusable without a pointing device: JSDOM requires it to fire focus
// events on contenteditable cells, and a keyboard user tabbing into the
// table from outside lands on the first cell thanks to it.
td.tabIndex = 0;
td.dataset.orig = td.textContent;
td.addEventListener("input", () => {
td.classList.add("xlsx-dirty");
syncFormulaBar(td);
refreshSaveState();
});
td.addEventListener("focus", () => setActiveCell(td));
td.addEventListener("keydown", (e) => {
if (e.key === "Enter") { e.preventDefault(); td.blur(); }
if (e.key === "Enter" && !e.shiftKey) { e.preventDefault(); td.blur(); }
if (e.key === "Escape") {
td.textContent = td.dataset.orig;
td.classList.remove("xlsx-dirty");
syncFormulaBar(td);
refreshSaveState();
}
if (e.key === "Tab" || e.key === "ArrowUp" || e.key === "ArrowDown"
|| e.key === "ArrowLeft" || e.key === "ArrowRight") {
// Excel-like: arrows and Tab move to the neighbour cell. Direction is
// the key's own (Left/Up = back, Right/Down = forward); Tab follows
// Shift. Prevented so Tab never leaves the table and arrows never
// move the caret (they move the SELECTION instead).
e.preventDefault();
const forward = e.key === "Tab"
? !e.shiftKey
: (e.key === "ArrowRight" || e.key === "ArrowDown");
moveActive(td, e.key, forward);
}
});
td.addEventListener("paste", (e) => {
e.preventDefault();
const text = (e.clipboardData || window.clipboardData).getData("text").replace(/\r?\n/g, " ");
document.execCommand("insertText", false, text);
});
};
// #153 A12 — the backend marks the last value Excel computed; the wording is
// translated here so the tooltip follows the UI language.
area.querySelectorAll(".xlsx-cached[data-cached-value]").forEach((el) => {
el.title = t("xlsx.cached_value_title");
});
// Editable cells: Enter blurs, Escape reverts, arrows/Tab navigate.
area.querySelectorAll(".xlsx-table td").forEach(setupCell);
panelEls.forEach(wireLazyRows);
panelEls.forEach(applySheetMeta);
// The formula bar starts disabled: nothing is selected yet.
setActiveCell(null);
area.querySelectorAll(".xlsx-tab").forEach((tab) => {
tab.addEventListener("click", () => {
const idx = tab.dataset.sheet;
@@ -1061,6 +1406,23 @@ function renderXlsxViewer(area, data) {
});
});
// Formula toggle (#153 A4) — opt-in for this viewing session only.
const formulaBtn = area.querySelector("#xlsx-formula-btn");
formulaBtn.addEventListener("click", () => {
allowFormula = !allowFormula;
formulaBtn.setAttribute("aria-pressed", String(allowFormula));
formulaBtn.classList.toggle("active", allowFormula);
});
const putSheet = (job, force) => api(
`/api/file/${encodeURIComponent(data.vault)}/xlsx/save?path=${encodeURIComponent(data.path)}`,
{
method: "PUT",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ ...job, allow_formula: allowFormula, force }),
},
);
saveBtn.addEventListener("click", async () => {
// One PUT per sheet (dirty cells can span tabs before a save).
const jobs = panelEls
@@ -1074,21 +1436,41 @@ function renderXlsxViewer(area, data) {
saveBtn.disabled = true;
try {
for (const job of jobs) {
await api(`/api/file/${encodeURIComponent(data.vault)}/xlsx/save?path=${encodeURIComponent(data.path)}`, {
method: "PUT",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(job),
});
// 409 xlsx_lossy_content → confirm once, then retry with force: true.
// (Also covers a workbook that became lossy while it was open.)
let force = lossyConfirmed;
for (;;) {
try {
await putSheet(job, force);
break;
} catch (err) {
if (err && err.code === "xlsx_lossy_content" && !lossyConfirmed) {
const features = (err.details && err.details.features) || lossy;
const labels = features.map((f) => t("xlsx.feature_" + f)).join(", ");
if (!confirm(t("xlsx.lossy_confirm", { features: labels }))) throw err;
lossyConfirmed = true;
force = true;
continue;
}
throw err;
}
}
}
area.querySelectorAll("td.xlsx-dirty").forEach((td) => {
td.classList.remove("xlsx-dirty");
td.dataset.orig = td.textContent;
});
if (activeTd) syncFormulaBar(activeTd);
refreshSaveState();
showToast(t("editor.saved"), "success");
} catch (err) {
refreshSaveState();
showToast(`${t("editor.save_error")}: ${err.message || err}`, "error");
// A refused confirmation is a decision, not a failure: neutral toast.
if (err && err.code === "xlsx_lossy_content") {
showToast(t("xlsx.lossy_cancelled"), "info");
} else {
showToast(`${t("editor.save_error")}: ${err.message || err}`, "error");
}
}
});
@@ -1096,6 +1478,286 @@ function renderXlsxViewer(area, data) {
window.open(`/api/file/${encodeURIComponent(data.vault)}/download?path=${encodeURIComponent(data.path)}`, "_blank");
});
// ── #153 A13 — sort / filter / find in the sheet + CSV export ──────────
// ALL of these act on the RENDERED grid only: the workbook is never
// rewritten by a sort or a filter (the save pipeline stays the only write
// path, and the note in xlsx.sort_applied says so).
const visiblePanel = () =>
panelEls.find((p) => p.style.display !== "none") || panelEls[0];
// Sorting reorders <tr> rows by the text of one column. Dirty cells travel
// with their row, so a sort never loses an unsaved edit.
let sortState = null; // { col, dir } on the visible panel
const applySort = (panel, col, dir) => {
const tbody = panel.querySelector(".xlsx-table tbody");
if (!tbody) return;
const rows = [...tbody.querySelectorAll("tr")];
const numeric = rows.every((tr) => {
const td = tr.querySelector(`td[data-cell^="${columnName(col)}"]`);
const v = td ? td.textContent.trim() : "";
return v === "" || !isNaN(Number(v));
});
rows.sort((a, b) => {
const ta = a.querySelector(`td[data-cell^="${columnName(col)}"]`);
const tb = b.querySelector(`td[data-cell^="${columnName(col)}"]`);
const va = ta ? ta.textContent.trim() : "";
const vb = tb ? tb.textContent.trim() : "";
const cmp = numeric
? (parseFloat(va) || 0) - (parseFloat(vb) || 0)
: va.localeCompare(vb, "fr");
return dir === "asc" ? cmp : -cmp;
});
rows.forEach((tr) => tbody.appendChild(tr));
sortState = { col, dir };
sortResetBtn.style.display = "";
showToast(t("xlsx.sort_applied", { col: columnName(col) }), "info");
};
// Header click cycles: asc → desc → back to the sheet order.
const wireHeaderSort = (panel) => {
const thead = panel.querySelector(".xlsx-table thead");
if (!thead || thead.dataset.sortWired) return;
thead.dataset.sortWired = "1";
thead.querySelectorAll("th:not(.xlsx-corner)").forEach((th) => {
th.style.cursor = "pointer";
th.title = t("xlsx.sort_asc");
th.addEventListener("click", () => {
// The corner th is column 0, so the child index IS the column number.
const col = [...th.parentElement.children].indexOf(th);
const current = sortState && sortState.col === col ? sortState.dir : null;
const dir = current === "asc" ? "desc" : "asc";
applySort(panel, col, dir);
th.title = dir === "asc" ? t("xlsx.sort_desc") : t("xlsx.sort_asc");
});
});
};
// Filter: rows whose cell text lacks the needle are hidden.
const applyFilter = (panel, needle) => {
const n = needle.trim().toLowerCase();
panel.querySelectorAll(".xlsx-table tbody tr").forEach((tr) => {
const text = tr.textContent.toLowerCase();
tr.style.display = !n || text.includes(n) ? "" : "none";
});
if (n) sortResetBtn.style.display = "";
};
panelEls.forEach(wireHeaderSort);
const sortResetBtn = area.querySelector("#xlsx-sort-reset");
sortResetBtn.addEventListener("click", () => {
// Full reset: re-render the current file (server truth), clear find too.
sortResetBtn.style.display = "none";
renderXlsxViewer(area, data);
});
// Find-in-sheet: highlight matches, navigate with ↑/↓.
let findMatches = [];
let findIndex = -1;
const findInput = area.querySelector("#xlsx-find-input");
const findCount = area.querySelector("#xlsx-find-count");
let findCase = false;
const clearFind = () => {
findMatches.forEach(({ td, mark }) => {
mark.replaceWith(document.createTextNode(mark.textContent));
td.normalize();
});
findMatches = [];
findIndex = -1;
findCount.textContent = "";
};
const runFind = () => {
clearFind();
const needle = findInput.value;
if (!needle.trim()) return;
const panel = visiblePanel();
const hay = findCase ? (s) => s : (s) => s.toLowerCase();
const n = hay(needle);
panel.querySelectorAll(".xlsx-table td").forEach((td) => {
const text = td.textContent;
const pos = hay(text).indexOf(n);
if (pos === -1 || !text) return;
// Wrap the match in a <mark> by splitting the cell's first text node at
// the hit (cells are text-first; the cached-value span stays untouched).
const first = td.firstChild;
if (first && first.nodeType === 3) {
const mark = document.createElement("mark");
mark.className = "xlsx-find-hit";
mark.textContent = text.substr(pos, needle.length);
const tail = first.splitText(pos);
tail.splitText(needle.length);
td.replaceChild(mark, tail);
findMatches.push({ td, mark });
}
});
findCount.textContent = findMatches.length
? t("xlsx.find_count", { index: 1, count: findMatches.length })
: t("xlsx.find_no_match");
if (findMatches.length) focusMatch(0);
};
const focusMatch = (i) => {
findMatches.forEach(({ mark }) => mark.classList.remove("xlsx-find-current"));
findIndex = (i + findMatches.length) % findMatches.length;
const { td, mark } = findMatches[findIndex];
mark.classList.add("xlsx-find-current");
// scrollIntoView is missing in JSDOM; guard it (real browsers have it).
if (typeof td.scrollIntoView === "function") td.scrollIntoView({ block: "nearest" });
findCount.textContent = t("xlsx.find_count", { index: findIndex + 1, count: findMatches.length });
};
// One input drives both the highlight (find) and the row filter (A13):
// typing narrows the sheet to the matching rows AND highlights the hits.
findInput.addEventListener("input", () => {
panelEls.forEach((p) => applyFilter(p, findInput.value));
runFind();
});
findInput.addEventListener("keydown", (e) => {
if (e.key === "Enter") { e.preventDefault(); focusMatch(findIndex + (e.shiftKey ? -1 : 1)); }
});
area.querySelector("#xlsx-find-prev").addEventListener("click", () => focusMatch(findIndex - 1));
area.querySelector("#xlsx-find-next").addEventListener("click", () => focusMatch(findIndex + 1));
const caseBtn = area.querySelector("#xlsx-find-case");
caseBtn.addEventListener("click", () => {
findCase = !findCase;
caseBtn.setAttribute("aria-pressed", String(findCase));
caseBtn.classList.toggle("active", findCase);
runFind();
});
// ── #153 A14 — workbook structure menu (sheets, rows, columns) ─────────
// Every action is an explicit user gesture (prompt/confirm) and goes to
// PUT …/xlsx/structure — one locked, atomic rewrite with a backup.
const visibleSheetIndex = () =>
Number((visiblePanel() || panelEls[0])?.dataset.sheet) || 0;
const putStructure = async (actions, force = false) => {
await api(
`/api/file/${encodeURIComponent(data.vault)}/xlsx/structure?path=${encodeURIComponent(data.path)}`,
{
method: "PUT",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ actions, force }),
},
);
showToast(t("xlsx.structure_saved"), "success");
// Re-render from the server so the viewer shows the new structure.
const fresh = await api(
`/api/file/${encodeURIComponent(data.vault)}?path=${encodeURIComponent(data.path)}`,
);
renderXlsxViewer(area, fresh);
};
const structureError = (err, retryActions) => {
if (err && err.code === "xlsx_lossy_content") {
const features = (err.details && err.details.features) || lossy;
const labels = features.map((f) => t("xlsx.feature_" + f)).join(", ");
if (confirm(t("xlsx.lossy_confirm", { features: labels }))) {
return putStructure(retryActions, true); // re-emitted with force
}
showToast(t("xlsx.lossy_cancelled"), "info");
return null;
}
showToast(`${t("xlsx.structure_error")}: ${err.message || err}`, "error");
return null;
};
// Structure menu: built on demand, positioned under the button.
area.querySelector("#xlsx-structure-btn").addEventListener("click", (e) => {
const old = area.querySelector(".xlsx-structure-menu");
if (old) { old.remove(); return; }
const idx = visibleSheetIndex();
const sheetName = sheets[idx]?.name || "";
const activeRef = cellName(activeTd && activeTd.closest(".xlsx-panel") === visiblePanel() ? activeTd : null);
const parsed = parseRef(activeRef);
const menu = document.createElement("div");
menu.className = "xlsx-structure-menu";
const item = (label, fn) => {
const b = document.createElement("button");
b.type = "button";
b.className = "btn-action xlsx-structure-item";
b.textContent = label;
b.addEventListener("click", () => { menu.remove(); fn(); });
menu.appendChild(b);
};
item(t("xlsx.sheet_add"), async () => {
const name = prompt(t("xlsx.structure_prompt_add"));
if (!name) return;
const actions = [{ op: "sheet_add", name }];
try { await putStructure(actions); } catch (err) { structureError(err, actions); }
});
item(t("xlsx.sheet_rename"), async () => {
const to = prompt(t("xlsx.structure_prompt_rename"), sheetName);
if (!to || to === sheetName) return;
const actions = [{ op: "sheet_rename", from: sheetName, to }];
try { await putStructure(actions); } catch (err) { structureError(err, actions); }
});
item(t("xlsx.sheet_duplicate"), async () => {
const as = prompt(t("xlsx.structure_prompt_add"), `${sheetName} (copie)`);
if (!as) return;
const actions = [{ op: "sheet_duplicate", name: sheetName, as }];
try { await putStructure(actions); } catch (err) { structureError(err, actions); }
});
item(t("xlsx.sheet_delete"), async () => {
if (sheets.length <= 1) { showToast(t("xlsx.last_sheet"), "info"); return; }
if (!confirm(t("xlsx.structure_confirm_delete_sheet", { name: sheetName }))) return;
const actions = [{ op: "sheet_delete", name: sheetName }];
try { await putStructure(actions); } catch (err) { structureError(err, actions); }
});
if (parsed) {
menu.appendChild(Object.assign(document.createElement("div"), { className: "xlsx-structure-sep" }));
item(t("xlsx.row_insert"), async () => {
const actions = [{ op: "row_insert", sheet: sheetName, at: parsed.row }];
try { await putStructure(actions); } catch (err) { structureError(err, actions); }
});
item(t("xlsx.row_delete"), async () => {
if (!confirm(t("xlsx.structure_confirm_row", { n: parsed.row }))) return;
const actions = [{ op: "row_delete", sheet: sheetName, at: parsed.row }];
try { await putStructure(actions); } catch (err) { structureError(err, actions); }
});
item(t("xlsx.col_insert"), async () => {
const actions = [{ op: "col_insert", sheet: sheetName, at: parsed.col }];
try { await putStructure(actions); } catch (err) { structureError(err, actions); }
});
item(t("xlsx.col_delete"), async () => {
if (!confirm(t("xlsx.structure_confirm_col", { n: columnName(parsed.col) }))) return;
const actions = [{ op: "col_delete", sheet: sheetName, at: parsed.col }];
try { await putStructure(actions); } catch (err) { structureError(err, actions); }
});
}
e.target.closest(".xlsx-toolbar").appendChild(menu);
});
// CSV export of the visible sheet (post-trim, pre-save data).
area.querySelector("#xlsx-csv-btn").addEventListener("click", () => {
const panel = visiblePanel();
const idx = Number(panel.dataset.sheet);
const rows = [];
panel.querySelectorAll(".xlsx-table tbody tr").forEach((tr) => {
rows.push(
[...tr.querySelectorAll("td")].map((td) => {
const clone = td.cloneNode(true);
clone.querySelectorAll(".xlsx-cached").forEach((el) => el.remove());
return clone.textContent;
}),
);
});
const csv = rows
.map((r) => r.map((v) => (/[";\n]/.test(v) ? `"${v.replace(/"/g, '""')}"` : v)).join(";"))
.join("\n");
const blob = new Blob([`\uFEFF${csv}`], { type: "text/csv;charset=utf-8" });
const a = document.createElement("a");
a.href = URL.createObjectURL(blob);
a.download = `${sheets[idx]?.name || "feuille"}.csv`;
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
URL.revokeObjectURL(a.href);
});
safeCreateIcons();
}
@@ -1140,10 +1802,10 @@ export function renderFile(data) {
<div class="pdf-viewer-container">
<div class="pdf-toolbar">
<span class="pdf-info">PDF — ${pages} pages</span>
<button class="btn-action" onclick="window.open('${pdfUrl}', '_blank')">
<button class="btn-action" data-pdf-url="${pdfUrl}">
<i data-lucide="external-link" style="width:14px;height:14px"></i> Plein écran
</button>
<button class="btn-action" onclick="window.open('/api/file/${encodeURIComponent(data.vault)}/download?path=${encodeURIComponent(data.path)}', '_blank')">
<button class="btn-action" data-download-url="/api/file/${encodeURIComponent(data.vault)}/download?path=${encodeURIComponent(data.path)}">
<i data-lucide="download" style="width:14px;height:14px"></i> Télécharger
</button>
</div>
@@ -1152,6 +1814,11 @@ export function renderFile(data) {
<iframe src="${pdfUrl}" data-pdf-url="${pdfUrl}" class="pdf-iframe" title="${escapeHtml(data.title)}"></iframe>
</div>
</div>`;
area.querySelectorAll('.pdf-toolbar .btn-action').forEach((btn) => {
btn.addEventListener('click', () => {
window.open(btn.dataset.pdfUrl || btn.dataset.downloadUrl, '_blank');
});
});
area.querySelectorAll('.pdf-toc a[data-page]').forEach((link) => {
link.addEventListener('click', (e) => {
e.preventDefault();
+57
View File
@@ -1823,6 +1823,59 @@
"viewer.copy": "Copy",
"viewer.copy_error": "Copy error",
"viewer.download": "Download",
"xlsx.lossy_title": "Simplified save",
"xlsx.lossy_hint": "ObsiGate cannot preserve these elements: saving will ask for your confirmation.",
"xlsx.lossy_confirm": "Save anyway? The following will be lost: {features}",
"xlsx.lossy_cancelled": "Save cancelled",
"xlsx.formula_toggle_title": "Treat “=” and “@” as formulas (off by default)",
"xlsx.cached_value_title": "Last value calculated by Excel",
"xlsx.truncated_title": "Truncated sheet",
"xlsx.truncated_rows": "{shown} of {total} rows displayed.",
"xlsx.truncated_cols": "{shown} of {total} columns displayed.",
"xlsx.truncated_hint": "Cells outside the displayed area cannot be edited here; the workbook is unchanged.",
"xlsx.load_more": "Load more",
"xlsx.loading_more": "Loading…",
"xlsx.load_error": "Could not load the remaining rows",
"xlsx.formula_bar_placeholder": "Active cell content",
"xlsx.active_cell": "Cell",
"xlsx.find_placeholder": "Search in the sheet…",
"xlsx.find_prev": "Previous",
"xlsx.find_next": "Next",
"xlsx.find_case": "Match case",
"xlsx.find_no_match": "No match",
"xlsx.find_count": "{index}/{count}",
"xlsx.csv_export": "Export the sheet as CSV",
"xlsx.sort_asc": "Sort column A→Z",
"xlsx.sort_desc": "Sort column Z→A",
"xlsx.sort_applied": "Sort applied on {col} — display only, the workbook is unchanged",
"xlsx.sort_reset": "Reset sort and filter",
"xlsx.filter_placeholder": "Filter rows…",
"xlsx.structure_btn": "Sheet structure",
"xlsx.structure_title": "Edit the workbook structure",
"xlsx.sheet_add": "Add a sheet",
"xlsx.sheet_rename": "Rename the current sheet",
"xlsx.sheet_duplicate": "Duplicate the current sheet",
"xlsx.sheet_delete": "Delete the current sheet",
"xlsx.row_insert": "Insert a row above",
"xlsx.row_delete": "Delete the active cell's row",
"xlsx.col_insert": "Insert a column to the left",
"xlsx.col_delete": "Delete the active cell's column",
"xlsx.structure_prompt_add": "Name of the new sheet:",
"xlsx.structure_prompt_rename": "New name of the sheet:",
"xlsx.structure_confirm_delete_sheet": "Permanently delete the sheet “{name}”? This changes the file (a backup is created).",
"xlsx.structure_confirm_row": "Delete row {n}? This changes the file (a backup is created).",
"xlsx.structure_confirm_col": "Delete column {n}? This changes the file (a backup is created).",
"xlsx.structure_saved": "Structure updated",
"xlsx.structure_error": "Could not change the structure",
"xlsx.last_sheet": "The last sheet cannot be deleted",
"xlsx.feature_cached_values": "cached values",
"xlsx.feature_slicers": "slicers and timelines",
"xlsx.feature_form_controls": "form controls",
"xlsx.feature_connections": "connections and queries",
"xlsx.feature_custom_xml": "custom XML",
"xlsx.feature_signature": "digital signature",
"xlsx.feature_rich_comments": "rich comments",
"xlsx.feature_macros": "macros",
"viewer.download_md": "Download as .md",
"viewer.download_file": "Download file",
"viewer.pretty": "Pretty",
@@ -1987,6 +2040,10 @@
"ai.step.git_issues": "Searched issues: {value}",
"ai.step.git_file": "Read a repo file: {value}",
"ai.step.xlsx_create": "Spreadsheet proposed: {value}",
"ai.step.xlsx_sheets": "Workbook sheets listed: {value}",
"ai.step.xlsx_read": "Workbook read: {value}",
"ai.step.xlsx_update": "Cells edited: {value}",
"ai.step.xlsx_append": "Rows appended: {value}",
"ai.step.docx_create": "Word document proposed: {value}",
"ai.step.csv_create": "CSV file proposed: {value}",
"ai.step.pdf_create": "PDF document proposed: {value}",
+57
View File
@@ -1823,6 +1823,59 @@
"viewer.copy": "Copier",
"viewer.copy_error": "Erreur lors de la copie",
"viewer.download": "Télécharger",
"xlsx.lossy_title": "Enregistrement simplifié",
"xlsx.lossy_hint": "Ces éléments ne peuvent pas être conservés par ObsiGate : une sauvegarde vous demandera confirmation.",
"xlsx.lossy_confirm": "Enregistrer quand même ? Les éléments suivants seront perdus : {features}",
"xlsx.lossy_cancelled": "Sauvegarde annulée",
"xlsx.formula_toggle_title": "Interpréter « = » et « @ » comme des formules (désactivé par défaut)",
"xlsx.cached_value_title": "Dernière valeur calculée par Excel",
"xlsx.truncated_title": "Feuille tronquée",
"xlsx.truncated_rows": "{shown} lignes affichées sur {total}.",
"xlsx.truncated_cols": "{shown} colonnes affichées sur {total}.",
"xlsx.truncated_hint": "Les cellules hors de l'affichage ne sont pas éditables ici ; le classeur n'est pas modifié.",
"xlsx.load_more": "Charger la suite",
"xlsx.loading_more": "Chargement…",
"xlsx.load_error": "Chargement de la suite impossible",
"xlsx.formula_bar_placeholder": "Contenu de la cellule active",
"xlsx.active_cell": "Cellule",
"xlsx.find_placeholder": "Rechercher dans la feuille…",
"xlsx.find_prev": "Précédent",
"xlsx.find_next": "Suivant",
"xlsx.find_case": "Respecter la casse",
"xlsx.find_no_match": "Aucune correspondance",
"xlsx.find_count": "{index}/{count}",
"xlsx.csv_export": "Exporter la feuille en CSV",
"xlsx.sort_asc": "Trier la colonne A→Z",
"xlsx.sort_desc": "Trier la colonne Z→A",
"xlsx.sort_applied": "Tri appliqué sur {col} — l'affichage seul, le classeur n'est pas modifié",
"xlsx.sort_reset": "Réinitialiser le tri et le filtre",
"xlsx.filter_placeholder": "Filtrer les lignes…",
"xlsx.structure_btn": "Structure de la feuille",
"xlsx.structure_title": "Modifier la structure du classeur",
"xlsx.sheet_add": "Ajouter une feuille",
"xlsx.sheet_rename": "Renommer la feuille courante",
"xlsx.sheet_duplicate": "Dupliquer la feuille courante",
"xlsx.sheet_delete": "Supprimer la feuille courante",
"xlsx.row_insert": "Insérer une ligne au-dessus",
"xlsx.row_delete": "Supprimer la ligne de la cellule active",
"xlsx.col_insert": "Insérer une colonne à gauche",
"xlsx.col_delete": "Supprimer la colonne de la cellule active",
"xlsx.structure_prompt_add": "Nom de la nouvelle feuille :",
"xlsx.structure_prompt_rename": "Nouveau nom de la feuille :",
"xlsx.structure_confirm_delete_sheet": "Supprimer définitivement la feuille « {name} » ? Cette action modifie le fichier (un backup est créé).",
"xlsx.structure_confirm_row": "Supprimer la ligne {n} ? Cette action modifie le fichier (un backup est créé).",
"xlsx.structure_confirm_col": "Supprimer la colonne {n} ? Cette action modifie le fichier (un backup est créé).",
"xlsx.structure_saved": "Structure mise à jour",
"xlsx.structure_error": "Modification de la structure impossible",
"xlsx.last_sheet": "Impossible de supprimer la dernière feuille",
"xlsx.feature_cached_values": "valeurs calculées",
"xlsx.feature_slicers": "segments et chronologies",
"xlsx.feature_form_controls": "contrôles de formulaire",
"xlsx.feature_connections": "connexions et requêtes",
"xlsx.feature_custom_xml": "XML personnalisé",
"xlsx.feature_signature": "signature numérique",
"xlsx.feature_rich_comments": "commentaires enrichis",
"xlsx.feature_macros": "macros",
"viewer.download_md": "Télécharger en .md",
"viewer.download_file": "Télécharger le fichier",
"viewer.pretty": "Pretty",
@@ -1987,6 +2040,10 @@
"ai.step.git_issues": "Issues recherchées : {value}",
"ai.step.git_file": "Fichier de dépôt lu : {value}",
"ai.step.xlsx_create": "Tableur proposé : {value}",
"ai.step.xlsx_sheets": "Feuilles du classeur listées : {value}",
"ai.step.xlsx_read": "Classeur lu : {value}",
"ai.step.xlsx_update": "Cellules modifiées : {value}",
"ai.step.xlsx_append": "Lignes ajoutées : {value}",
"ai.step.docx_create": "Document Word proposé : {value}",
"ai.step.csv_create": "Fichier CSV proposé : {value}",
"ai.step.pdf_create": "Document PDF proposé : {value}",
+283 -2
View File
@@ -10931,6 +10931,7 @@ body.desktop-mode .editor-container {
gap: 10px;
margin-bottom: 8px;
flex-wrap: wrap;
position: relative; /* anchors the A14 structure menu */
}
.xlsx-toolbar-actions {
margin-left: auto;
@@ -10967,12 +10968,25 @@ body.desktop-mode .editor-container {
border-right: 1px solid var(--border-light, var(--border));
position: sticky;
left: 0;
z-index: 1;
/* #153 A8 — `top: auto` is load-bearing: `.csv-table th` pins EVERY `th`
at `top: 0`, so a row number left sticky on both axes piles up in the
top-left corner instead of tracking its own row. */
top: auto;
z-index: 2;
}
.xlsx-table th.xlsx-corner {
left: 0;
top: 0;
z-index: 2;
z-index: 4;
}
/* #153 A8 — the column headers stay visible while the sheet scrolls down.
Declared explicitly (and not inherited from `.csv-table th`) so the stacking
order is intentional: thead (3) < row numbers (2) < corner (4). */
.xlsx-table thead th {
position: sticky;
top: 0;
z-index: 3;
background: var(--surface);
}
.xlsx-table td[contenteditable] {
cursor: text;
@@ -10983,10 +10997,277 @@ body.desktop-mode .editor-container {
outline: 2px solid var(--accent, #4a90d9);
outline-offset: -2px;
}
/* #153 A7 — the active cell keeps its outline even when focus moves to the
formula bar, so the user never loses track of what the bar edits. */
.xlsx-table td.xlsx-active:not(:focus) {
outline: 2px dashed var(--accent, #4a90d9);
outline-offset: -2px;
}
/* #153 A7 — formula bar under the toolbar: [ A1 | > | input ] */
.xlsx-formula-bar {
display: flex;
align-items: center;
gap: 6px;
margin-bottom: 8px;
}
.xlsx-active-cell {
min-width: 52px;
padding: 4px 8px;
border: 1px solid var(--border);
border-radius: 4px;
background: var(--surface);
color: var(--text-primary);
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.8rem;
text-align: center;
font-variant-numeric: tabular-nums;
}
.xlsx-formula-sep {
width: 14px;
height: 14px;
flex: 0 0 auto;
color: var(--text-muted);
}
.xlsx-formula-input {
flex: 1;
min-width: 0;
padding: 5px 10px;
border: 1px solid var(--border);
border-radius: 4px;
background: var(--surface);
color: var(--text-primary);
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-size: 0.82rem;
}
.xlsx-formula-input:focus {
outline: 2px solid var(--accent, #4a90d9);
outline-offset: -2px;
}
.xlsx-formula-input:disabled {
opacity: 0.55;
}
/* #153 A13 — find-in-sheet + filter/sort controls */
.xlsx-find-group {
display: flex;
align-items: center;
gap: 4px;
margin-left: auto;
min-width: 0;
}
.xlsx-find-input {
flex: 1;
min-width: 120px;
max-width: 220px;
padding: 4px 8px;
border: 1px solid var(--border);
border-radius: 4px;
background: var(--surface);
color: var(--text-primary);
font-size: 0.8rem;
}
.xlsx-find-input:focus {
outline: 2px solid var(--accent, #4a90d9);
outline-offset: -2px;
}
.xlsx-find-count {
color: var(--text-muted);
font-size: 0.75rem;
white-space: nowrap;
}
.xlsx-find-btn,
.xlsx-find-case,
.xlsx-sort-reset {
padding: 3px 8px;
font-size: 0.78rem;
}
.xlsx-find-hit {
background: var(--warning, #e0a800);
color: var(--text-primary);
border-radius: 2px;
}
.xlsx-find-current {
outline: 2px solid var(--accent, #4a90d9);
outline-offset: 1px;
}
/* #153 A15 — freeze panes: frozen rows sit UNDER the sticky thead, frozen
columns stay left. z-index mirrors thead (3) without covering it. */
.xlsx-table tr.xlsx-frozen-row td,
.xlsx-table tr.xlsx-frozen-row th {
position: sticky;
top: 33px; /* thead height — keeps the frozen row below the header */
z-index: 2;
background: var(--surface);
}
.xlsx-table td.xlsx-frozen-col {
position: sticky;
left: 44px; /* the row-number column width */
background: var(--surface);
}
/* #153 A14 — structure menu (sheets / rows / columns) */
.xlsx-structure-menu {
position: absolute;
z-index: 30;
display: flex;
flex-direction: column;
gap: 2px;
min-width: 240px;
margin-top: 4px;
padding: 6px;
border: 1px solid var(--border);
border-radius: 6px;
background: var(--surface);
box-shadow: 0 8px 24px var(--shadow, rgba(0, 0, 0, 0.25));
}
.xlsx-structure-item {
text-align: left;
border: none;
background: transparent;
color: var(--text-primary);
}
.xlsx-structure-item:hover {
background: var(--bg-secondary);
}
.xlsx-structure-sep {
height: 1px;
margin: 4px 0;
background: var(--border);
}
/* JSDOM shims for the tests that click anchors */
mark {
font: inherit;
}
.xlsx-table td.xlsx-dirty {
background: rgba(255, 196, 0, 0.18);
}
/* #153 A12 — last result Excel computed, shown under a formula cell.
Discreet by design: the formula is what the user edits, the cached value is
context (stale until Excel recalculates). */
.xlsx-cached {
display: block;
margin-top: 2px;
padding-left: 6px;
border-left: 2px solid var(--border, #d0d7de);
color: var(--text-muted);
font-size: 0.85em;
font-variant-numeric: tabular-nums;
white-space: nowrap;
}
/* #153 A1/A4 — lossy-save warning + formula toggle */
.xlsx-warning {
display: flex;
align-items: flex-start;
gap: 8px;
padding: 8px 10px;
margin-bottom: 8px;
border: 1px solid var(--warning, #e0a800);
border-left-width: 3px;
border-radius: 4px;
background: var(--surface);
color: var(--text-secondary);
font-size: 0.82rem;
line-height: 1.45;
}
.xlsx-warning-icon {
width: 16px;
height: 16px;
flex: 0 0 auto;
margin-top: 1px;
color: var(--warning, #e0a800);
}
.xlsx-warning-body {
display: flex;
flex-direction: column;
gap: 3px;
min-width: 0;
}
.xlsx-warning-body strong {
color: var(--text-primary);
font-weight: 600;
}
.xlsx-warning-list {
display: flex;
flex-wrap: wrap;
gap: 4px;
}
.xlsx-warning-tag {
padding: 1px 6px;
border: 1px solid var(--border);
border-radius: 10px;
background: var(--bg-secondary);
color: var(--text-secondary);
font-size: 0.75rem;
white-space: nowrap;
}
.xlsx-warning-hint {
color: var(--text-secondary);
opacity: 0.85;
}
/* #153 A8 — "feuille tronquée" notice. Deliberately NOT the `.xlsx-warning`
look: that one is a data-loss alert, this one only says part of the sheet is
out of view. */
.xlsx-truncated {
display: flex;
align-items: flex-start;
gap: 8px;
padding: 8px 10px;
margin-bottom: 8px;
border: 1px solid var(--border);
border-left: 3px solid var(--accent, #4a90d9);
border-radius: 4px;
background: var(--bg-secondary);
color: var(--text-secondary);
font-size: 0.82rem;
line-height: 1.45;
}
.xlsx-truncated-icon {
width: 16px;
height: 16px;
flex: 0 0 auto;
margin-top: 1px;
color: var(--accent, #4a90d9);
}
/* #153 A9bis — “charger la suite” footnote under a truncated sheet. Also the
scroll sentinel target: clickable whole, disabled look once the sheet is
fully loaded. */
.xlsx-load-more {
display: block;
margin: 6px 0 10px;
padding: 6px 12px;
border: 1px dashed var(--border);
border-radius: 4px;
background: var(--bg-secondary);
color: var(--text-secondary);
font-size: 0.82rem;
text-align: center;
cursor: pointer;
user-select: none;
}
.xlsx-load-more:hover {
border-color: var(--accent, #4a90d9);
color: var(--text-primary);
}
.xlsx-load-more.done {
display: none;
}
.xlsx-formula-toggle {
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
font-weight: 600;
}
.xlsx-formula-toggle.active {
background: var(--accent, #4a90d9);
border-color: var(--accent, #4a90d9);
color: #fff;
}
/* ── JSON Viewer ── */
.json-viewer {
font-family: 'JetBrains Mono', 'Fira Code', 'Consolas', monospace;
+1 -1
View File
@@ -11,7 +11,7 @@
* cache or Cloudflare does NOT clear the Service Worker Cache Storage, which is
* a separate store. Bumping SW_VERSION invalidates it on every release.
*/
const SW_VERSION = 'v26';
const SW_VERSION = 'v27';
const CODE_CACHE = `obsigate-code-${SW_VERSION}`;
const RUNTIME_CACHE = `obsigate-runtime-${SW_VERSION}`;
const API_CACHE = `obsigate-api-${SW_VERSION}`;
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "obsigate",
"version": "2.27.9",
"version": "2.37.0",
"description": "**Porte d'entrée web ultra-léger pour vos vaults Obsidian** — Accédez, naviguez et recherchez dans toutes vos notes Obsidian depuis n'importe quel appareil via une interface web moderne et responsive.",
"main": "patch.js",
"directories": {
+69 -1
View File
@@ -1,4 +1,9 @@
# Plan: Incremental InvertedIndex for 40k+ files
# Incremental InvertedIndex for 40k+ files — livré
> **Statut : LIVRÉ (BUG-033, v2.3.0).** Ce fichier a servi de plan
> d'exécution ; il est conservé comme **trace de conception**. Le code réel
> a divergé sur plusieurs points (voir [État réel](#état-réel-corrigé-au-2026-09-27))
> — ne pas lire les extraits de code ci-dessous comme du code actuel.
## Problem Summary
@@ -17,6 +22,11 @@ Then hook these into `_add_file_to_structures` and `_remove_file_from_structures
Remove the `is_stale()` / `rebuild()` / cooldown mechanism entirely. The inverted index is always current.
> ⚠️ **Nuance retenue à l'implémentation** : un unique `rebuild()` reste nécessaire au
> démarrage (le hook est inerte tant que l'index n'est pas prêt) et au reindex manuel
> d'une vault. Ce qui disparaît, c'est la *staleness* : plus de compteur de génération,
> plus de cooldown, plus de rebuild paresseux.
## Dependency Architecture
**Current import chain:**
@@ -346,3 +356,61 @@ This hack was only needed to reduce the number of inverted index rebuilds. With
4. **Sorted tokens performance:** `bisect.insort` and `list.pop(idx)` are O(V) worst case for large V. For 40k files, the vocabulary size V is typically 50k-200k tokens. O(V) for a single insertion is ~0.001ms, acceptable. The rebuild() call at startup handles the initial bulk.
5. **tag_norm_map / tag_prefix_index growth:** These grow monotonically (never shrink on incremental remove). With 40k files and thousands of tags, this is a few thousand entries — negligible. A manual "Réindexer" button triggers a full `rebuild()` to clean up.
---
## État réel (corrigé le 2026-09-27)
Le plan ci-dessus a servi de brouillon : **le code livré en est différent sur
quatre points**. Relevé fait sur `backend/search.py`, `backend/indexer.py` et
`backend/main.py`, pas de mémoire.
| Point prévu | État réel |
|---|---|
| Étapes 1-2 : hook + `add_document()` / `remove_document()` | ✅ livré tel que prévu |
| Étapes 4-5 : `rebuild()` initial via `init_inverted_index()` appelé depuis la lifespan | ✅ livré (`backend/main.py:297`, dans l'exécuteur de recherche) |
| Étape 6 : retirer `is_stale()` + `_last_rebuild` / `_rebuild_cooldown` / `_source_generation` | ✅ **déjà fait** avant cette relecture — aucun de ces symboles ne subsiste |
| Étape 7 : retirer le hack de coalescence `_index_generation` dans `_on_vault_change` | ✅ **déjà fait** — `_on_vault_change` n'existe plus |
| `get_inverted_index()` simplifié | ✅ mais **sans le fallback `_needs_rebuild`** prévu par le plan |
### Écarts assumés
1. **`is_stale()` a survécu sous un autre nom.** L'étape 6 est faite, mais la
méthode a été conservée car elle répond à une autre question : *l'index
initial est-il construit ?* Elle ne mesure plus aucune staleness (le compteur
de génération et le cooldown ont disparu) et le nom était trompeur. Elle est
donc renommée `is_ready()` — cohérent avec le `is_ready()` déjà exposé par
`SemanticIndex` (`backend/semantic_search.py`). L'alias `is_stale()` de
`SemanticIndex`, sans aucun appelant, est supprimé.
Impact : le champ de `/api/diagnostics` passe de `is_stale` à `is_ready`
(libellé « Index prêt » côté `frontend/js/config.js`).
2. **Pas de repli `_needs_rebuild`.** Le plan prévoyait qu'un échec
d'incrémentation marque l'index pour reconstruction. L'implémentation
retenue se contente de logger un warning et de continuer à servir l'index.
Choix assumé : un échec d'incrémentation est exceptionnel, et reconstruire
silencieusement serait plus coûteux que l'état dégradé. **Si ce compromis
devient critiquique, c'est le point à rouvrir.**
3. **`_ready` remplace `doc_count == 0`.** Le plan prévoyait de sauter le hook
« index vide » ; le drapeau explicite `_ready` est plus sûr (un vault
réellement vide serait sinon pris pour un index non construit).
4. **`rebuild()` reste nécessaire** au démarrage et au reindex manuel d'une
vault. Le plan parlait de le supprimer de `get_inverted_index()`, ce qui est
fait, mais la méthode elle-même est conservée.
### Bug trouvé pendant cette relecture (corrigé ici)
`remove_vault_from_index()` (`backend/indexer.py`) ne notifiait pas le hook.
Conséquence mesurée : après suppression d'une vault, ses 8 documents test
restaient dans l'index inversé — `postings`, `doc_info`, `doc_vault`,
`vault_docs` — et continuaient de correspondre aux recherches pour une vault
inexistante. Seul un reindex manuel les effaçait.
Le correctif déclenche `_on_index_change('remove', …)` pour chaque fichier de
la vault, et `_remove_doc_internals()` supprime désormais la clé `vault_docs`
quand son set devient vide (c'est un `defaultdict` : une simple lecture la
ré créait). Test de non-régression :
`TestVaultRemovalPurgesInvertedIndex` (contre-preuve : échoue sans le patch).
+9
View File
@@ -9,6 +9,15 @@ export default defineConfig({
reporter: process.env.CI ? 'github' : 'list',
timeout: 60000,
expect: { timeout: 10000 },
// BUG-080 : la suite (~120 tests, workers: 1, ~10-15 s/test sur un poste
// chargé) ne doit jamais pendre toute la nuit. Au-delà du timeout global,
// Playwright abandonne avec un échec explicite au lieu de bloquer.
// Surchargable : E2E_GLOBAL_TIMEOUT_MS.
globalTimeout: Number(
process.env.E2E_GLOBAL_TIMEOUT_MS ??
(process.env.CI ? 30 * 60 * 1000 : 25 * 60 * 1000),
),
reportSlowTests: process.env.CI ? null : { max: 5, threshold: 30000 },
use: {
baseURL: process.env.BASE_URL || 'http://localhost:2029',
+161
View File
@@ -0,0 +1,161 @@
<#
.SYNOPSIS
ObsiGate — cycle de vie du serveur E2E local, avec progression visible.
.DESCRIPTION
Remplace le one-liner opaque de démarrage : chaque étape affiche sa
progression (port, PID, attente du health check seconde par seconde,
version servie). Memes conditions que le job CI `e2e` et que
`scripts/run-e2e-local.ps1` : uvicorn natif, auth désactivée, fixtures
TestVault/TestDir, port 2029.
Le PID est persisté dans `data/e2e-server.pid` pour un arrêt propre
(`stop`) — plus de serveurs orphelins qui squattent le port.
.EXAMPLE
./scripts/e2e-server.ps1 start # démarre + attend READY (défaut)
./scripts/e2e-server.ps1 status # port, PID, version servie
./scripts/e2e-server.ps1 logs # queues des logs serveur
./scripts/e2e-server.ps1 stop # arrête le serveur + libère le port
#>
[CmdletBinding()]
param(
[Parameter(Position = 0)]
[ValidateSet("start", "stop", "status", "logs")]
[string]$Command = "start",
[string]$Port = $(if ($env:E2E_PORT) { $env:E2E_PORT } else { "2029" })
)
$ErrorActionPreference = "Stop"
$Root = Split-Path -Parent $PSScriptRoot
Set-Location -LiteralPath $Root
$BaseUrl = "http://127.0.0.1:$Port"
$Python = ".\.venv\Scripts\python.exe"
$PidFile = "data/e2e-server.pid"
$OutLog = "data/e2e-server.log"
$ErrLog = "data/e2e-server.err.log"
function Get-PortOwner {
$conn = Get-NetTCPConnection -LocalPort $Port -State Listen -ErrorAction SilentlyContinue |
Select-Object -First 1
if (-not $conn) { return $null }
$proc = Get-Process -Id $conn.OwningProcess -ErrorAction SilentlyContinue
return @{ Pid = $conn.OwningProcess; Name = $(if ($proc) { $proc.ProcessName } else { "?" }) }
}
function Stop-Server {
param([string]$Why = "")
$killed = @()
if (Test-Path -LiteralPath $PidFile) {
$srvPid = (Get-Content -LiteralPath $PidFile -TotalCount 1).Trim()
if ($srvPid -match '^\d+$') {
# BUG-080 : le PID enregistré peut avoir ré-exécuté uvicorn dans un
# processus enfant (constaté : parent .venv + enfant uv-python sur
# le port) — tuer l'arbre complet, pas seulement la racine.
Get-CimInstance Win32_Process -Filter "ParentProcessId=$srvPid" -ErrorAction SilentlyContinue |
ForEach-Object { Stop-Process -Id $_.ProcessId -Force -ErrorAction SilentlyContinue; $killed += $_.ProcessId }
Stop-Process -Id $srvPid -Force -ErrorAction SilentlyContinue
$killed += $srvPid
}
Remove-Item -LiteralPath $PidFile -Force -ErrorAction SilentlyContinue
}
$owner = Get-PortOwner
if ($owner) {
Stop-Process -Id $owner.Pid -Force -ErrorAction SilentlyContinue
$killed += $owner.Pid
}
if ($killed.Count) { Write-Host "[OK] Serveur arrêté (PID $($killed -join ', ')). $Why" }
else { Write-Host "[OK] Aucun serveur en cours (port $Port libre)." }
}
switch ($Command) {
"stop" {
Write-Host "[1/1] Arrêt du serveur E2E (port $Port)..."
Stop-Server
}
"status" {
$owner = Get-PortOwner
if (-not $owner) { Write-Host "[INFO] Port $Port libre, aucun serveur."; break }
Write-Host "[INFO] Port $Port occupé par PID $($owner.Pid) ($($owner.Name))."
try {
$health = Invoke-WebRequest -Uri "$BaseUrl/api/health" -TimeoutSec 5 -UseBasicParsing |
Select-Object -ExpandProperty Content | ConvertFrom-Json
Write-Host "[OK] Health 200 — version $($health.version), $($health.total_files) fichiers indexés."
} catch {
Write-Host "[WARN] Processus présent mais health injoignable : $($_.Exception.Message)"
}
}
"logs" {
Write-Host "===== $OutLog (stdout) ====="
Get-Content -LiteralPath $OutLog -Tail 15 -ErrorAction SilentlyContinue
Write-Host "===== $ErrLog (stderr) ====="
Get-Content -LiteralPath $ErrLog -Tail 25 -ErrorAction SilentlyContinue
}
"start" {
Write-Host "[1/4] Port $Port..."
$owner = Get-PortOwner
if ($owner) {
Write-Host "[ERR] Port $Port déjà occupé par PID $($owner.Pid) ($($owner.Name))."
Write-Host " Lancez d'abord : ./scripts/e2e-server.ps1 stop"
exit 1
}
Write-Host " libre."
Write-Host "[2/4] Interpréteur $Python..."
if (-not (Test-Path -LiteralPath $Python)) {
Write-Host "[ERR] $Python introuvable. Créez le venv (voir AGENTS.md)."
exit 1
}
Write-Host " présent."
New-Item -ItemType Directory -Force -Path "data" | Out-Null
Write-Host "[3/4] Démarrage uvicorn (auth désactivée, TestVault/TestDir)..."
$env:OBSIGATE_AUTH_ENABLED = "false"
$env:VAULT_1_NAME = "TestVault"
$env:VAULT_1_PATH = (Resolve-Path -LiteralPath "test_vault").Path
$env:DIR_1_NAME = "TestDir"
$env:DIR_1_PATH = (Resolve-Path -LiteralPath "test_dir").Path
$server = Start-Process -FilePath $Python `
-ArgumentList "-m", "uvicorn", "backend.main:app", "--host", "127.0.0.1", "--port", $Port `
-RedirectStandardOutput $OutLog -RedirectStandardError $ErrLog `
-PassThru -WindowStyle Hidden
$server.Id | Set-Content -LiteralPath $PidFile
Write-Host " PID $($server.Id) (logs : $OutLog / $ErrLog)."
Write-Host "[4/4] Attente du health check (30 s max)..."
$ready = $false
for ($i = 1; $i -le 30; $i++) {
try {
Invoke-WebRequest -Uri "$BaseUrl/api/health" -TimeoutSec 2 -UseBasicParsing | Out-Null
$ready = $true
break
} catch {
if ($server.HasExited) {
Write-Host "[ERR] Le serveur a quitté (code $($server.ExitCode)). Fin du log :"
Get-Content -LiteralPath $ErrLog -Tail 15 -ErrorAction SilentlyContinue
exit 1
}
if ($i % 5 -eq 0) { Write-Host " ... $i/30 s (indexation en cours, voir $ErrLog)" }
Start-Sleep -Seconds 1
}
}
if (-not $ready) {
Write-Host "[ERR] Injoignable après 30 s. Fin du log :"
Get-Content -LiteralPath $ErrLog -Tail 15 -ErrorAction SilentlyContinue
exit 1
}
$health = Invoke-WebRequest -Uri "$BaseUrl/api/health" -TimeoutSec 5 -UseBasicParsing |
Select-Object -ExpandProperty Content | ConvertFrom-Json
# BUG-080 : le PID `Start-Process` peut ne pas être celui qui écoute
# (ré-exécution enfant constatée) — persister le vrai propriétaire du
# port pour un `stop` fiable, sans serveurs orphelins.
$owner = Get-PortOwner
if ($owner) { $owner.Pid | Set-Content -LiteralPath $PidFile }
Write-Host "[OK] READY — ObsiGate v$($health.version) sur $BaseUrl ($($health.total_files) fichiers)."
}
}
+59 -7
View File
@@ -10,7 +10,7 @@
conditions que le job CI `e2e`), lance la suite Playwright puis nettoie.
.PARAMETER PlaywrightArgs
Arguments transmis à `npx playwright test`, ex. `-g "image viewer"`,
Arguments transmis à `playwright test` (via `node`), ex. `-g "image viewer"`,
`--headed`.
.EXAMPLE
@@ -34,6 +34,40 @@ $BaseUrl = "http://127.0.0.1:$Port"
$ServerLog = "data/e2e-server.log"
$ServerErrLog = "data/e2e-server.err.log"
# BUG-080 : garde-fous anti-blocage (un run E2E ne doit jamais pendre toute la nuit).
# E2E_TIMEOUT_SEC dépasse volontairement le globalTimeout Playwright (25 min en
# local) pour que ce soit Playwright qui abandonne proprement (avec rapport) en premier.
$TestTimeoutSec = if ($env:E2E_TIMEOUT_SEC) { [int]$env:E2E_TIMEOUT_SEC } else { 1800 }
$BrowserTimeoutSec = if ($env:E2E_BROWSER_INSTALL_TIMEOUT_SEC) { [int]$env:E2E_BROWSER_INSTALL_TIMEOUT_SEC } else { 600 }
function Invoke-NativeWithTimeout([string]$Label, [int]$TimeoutSec, [string]$Exe, [string[]]$Arguments) {
# Lance un processus natif en gardant la sortie console en direct, et le
# tue après $TimeoutSec s'il n'a pas terminé (exit 124, comme `timeout`).
# NOTE : le paramètre NE DOIT PAS s'appeler `$Args` (variable automatique
# PowerShell qui l'écraserait → `node` lancé sans arguments, exit 0
# silencieux immédiat en lisant un stdin vide).
$stamp = Get-Date -Format "HH:mm:ss"
Write-Host "[$stamp] $Label (timeout ${TimeoutSec}s)..."
$proc = Start-Process -FilePath $Exe -ArgumentList $Arguments -NoNewWindow -PassThru
$proc | Wait-Process -Timeout $TimeoutSec -ErrorAction SilentlyContinue
if (-not $proc.HasExited) {
Write-Host "[ERR] $Label : timeout après ${TimeoutSec}s, arrêt du processus (PID $($proc.Id))."
Stop-Process -Id $proc.Id -Force -ErrorAction SilentlyContinue
return 124
}
return $proc.ExitCode
}
function Test-ChromiumInstalled {
$base = Join-Path $env:USERPROFILE "AppData\Local\ms-playwright"
if (-not (Test-Path -LiteralPath $base)) { return $false }
$hit = Get-ChildItem -LiteralPath $base -Directory -ErrorAction SilentlyContinue |
Where-Object { $_.Name -like "chromium-*" } |
Where-Object { Test-Path -LiteralPath (Join-Path $_.FullName "chrome-win\chrome.exe") } |
Select-Object -First 1
return ($null -ne $hit)
}
function Assert-Command([string]$Name, [string]$Hint) {
if (-not (Get-Command $Name -ErrorAction SilentlyContinue)) {
throw "[ERR] $Name introuvable. $Hint"
@@ -41,7 +75,7 @@ function Assert-Command([string]$Name, [string]$Hint) {
}
Assert-Command "uv" "Installez-le : https://docs.astral.sh/uv/"
Assert-Command "npx" "Installez Node.js (>= 20)."
Assert-Command "node" "Installez Node.js (>= 20)."
# ----- Venv Python 3.11 (créé une seule fois) -----
$Python = ".venv-e2e/Scripts/python.exe"
@@ -101,14 +135,32 @@ try {
}
Write-Host "[OK] Serveur prêt."
# ----- Browsers Playwright (no-op s'ils sont déjà installés) -----
npx playwright install chromium
# ----- Playwright via node direct (pas npx) -----
# BUG-080 : `Start-Process` ne peut pas lancer `npx` (ni le `.ps1` ni le
# `.cmd` ne sont des applications Win32 directes) → on appelle la CLI
# locale via `node.exe`, sans prompt interactif possible. Skip de
# l'install si un chromium est déjà présent (sauf E2E_INSTALL_BROWSERS=1),
# timeouts dédiés sur chaque étape.
$PlaywrightCli = Join-Path $Root "node_modules/@playwright/test/cli.js"
if (-not (Test-Path -LiteralPath $PlaywrightCli)) {
throw "[ERR] $PlaywrightCli introuvable. Lancez d'abord : npm ci"
}
if (($env:E2E_INSTALL_BROWSERS -eq "1") -or (-not (Test-ChromiumInstalled))) {
$code = Invoke-NativeWithTimeout "playwright install chromium" $BrowserTimeoutSec "node" @($PlaywrightCli, "install", "chromium")
if ($code -ne 0) { exit $code }
} else {
Write-Host "[INFO] Chromium Playwright déjà installé, étape sautée (E2E_INSTALL_BROWSERS=1 pour forcer)."
}
# ----- Exécution de la suite (projet CI : chromium-desktop) -----
Write-Host "[INFO] BASE_URL=$BaseUrl npx playwright test --project=chromium-desktop $($PlaywrightArgs -join ' ')"
Write-Host "[INFO] BASE_URL=$BaseUrl node $PlaywrightCli test --project=chromium-desktop $($PlaywrightArgs -join ' ')"
$env:BASE_URL = $BaseUrl
& npx playwright test --project=chromium-desktop @PlaywrightArgs
$exitCode = $LASTEXITCODE
$testArgs = @($PlaywrightCli, "test", "--project=chromium-desktop")
if ($PlaywrightArgs) { $testArgs += @($PlaywrightArgs) }
$exitCode = Invoke-NativeWithTimeout "playwright test" $TestTimeoutSec "node" $testArgs
} catch {
Write-Host "[ERR] $($_.Exception.Message)"
$exitCode = 1
} finally {
Write-Host "[INFO] Arrêt du serveur (PID $($server.Id))..."
if (-not $server.HasExited) { Stop-Process -Id $server.Id -Force -ErrorAction SilentlyContinue }
+23 -2
View File
@@ -23,6 +23,21 @@ cd "$(dirname "$0")/.."
PORT="${E2E_PORT:-2029}"
BASE_URL="http://127.0.0.1:$PORT"
SERVER_LOG="data/e2e-server.log"
# BUG-080 : garde-fous anti-blocage (un run E2E ne doit jamais pendre toute la nuit).
E2E_TIMEOUT_SEC="${E2E_TIMEOUT_SEC:-900}"
E2E_BROWSER_INSTALL_TIMEOUT_SEC="${E2E_BROWSER_INSTALL_TIMEOUT_SEC:-600}"
# Exécute "$@" avec un timeout dur (exit 124 comme `timeout`), sans timeout si
# la commande `timeout` est absente (ex. macOS sans coreutils).
run_with_timeout() {
local limit="$1"; shift
if command -v timeout &>/dev/null; then
timeout "$limit" "$@"
else
echo "[WARN] commande 'timeout' absente : $1 sans limite de ${limit}s" >&2
"$@"
fi
}
# ----- Prérequis -----
if ! command -v uv &>/dev/null; then
@@ -102,8 +117,14 @@ curl -sf "$BASE_URL/api/health" >/dev/null || {
}
# ----- Browsers Playwright (no-op s'ils sont déjà installés) -----
npx playwright install chromium
# BUG-080 : `--yes` (jamais de prompt interactif npx qui pend), skip si un
# chromium est déjà présent (sauf E2E_INSTALL_BROWSERS=1), timeout dédié.
if [[ "${E2E_INSTALL_BROWSERS:-0}" == "1" ]] || ! ls -d ~/.cache/ms-playwright/chromium-*/chrome-linux/chrome &>/dev/null; then
run_with_timeout "$E2E_BROWSER_INSTALL_TIMEOUT_SEC" npx --yes playwright install chromium
else
echo "[INFO] Chromium Playwright déjà installé, étape sautée (E2E_INSTALL_BROWSERS=1 pour forcer)."
fi
# ----- Exécution de la suite (projet CI : chromium-desktop) -----
echo "[INFO] BASE_URL=$BASE_URL npx playwright test --project=chromium-desktop $*"
BASE_URL="$BASE_URL" npx playwright test --project=chromium-desktop "$@"
BASE_URL="$BASE_URL" run_with_timeout "$E2E_TIMEOUT_SEC" npx --yes playwright test --project=chromium-desktop "$@"
+70
View File
@@ -0,0 +1,70 @@
# ObsiGate — règles Semgrep locales (#87 T7).
#
# Volontairement LOCALES (aucun `--config auto`/registre) : le runner CI a un
# accès réseau fragile, et ces règles n'ont besoin d'aucun téléchargement.
# Exécution : `semgrep --config semgrep-rules/ backend/` (job CI `lint`,
# bloquant). Chaque règle est un garde-fou : aucun code existant ne doit
# la déclencher (vérifié à l'ajout) ; toute violation future échoue le CI.
rules:
- id: obsigate-no-eval-exec
message: "Interdit : eval()/exec() sur du contenu dynamique (injection de code). Restructurer sans exécution de code."
severity: ERROR
languages: [python]
pattern-either:
- pattern: eval(...)
- pattern: exec(...)
- id: obsigate-no-shell-true
message: "Interdit : subprocess avec shell=True (injection shell). Passer argv en liste, shell=False."
severity: ERROR
languages: [python]
pattern-either:
- pattern: subprocess.run(..., shell=True, ...)
- pattern: subprocess.Popen(..., shell=True, ...)
- pattern: subprocess.call(..., shell=True, ...)
- pattern: subprocess.check_output(..., shell=True, ...)
- pattern: subprocess.check_call(..., shell=True, ...)
- id: obsigate-no-os-system
message: "Interdit : os.system() (shell implicite). Utiliser subprocess avec argv en liste."
severity: ERROR
languages: [python]
pattern: os.system(...)
- id: obsigate-no-pickle-load
message: "Interdit : pickle.load/loads sur des données non fiables (exécution arbitraire). Utiliser JSON."
severity: ERROR
languages: [python]
pattern-either:
- pattern: pickle.load(...)
- pattern: pickle.loads(...)
- id: obsigate-no-yaml-unsafe-load
message: "Interdit : yaml.load() sans Loader (exécution arbitraire). Utiliser yaml.safe_load()."
severity: ERROR
languages: [python]
patterns:
- pattern: yaml.load(...)
- pattern-not: yaml.load(..., Loader=...)
- id: obsigate-no-unverified-tls
message: "Interdit : verify=False (MITM). Ne jamais désactiver la vérification TLS."
severity: ERROR
languages: [python]
pattern-either:
- pattern: requests.$METHOD(..., verify=False, ...)
- pattern: httpx.$METHOD(..., verify=False, ...)
- pattern: httpx.Client(..., verify=False, ...)
- pattern: httpx.AsyncClient(..., verify=False, ...)
- id: obsigate-no-markupsafe-markup
message: "Interdit : markupsafe.Markup() (contourne l'échappement XSS, BUG-021/022). Le sanitizer serveur est la seule voie."
severity: ERROR
languages: [python]
pattern: Markup(...)
- id: obsigate-no-tempfile-mktemp
message: "Interdit : tempfile.mktemp() (race symlink, CWE-377). Utiliser NamedTemporaryFile/mkdtemp."
severity: ERROR
languages: [python]
pattern: tempfile.mktemp(...)
Binary file not shown.
Binary file not shown.
+258
View File
@@ -0,0 +1,258 @@
/**
* E2E tests — Excel viewer, write guards (ROADMAP #153 P0).
*
* Fixture : `test_vault/sample-xlsx-lossy.xlsx` — a plain 2x2 workbook whose
* sheet XML carries a cached formula result (`<f>B1*2</f><v>200</v>`) and whose
* package contains `xl/slicers/slicer1.xml`. Both are dropped by an openpyxl
* round-trip, so the read response must report
* `xlsx_lossy_features: ["cached_values", "slicers"]` (BUG-085 A1).
*
* Covered :
* - the warning banner lists both features ;
* - saving a cell on that workbook asks for confirmation (native dialog) and
* then succeeds (the client retries with `force: true`) ;
* - the f(x) toggle is off by default, so "=B1*3" is stored as text ;
* - the value Excel last computed is shown under the formula (#153 A12).
*
* Second describe block — `test_vault/sample-xlsx-large.xlsx` (520 rows) :
* - a sheet over the render caps SAYS it instead of looking complete (#153 A8) ;
* - the column headers stay pinned while the sheet scrolls (#153 A8) ;
* - `GET …/xlsx/sheet?offset=500` serves the rows the caps used to hide,
* with the real A1 coordinates (#153 A9).
*
* The fixture is restored byte-for-byte in `afterAll` so a local run never
* dirties the working copy.
*
* Run (local) : BASE_URL=http://localhost:2029 npx playwright test tests/e2e/xlsx-viewer.spec.js
*/
import { test, expect } from '@playwright/test';
import { readFileSync, writeFileSync, existsSync } from 'node:fs';
import path from 'node:path';
const BASE = process.env.BASE_URL || 'http://localhost:2029';
const VAULT = 'TestVault';
const FIXTURE = 'sample-xlsx-lossy.xlsx';
// #153 A8 — 520 rows x 3 columns: the sheet exceeds the 500-row render cap, so
// the viewer must SAY so. Generated once with openpyxl (header + 519 lines) and
// committed next to the other fixture; nothing in the suite writes to it.
const LARGE = 'sample-xlsx-large.xlsx';
// Playwright runs from the repository root (run-e2e-local.* / CI both do).
const FIXTURE_PATH = path.resolve(process.cwd(), 'test_vault', FIXTURE);
let originalBytes = null;
async function login(page) {
await page.goto(BASE);
const loginForm = page.locator('#login-screen');
await expect(loginForm).toBeVisible({ timeout: 5000 }).catch(() => {});
if (await loginForm.isVisible()) {
await page.fill('#login-username', process.env.OBSIGATE_USER || 'admin');
await page.fill('#login-password', process.env.OBSIGATE_PASS || 'test123');
await page.click('#login-btn');
}
await page.waitForFunction(() => window.__OBSIGATE_BOOTED === true, { timeout: 20000 });
}
async function openFixture(page) {
return openXlsx(page, FIXTURE);
}
async function openXlsx(page, file) {
const treeItem = page.locator(`.tree-item[data-vault="${VAULT}"][data-path="${file}"]`);
if (!(await treeItem.count())) {
await page.locator(`.tree-item.vault-item[data-vault="${VAULT}"]`).first().click();
await treeItem.waitFor({ state: 'attached', timeout: 8000 });
}
await treeItem.dblclick({ timeout: 5000 });
await expect(page.locator('#content-area .xlsx-table')).toBeVisible({ timeout: 15000 });
}
test.describe('Excel viewer — garde-fous d\'écriture et valeurs calculées (#153)', () => {
test.beforeAll(() => {
if (existsSync(FIXTURE_PATH)) originalBytes = readFileSync(FIXTURE_PATH);
});
test.afterAll(() => {
if (originalBytes) writeFileSync(FIXTURE_PATH, originalBytes);
});
// Read-only assertions come FIRST, before the mutating tests: saving through
// the viewer rewrites the workbook and an openpyxl round-trip drops the cached
// formula results (BUG-085), so the shadow line only exists on a pristine
// fixture.
test('affiche la valeur calculée en cache sous la formule (#153 A12)', async ({ page }) => {
await login(page);
await openFixture(page);
// B2 is "=B1*2" and the package keeps its last result (<v>200</v>).
const formulaCell = page.locator('#content-area td[data-cell="B2"]');
await expect(formulaCell).toContainText('=B1*2');
const cached = formulaCell.locator('.xlsx-cached');
await expect(cached).toHaveCount(1);
await expect(cached).toHaveText('200');
// The tooltip is translated client-side, never hardcoded by the backend.
await expect(cached).toHaveAttribute('title', /Excel/);
// A plain value cell must not be duplicated with a shadow line.
await expect(page.locator('#content-area td[data-cell="B1"] .xlsx-cached')).toHaveCount(0);
});
test('affiche la bannière listant les éléments non préservés', async ({ page }) => {
await login(page);
await openFixture(page);
const banner = page.locator('#content-area .xlsx-warning');
await expect(banner).toBeVisible();
// 2 features : valeurs calculées + segments (jamais de couleur codée en dur,
// les libellés viennent bien des locales).
await expect(banner.locator('.xlsx-warning-tag')).toHaveCount(2);
await expect(banner).toContainText('segments');
await expect(banner).toContainText('valeurs calculées');
});
test('demande confirmation puis enregistre la cellule', async ({ page }) => {
await login(page);
await openFixture(page);
let dialogMessage = null;
page.on('dialog', async (dialog) => {
dialogMessage = dialog.message();
await dialog.accept();
});
const cell = page.locator('#content-area td[data-cell="A2"]');
await cell.click();
await cell.fill('Total confirmé');
await cell.press('Enter');
const save = page.locator('#xlsx-save-btn');
await expect(save).toBeEnabled();
await save.click();
await expect.poll(() => dialogMessage, { timeout: 10000 }).toContain('segments');
await expect(page.locator('.toast-success')).toBeVisible({ timeout: 10000 });
// La cellule reste modifiée côté UI (plus de marque « sale »).
await expect(page.locator('#content-area td.xlsx-dirty')).toHaveCount(0);
});
test('le toggle f(x) est désactivé par défaut (formule stockée en texte)', async ({ page }) => {
await login(page);
await openFixture(page);
const toggle = page.locator('#xlsx-formula-btn');
await expect(toggle).toHaveAttribute('aria-pressed', 'false');
// 409 → confirmation, puis reprise avec force (le toggle reste désactivé).
page.on('dialog', (dialog) => dialog.accept());
const cell = page.locator('#content-area td[data-cell="B2"]');
await cell.click();
await cell.fill('=B1*3');
await cell.press('Enter');
await page.locator('#xlsx-save-btn').click();
await expect(page.locator('.toast-success')).toBeVisible({ timeout: 10000 });
});
});
// ── A8 — troncature annoncée + en-têtes figés ───────────────────────────────
test.describe('Excel viewer — troncature et navigation (#153 A8/A9)', () => {
test('annonce la feuille tronquée au lieu de la couper en silence', async ({ page }) => {
await login(page);
await openXlsx(page, LARGE);
const note = page.locator('#content-area .xlsx-truncated');
await expect(note).toBeVisible();
// Libellé traduit (jamais de texte UI backend, jamais de couleur en dur).
await expect(note).toContainText('Feuille tronquée');
await expect(note).toContainText('500 lignes affichées sur 520');
// La dernière ligne rendue est la 500e ; les suivantes ne sont pas là.
await expect(page.locator('#content-area td[data-cell="A500"]')).toHaveCount(1);
await expect(page.locator('#content-area td[data-cell="A501"]')).toHaveCount(0);
});
test('garde les en-têtes de colonnes visibles au défilement', async ({ page }) => {
await login(page);
await openXlsx(page, LARGE);
const header = page.locator('#content-area .xlsx-table thead th').nth(1);
const before = await header.boundingBox();
await page.locator('#content-area .csv-table-wrapper').evaluate((el) => { el.scrollTop = 800; });
await expect.poll(async () => (await header.boundingBox()).y, { timeout: 5000 })
.toBeLessThanOrEqual(before.y + 1);
// Les numéros de ligne ne se superposent pas en haut à gauche (le `top: auto`
// de A8) et la première ligne de données reste lisible sous l'en-tête.
const first = await page.locator('#content-area th.xlsx-rownum').first().boundingBox();
const second = await page.locator('#content-area th.xlsx-rownum').nth(1).boundingBox();
expect(second.y - first.y).toBeGreaterThan(4);
});
test('l\'endpoint de fenêtre sert les lignes au-delà du plafond (#153 A9)', async ({ page }) => {
await login(page);
await openXlsx(page, LARGE);
const res = await page.request.get(
`${BASE}/api/file/${VAULT}/xlsx/sheet?path=${encodeURIComponent(LARGE)}&sheet=Journal&offset=500&limit=50`
);
expect(res.status()).toBe(200);
const win = await res.json();
expect(win.total_rows).toBe(520);
expect(win.offset).toBe(500);
expect(win.has_more).toBe(false);
// Les coordonnées A1 sont celles de la feuille, pas celles de la fenêtre :
// la ligne 520 est servie comme A520, pas comme A20.
expect(win.html).toContain('data-cell="A520"');
expect(win.html).toContain('Operation 519');
expect(win.html).not.toContain('data-cell="A1"');
});
test('le bouton « charger la suite » ajoute les lignes cachées (#153 A9bis)', async ({ page }) => {
await login(page);
await openXlsx(page, LARGE);
// La ligne 500 est la dernière rendue ; le pied de page l'annonce.
const foot = page.locator('#content-area .xlsx-load-more');
await expect(foot).toBeVisible();
await expect(foot).toContainText('Charger la suite');
await expect(page.locator('#content-area td[data-cell="A501"]')).toHaveCount(0);
// Un clic fetch la suite (offset 500, 20 lignes) et l'insère dans la table.
await foot.click();
await expect(page.locator('#content-area td[data-cell="A520"]')).toBeVisible({ timeout: 10000 });
// Une ligne nouvellement arrivée est éditable comme les autres.
const cell = page.locator('#content-area td[data-cell="A520"]');
await cell.click();
await expect(cell).toBeFocused();
// Tout est chargé → le pied de page est masqué.
await expect(foot).toBeHidden();
});
test('barre de formule et navigation clavier (#153 A7)', async ({ page }) => {
await login(page);
await openXlsx(page, LARGE);
// Un clic sur une cellule active la barre avec son nom et son contenu.
const cell = page.locator('#content-area td[data-cell="B2"]');
await cell.click();
await expect(page.locator('#xlsx-active-cell')).toHaveText('B2');
const bar = page.locator('#xlsx-formula-input');
await expect(bar).toHaveValue(/Operation 1/);
// Les flèches déplacent la cellule active.
await cell.press('ArrowDown');
await expect(page.locator('#xlsx-active-cell')).toHaveText('B3');
await page.locator('#content-area td[data-cell="B3"]').press('ArrowRight');
await expect(page.locator('#xlsx-active-cell')).toHaveText('C3');
// Éditer depuis la barre marque la cellule dirty, Échap annule.
await bar.fill('Operation 2 modifiee');
await expect(page.locator('#content-area td[data-cell="C3"]')).toHaveClass(/xlsx-dirty/);
await page.locator('#content-area td[data-cell="C3"]').press('Escape');
await expect(page.locator('#content-area td.xlsx-dirty')).toHaveCount(0);
});
});
+142
View File
@@ -0,0 +1,142 @@
/**
* E2E tests — XSS stocké : page publique de partage + lecteur markdown (#87 T4).
*
* Non-régression BUG-021 (sanitizer serveur du rendu markdown) et BUG-022
* (échappement `title`/frontmatter + neutralisation `</script>` sur `/s/{token}`).
* Chaque test arme un guetteur de dialogues `alert` : le moindre JS exécuté
* fait échouer le test, en plus des assertions DOM (contenu échappé, aucun
* attribut `on*` vivant).
*
* Run (local, instance de test port 2029, auth désactivée — cf. scripts/run-e2e-local.ps1) :
* BASE_URL=http://localhost:2029 npx playwright test tests/e2e/xss.spec.js --project=chromium-desktop
*/
import { test, expect } from '@playwright/test';
const BASE = process.env.BASE_URL || 'http://localhost:2029';
const VAULT = 'TestVault';
const XSS_FILE = 'e2e-xss-probe.md';
const XSS_TITLE = '<img src=x onerror="window.__xss_title=1">';
const XSS_BODY = [
'# Sonde XSS',
'',
'<img src=x onerror="window.__xss_body=1">',
'',
'<script>window.__xss_script=1</script>',
'',
'[xss](javascript:window.__xss_js=1)',
].join('\n');
async function api(request, method, path, data) {
const resp = await request.fetch(`${BASE}${path}`, {
method,
data,
headers: { 'Content-Type': 'application/json' },
});
if (!resp.ok()) {
throw new Error(`${method} ${path} → ${resp.status()} ${await resp.text()}`);
}
return resp.json();
}
async function precleanProbeFile(request) {
// Idempotence : un run précédent interrompu a pu laisser le fichier sonde.
await request.fetch(`${BASE}/api/file/${VAULT}?path=${encodeURIComponent(XSS_FILE)}`, {
method: 'DELETE',
}).catch(() => {});
}
async function armAlertTrap(page) {
const dialogs = [];
page.on('dialog', async (d) => {
dialogs.push(d.message());
await d.dismiss();
});
return dialogs;
}
async function openFile(page, vault, filePath) {
const treeItem = page.locator(`.tree-item[data-vault="${vault}"][data-path="${filePath}"]`);
if (!(await treeItem.count())) {
await page.locator(`.tree-item.vault-item[data-vault="${vault}"]`).first().click();
await treeItem.waitFor({ state: 'attached', timeout: 8000 });
}
await treeItem.dblclick({ timeout: 5000 });
}
test.describe('XSS — page publique de partage (/s/{token}, BUG-022)', () => {
test('un titre/frontmatter hostile est échappé et aucun script ne tourne', async ({ page, request }) => {
const dialogs = await armAlertTrap(page);
await precleanProbeFile(request);
await api(request, 'POST', `/api/file/${VAULT}`, {
path: XSS_FILE,
// Titre entre quotes simples YAML (les doubles quotes internes restent
// des caractères ordinaires et arrivent intactes au backend).
content: `---\ntitle: '${XSS_TITLE}'\n---\n\n${XSS_BODY}\n`,
});
const share = await api(request, 'POST', `/api/share/${VAULT}`, { path: XSS_FILE });
await page.goto(`${BASE}/s/${share.token}`);
await expect(page.locator('.share-banner')).toBeVisible({ timeout: 10000 });
// Le titre affiché est le texte brut (balise neutralisée), pas un <img> vivant.
await expect(page.locator('.toolbar-title')).toContainText('<img src=x', { timeout: 5000 });
expect(await page.locator('.toolbar-title img').count()).toBe(0);
expect(await page.locator('img[onerror]').count()).toBe(0);
// Les 2 <script> de la page sont son code statique : le JSON embarqué
// (`#raw-content`) doit être neutralisé (aucun `</script>` littéral).
const rawEmbedded = await page.evaluate(() => {
const el = document.getElementById('raw-content');
return { text: el ? el.textContent : null };
});
expect(rawEmbedded.text).not.toBeNull();
expect(rawEmbedded.text).not.toContain('</script');
expect(rawEmbedded.text).toContain('\\u003c');
// Aucun payload n'a tourné (titre, corps, bloc script, lien javascript:).
const flags = await page.evaluate(() => ({
title: window.__xss_title,
body: window.__xss_body,
script: window.__xss_script,
js: window.__xss_js,
}));
expect(flags).toEqual({ title: undefined, body: undefined, script: undefined, js: undefined });
expect(dialogs).toEqual([]);
await api(request, 'DELETE', `/api/share/${share.id}`);
await api(request, 'DELETE', `/api/file/${VAULT}?path=${encodeURIComponent(XSS_FILE)}`);
});
});
test.describe('XSS — lecteur markdown (BUG-021)', () => {
test('le HTML injecté dans une note est neutralisé à l\'affichage', async ({ page, request }) => {
const dialogs = await armAlertTrap(page);
await precleanProbeFile(request);
await api(request, 'POST', `/api/file/${VAULT}`, { path: XSS_FILE, content: `${XSS_BODY}\n` });
await page.goto(BASE);
await page.waitForFunction(() => window.__OBSIGATE_BOOTED === true, { timeout: 20000 });
await openFile(page, VAULT, XSS_FILE);
const content = page.locator('#content-area');
await expect(content).toContainText('Sonde XSS', { timeout: 10000 });
// Le sanitizer serveur a retiré les vecteurs : pas d'onerror, pas de script,
// pas de lien javascript: exécutable dans la zone de lecture.
expect(await content.locator('img[onerror]').count()).toBe(0);
expect(await content.locator('script').count()).toBe(0);
expect(await content.locator('a[href^="javascript:"]').count()).toBe(0);
const flags = await page.evaluate(() => ({
body: window.__xss_body,
script: window.__xss_script,
js: window.__xss_js,
}));
expect(flags).toEqual({ body: undefined, script: undefined, js: undefined });
expect(dialogs).toEqual([]);
await api(request, 'DELETE', `/api/file/${VAULT}?path=${encodeURIComponent(XSS_FILE)}`);
});
});
+3 -2
View File
@@ -100,7 +100,7 @@ await test("module exports renderExcalidraw + helpers", () => {
assert.equal(typeof destroyExcalidrawEditor, "function");
});
await test("renderExcalidraw creates an iframe with sandbox + static src", () => {
await test("renderExcalidraw creates an iframe with sandbox + routed src", () => {
const container = document.getElementById("content-area");
const data = {
is_excalidraw: true,
@@ -111,7 +111,8 @@ await test("renderExcalidraw creates an iframe with sandbox + static src", () =>
renderExcalidraw(container, data, "TestVault", "diagram.excalidraw");
const iframe = container.querySelector("iframe");
assert.ok(iframe, "iframe should be created");
assert.ok(iframe.src.includes("/static/excalidraw-editor.html"), `src: ${iframe.src}`);
assert.ok(iframe.src.includes("/excalidraw-editor.html"), `src: ${iframe.src}`);
assert.ok(!iframe.src.includes("/static/excalidraw-editor.html"), `route avec nonce CSP: ${iframe.src}`);
assert.ok(iframe.sandbox.contains("allow-scripts"), "sandbox allow-scripts");
assert.ok(iframe.sandbox.contains("allow-same-origin"), "sandbox allow-same-origin");
assert.match(iframe.style.cssText, /100%/);
+679
View File
@@ -0,0 +1,679 @@
#!/usr/bin/env node
/**
* ObsiGate — JSDOM integration tests for the Excel viewer (ROADMAP #153 P0).
*
* Loads the real viewer.js module and drives renderXlsxViewer():
* - A1 : `xlsx_lossy_features` renders a warning banner; a save on such a
* workbook gets 409 `xlsx_lossy_content`, asks for confirmation and
* retries with `force: true` (or gives up when refused);
* - A4 : the f(x) toggle flips `allow_formula` in the save payload.
* - A8 : a sheet bigger than the render caps shows the truncation notice.
* - A9bis : the tail of a truncated sheet is fetched window by window from
* GET …/xlsx/sheet (scroll sentinel + click), and the appended rows are
* editable like the initial ones.
* - A7 : formula bar mirrors the active cell; arrows/Tab navigate; editing
* from the bar marks the cell dirty; Escape reverts.
* - A13 : header click sorts the rendered rows, the filter hides rows, the
* find highlights matches, CSV export downloads the visible sheet.
* - A14 : the structure menu sends one PUT …/xlsx/structure with the action,
* then re-renders from the server; destructive actions confirm first.
*
* Usage: node tests/frontend/xlsx-viewer.test.mjs
*/
import { strict as assert } from "node:assert";
import { JSDOM } from "jsdom";
import { fileURLToPath, pathToFileURL } from "node:url";
import { readFileSync } from "node:fs";
import path from "node:path";
const __filename = fileURLToPath(import.meta.url);
const __dirname = path.dirname(__filename);
const REPO_ROOT = path.resolve(__dirname, "..", "..");
// ── JSDOM bootstrap ─────────────────────────────────────────────────────────
const dom = new JSDOM(
`<!DOCTYPE html>
<html>
<body>
<div id="content-area"></div>
</body>
</html>`,
{ url: "http://localhost/", pretendToBeVisual: true }
);
const w = dom.window;
globalThis.window = w;
globalThis.document = w.document;
globalThis.DOMParser = w.DOMParser;
globalThis.HTMLElement = w.HTMLElement;
globalThis.Element = w.Element;
globalThis.Node = w.Node;
globalThis.Event = w.Event;
globalThis.CustomEvent = w.CustomEvent;
globalThis.MouseEvent = w.MouseEvent;
globalThis.localStorage = w.localStorage;
globalThis.sessionStorage = w.sessionStorage;
globalThis.requestAnimationFrame = (cb) => setTimeout(() => cb(Date.now()), 0);
Object.defineProperty(globalThis, "navigator", {
value: w.navigator,
configurable: true,
writable: true,
});
// ── fetch / confirm doubles ─────────────────────────────────────────────────
let calls = [];
let nextResponse = () => ({ ok: true, status: 200, body: { status: "ok" } });
let confirmAnswer = true;
let confirmCalls = 0;
// Every confirm() prompt is captured so the tests can assert on its text.
let confirmPrompts = [];
const FR = JSON.parse(
readFileSync(path.join(REPO_ROOT, "frontend", "locales", "fr.json"), "utf8")
);
globalThis.fetch = async (url, opts = {}) => {
// The i18n bootstrap fetches the locale files: serve the real FR one so the
// assertions run on the shipped strings, not on raw keys.
if (url.includes("/static/locales/")) {
return { ok: true, status: 200, json: async () => FR };
}
calls.push({ url, body: opts.body ? JSON.parse(opts.body) : null });
const res = nextResponse(url, opts);
return {
ok: res.ok,
status: res.status,
json: async () => res.body,
};
};
globalThis.confirm = (msg) => { confirmCalls++; confirmPrompts.push(msg); return confirmAnswer; };
w.confirm = globalThis.confirm;
// ── Helpers ─────────────────────────────────────────────────────────────────
let testCount = 0;
let passCount = 0;
async function test(name, fn) {
testCount++;
calls = [];
confirmCalls = 0;
confirmPrompts = [];
confirmAnswer = true;
apiQueue = [];
nextResponse = () => {
if (apiQueue.length) return apiQueue.shift();
return { ok: true, status: 200, body: { status: "ok" } };
};
try {
await fn();
console.log(` ✓ ${name}`);
passCount++;
} catch (e) {
console.log(` ✗ ${name}`);
console.log(` ${e.message}`);
if (e.stack) console.log(` ${e.stack.split("\n").slice(1, 3).join("\n ")}`);
}
}
const { renderXlsxViewer } = await import(
pathToFileURL(path.join(REPO_ROOT, "frontend", "js", "viewer.js")).href
);
// Load the FR catalog so t() resolves the real strings.
const { initI18n } = await import(
pathToFileURL(path.join(REPO_ROOT, "frontend", "js", "i18n.js")).href
);
await initI18n();
const sheetHtml = (value) =>
'<div class="csv-table-wrapper"><table class="csv-table xlsx-table">' +
'<thead><tr><th class="xlsx-corner"></th><th>A</th><th>B</th></tr></thead><tbody>' +
`<tr><th class="xlsx-rownum">1</th><td data-cell="A1">${value}</td><td data-cell="B1">B1</td></tr>` +
'<tr><th class="xlsx-rownum">2</th><td data-cell="A2">A2</td><td data-cell="B2">B2</td></tr>' +
"</tbody></table></div>";
// The JSDOM fetch double serves locale files; everything else is a recorded
// API call answered by `nextResponse`. `apiQueue` lets a test script the
// successive windows a lazy-loading sheet will request.
let apiQueue = [];
function mount({ lossy = [], sheet = {} } = {}) {
const area = document.getElementById("content-area");
area.innerHTML = "";
renderXlsxViewer(area, {
vault: "V",
path: "data.xlsx",
is_xlsx: true,
xlsx_sheets: [{ name: "Feuille1", html: sheetHtml("100"), ...sheet }],
xlsx_lossy_features: lossy,
});
return area;
}
/** Mount a sheet flagged truncated so wireLazyRows() arms the footnote. */
function mountTruncated({ total = 520, rows = 1 } = {}) {
return mount({
sheet: { rows, cols: 3, total_rows: total, total_cols: 3, max_rows: 500, max_cols: 40, truncated: total > 500 },
});
}
/** Mark a cell dirty the way a user edit would. */
function editCell(area, ref, text) {
const td = area.querySelector(`td[data-cell="${ref}"]`);
td.textContent = text;
td.dispatchEvent(new w.Event("input", { bubbles: true }));
return td;
}
const lossyError = {
ok: false,
status: 409,
body: {
detail: "Saving this workbook would drop features…",
code: "xlsx_lossy_content",
details: { features: ["slicers"] },
},
};
console.log("\n── xlsx viewer JSDOM integration tests (#153 P0) ──\n");
// ── A1 — warning banner ─────────────────────────────────────────────────────
await test("no banner when the workbook has nothing at risk", () => {
const area = mount();
assert.equal(area.querySelector(".xlsx-warning"), null);
});
await test("banner lists every lossy feature reported by the backend", () => {
const area = mount({ lossy: ["cached_values", "slicers"] });
const banner = area.querySelector(".xlsx-warning");
assert.ok(banner, "banner absent");
const tags = [...banner.querySelectorAll(".xlsx-warning-tag")].map((n) => n.textContent);
assert.equal(tags.length, 2);
assert.ok(tags.includes(FR["xlsx.feature_cached_values"]), tags.join("|"));
assert.ok(tags.includes(FR["xlsx.feature_slicers"]), tags.join("|"));
assert.ok(banner.textContent.includes(FR["xlsx.lossy_title"]));
assert.ok(banner.textContent.includes(FR["xlsx.lossy_hint"]));
});
await test("formula toggle is present and starts unpressed", () => {
const area = mount();
const btn = area.querySelector("#xlsx-formula-btn");
assert.ok(btn);
assert.equal(btn.getAttribute("aria-pressed"), "false");
assert.equal(btn.getAttribute("title"), FR["xlsx.formula_toggle_title"]);
});
// ── Save payload ────────────────────────────────────────────────────────────
await test("save sends one PUT per dirty sheet with the cell map", async () => {
const area = mount();
editCell(area, "A1", "250");
area.querySelector("#xlsx-save-btn").click();
await new Promise((r) => setTimeout(r, 0));
assert.equal(calls.length, 1);
assert.match(calls[0].url, /\/api\/file\/V\/xlsx\/save\?path=data\.xlsx/);
assert.deepEqual(calls[0].body.cells, { A1: "250" });
assert.equal(calls[0].body.sheet, "Feuille1");
assert.equal(calls[0].body.force, false);
assert.equal(calls[0].body.allow_formula, false);
});
await test("save button stays disabled when nothing is dirty", async () => {
const area = mount();
const btn = area.querySelector("#xlsx-save-btn");
assert.equal(btn.disabled, true);
btn.click();
await new Promise((r) => setTimeout(r, 0));
assert.equal(calls.length, 0);
});
// ── A4 — formula toggle ─────────────────────────────────────────────────────
await test("f(x) toggle flips allow_formula on the next save", async () => {
const area = mount();
area.querySelector("#xlsx-formula-btn").click();
assert.equal(area.querySelector("#xlsx-formula-btn").getAttribute("aria-pressed"), "true");
editCell(area, "A1", "=B1*2");
area.querySelector("#xlsx-save-btn").click();
await new Promise((r) => setTimeout(r, 0));
assert.equal(calls[0].body.allow_formula, true);
});
// ── A1 — 409 confirmation & force retry ─────────────────────────────────────
await test("409 xlsx_lossy_content asks once then retries with force", async () => {
const area = mount({ lossy: ["slicers"] });
editCell(area, "A1", "250");
nextResponse = () => (calls.length === 1 ? lossyError : { ok: true, status: 200, body: {} });
area.querySelector("#xlsx-save-btn").click();
await new Promise((r) => setTimeout(r, 10));
assert.equal(confirmCalls, 1);
assert.equal(calls.length, 2);
assert.equal(calls[0].body.force, false);
assert.equal(calls[1].body.force, true);
// The prompt names the features the backend reported.
assert.ok(confirmPrompts[0].includes(FR["xlsx.feature_slicers"]), confirmPrompts[0]);
// Save succeeded → cells are no longer dirty.
assert.equal(area.querySelectorAll("td.xlsx-dirty").length, 0);
});
await test("confirming once is enough for the following saves", async () => {
const area = mount({ lossy: ["cached_values"] });
editCell(area, "A1", "1");
nextResponse = () => (calls.length === 1 ? lossyError : { ok: true, status: 200, body: {} });
area.querySelector("#xlsx-save-btn").click();
await new Promise((r) => setTimeout(r, 10));
editCell(area, "A1", "2");
area.querySelector("#xlsx-save-btn").click();
await new Promise((r) => setTimeout(r, 10));
assert.equal(confirmCalls, 1, "the user is not asked twice");
assert.equal(calls.length, 3);
assert.equal(calls[2].body.force, true);
});
await test("refusing the confirmation writes nothing and keeps the cells dirty", async () => {
const area = mount({ lossy: ["slicers"] });
editCell(area, "A1", "250");
nextResponse = () => lossyError;
confirmAnswer = false;
area.querySelector("#xlsx-save-btn").click();
await new Promise((r) => setTimeout(r, 10));
assert.equal(confirmCalls, 1);
assert.equal(calls.length, 1, "no retry after a refusal");
assert.equal(area.querySelectorAll("td.xlsx-dirty").length, 1);
assert.equal(area.querySelector("#xlsx-save-btn").disabled, false);
});
await test("a non-409 failure is not retried", async () => {
const area = mount();
editCell(area, "A1", "250");
nextResponse = () => ({ ok: false, status: 500, body: { detail: "boom" } });
area.querySelector("#xlsx-save-btn").click();
await new Promise((r) => setTimeout(r, 10));
assert.equal(calls.length, 1);
assert.equal(confirmCalls, 0);
assert.equal(area.querySelectorAll("td.xlsx-dirty").length, 1);
});
// ── A8 — truncation notice ───────────────────────────────────────────────────
await test("no notice when the sheet fits within the render caps", () => {
const area = mount({
sheet: { rows: 500, cols: 40, total_rows: 500, total_cols: 40, max_rows: 500, max_cols: 40, truncated: false },
});
assert.equal(area.querySelector(".xlsx-truncated"), null);
});
await test("notice states the cap and the real size of a truncated sheet", () => {
const area = mount({
sheet: { rows: 500, cols: 12, total_rows: 1200, total_cols: 12, max_rows: 500, max_cols: 40, truncated: true },
});
const note = area.querySelector(".xlsx-truncated");
assert.ok(note, "notice absent");
const txt = note.textContent;
assert.ok(txt.includes(FR["xlsx.truncated_title"]), txt);
// {shown} is the CAP (500), not the post-trim row count: a sparse sheet
// renders 1 row but the view still reaches 500 of them.
const expected = FR["xlsx.truncated_rows"].replace("{shown}", "500").replace("{total}", "1200");
assert.ok(txt.includes(expected), `${txt} !includes ${expected}`);
// Nothing to say about the columns here (12 < 40).
assert.ok(!txt.includes(FR["xlsx.truncated_cols"]), txt);
});
await test("notice mentions both axes when rows AND columns overflow", () => {
const area = mount({
sheet: { rows: 1, cols: 40, total_rows: 501, total_cols: 45, max_rows: 500, max_cols: 40, truncated: true },
});
const txt = area.querySelector(".xlsx-truncated").textContent;
assert.ok(
txt.includes(FR["xlsx.truncated_cols"].replace("{shown}", "40").replace("{total}", "45")),
txt
);
});
await test("a payload without the dimensions shows no notice", () => {
// Backward compatibility: an older cached response must not produce "NaN".
const area = mount({ sheet: { name: "Feuille1" } });
assert.equal(area.querySelector(".xlsx-truncated"), null);
assert.ok(!area.textContent.includes("NaN"));
});
// ── A9bis — lazy loading of the truncated tail ──────────────────────────────
const windowHtml = (from, to) =>
'<div class="csv-table-wrapper"><table class="csv-table xlsx-table">' +
"<tbody>" +
Array.from({ length: to - from + 1 }, (_, i) => {
const r = from + i;
return `<tr><th class="xlsx-rownum">${r}</th><td data-cell="A${r}">Ligne ${r}</td></tr>`;
}).join("") +
"</tbody></table></div>";
await test("a truncated sheet gets a load-more footnote, a normal one does not", () => {
const truncated = mountTruncated();
assert.ok(truncated.querySelector(".xlsx-load-more"), "footnote absent");
assert.ok(truncated.querySelector(".xlsx-load-more").textContent.includes(FR["xlsx.load_more"]));
const plain = mount({ sheet: { rows: 10, cols: 2, total_rows: 10, total_cols: 2, max_rows: 500, max_cols: 40, truncated: false } });
assert.equal(plain.querySelector(".xlsx-load-more"), null);
});
await test("clicking the footnote fetches the next window with the right query", async () => {
const area = mountTruncated();
apiQueue.push({
ok: true,
status: 200,
body: { sheet: "Feuille1", offset: 1, limit: 500, rows: 2, total_rows: 3, truncated: true, has_more: false, html: windowHtml(2, 3) },
});
area.querySelector(".xlsx-load-more").click();
await new Promise((r) => setTimeout(r, 5));
assert.equal(calls.length, 1);
assert.match(calls[0].url, /\/api\/file\/V\/xlsx\/sheet\?path=data\.xlsx&sheet=Feuille1&offset=1&limit=500/);
// Rows 2 and 3 landed in the table with their real coordinates.
assert.ok(area.querySelector('td[data-cell="A2"]'));
assert.ok(area.querySelector('td[data-cell="A3"]'));
assert.ok(area.querySelector('th.xlsx-rownum') && area.textContent.includes("Ligne 3"));
// Everything loaded → the footnote is hidden (kept in the DOM, class `done`).
const foot = area.querySelector(".xlsx-load-more");
assert.ok(foot, "the footnote element survives");
assert.equal(foot.classList.contains("done"), true);
assert.ok(!foot.textContent.includes(FR["xlsx.load_more"]), foot.textContent);
});
await test("appended rows are editable and tracked as dirty", async () => {
const area = mountTruncated();
apiQueue.push({
ok: true,
status: 200,
body: { sheet: "Feuille1", offset: 1, limit: 500, rows: 1, total_rows: 3, truncated: true, has_more: false, html: windowHtml(2, 2) },
});
area.querySelector(".xlsx-load-more").click();
await new Promise((r) => setTimeout(r, 5));
editCell(area, "A2", "modifié");
area.querySelector("#xlsx-save-btn").click();
await new Promise((r) => setTimeout(r, 5));
assert.equal(calls.length, 2, "the save went out");
assert.equal(calls[1].url.includes("/xlsx/save"), true);
assert.deepEqual(calls[1].body.cells, { A2: "modifié" });
assert.equal(calls[1].body.sheet, "Feuille1");
});
await test("a failed window fetch keeps the footnote and shows an error toast", async () => {
const area = mountTruncated();
apiQueue.push({ ok: false, status: 500, body: { detail: "boom" } });
area.querySelector(".xlsx-load-more").click();
await new Promise((r) => setTimeout(r, 5));
const foot = area.querySelector(".xlsx-load-more");
assert.ok(foot, "footnote must survive a failed fetch");
assert.ok(foot.textContent.includes(FR["xlsx.load_more"]), foot.textContent);
// `api()` itself toasts the failure (shared behaviour, asserted in E2E);
// here we assert the local consequence: the footnote keeps its label.
// Retrying works once the server answers again.
apiQueue.push({
ok: true,
status: 200,
body: { sheet: "Feuille1", offset: 1, limit: 500, rows: 1, total_rows: 3, truncated: true, has_more: false, html: windowHtml(2, 2) },
});
foot.click();
await new Promise((r) => setTimeout(r, 5));
assert.ok(area.querySelector('td[data-cell="A2"]'));
assert.equal(area.querySelector(".xlsx-load-more").classList.contains("done"), true);
});
// ── A7 — formula bar & keyboard navigation ──────────────────────────────────
await test("the formula bar starts empty and disabled", () => {
const area = mount();
assert.equal(area.querySelector("#xlsx-active-cell").textContent, "—");
assert.equal(area.querySelector("#xlsx-formula-input").disabled, true);
});
await test("focusing a cell shows its name and content in the bar", () => {
const area = mount();
const td = area.querySelector('td[data-cell="A1"]');
td.dispatchEvent(new w.Event("focus", { bubbles: false }));
// JSDOM does not run the default focus behaviour on dispatchEvent, so go
// through the real API:
td.focus();
assert.equal(area.querySelector("#xlsx-active-cell").textContent, "A1");
assert.equal(area.querySelector("#xlsx-formula-input").value, "100");
assert.equal(area.querySelector("#xlsx-formula-input").disabled, false);
});
await test("typing in the bar edits the cell live and marks it dirty", () => {
const area = mount();
const td = area.querySelector('td[data-cell="A1"]');
td.focus();
const input = area.querySelector("#xlsx-formula-input");
input.value = "depuis la barre";
input.dispatchEvent(new w.Event("input", { bubbles: true }));
assert.equal(td.textContent, "depuis la barre");
assert.equal(td.classList.contains("xlsx-dirty"), true);
assert.equal(area.querySelector("#xlsx-save-btn").disabled, false);
// The save payload carries the cell edit.
area.querySelector("#xlsx-save-btn").click();
return new Promise((r) => setTimeout(r, 5)).then(() => {
assert.deepEqual(calls[0].body.cells, { A1: "depuis la barre" });
});
});
await test("Tab and arrows move to the neighbour cell", () => {
const area = mount();
const a1 = area.querySelector('td[data-cell="A1"]');
a1.focus();
a1.dispatchEvent(new w.KeyboardEvent("keydown", { key: "Tab", bubbles: true }));
assert.equal(area.querySelector("#xlsx-active-cell").textContent, "B1");
const b1 = area.querySelector('td[data-cell="B1"]');
b1.dispatchEvent(new w.KeyboardEvent("keydown", { key: "ArrowDown", bubbles: true }));
assert.equal(area.querySelector("#xlsx-active-cell").textContent, "B2");
const b2 = area.querySelector('td[data-cell="B2"]');
b2.dispatchEvent(new w.KeyboardEvent("keydown", { key: "ArrowLeft", bubbles: true }));
assert.equal(area.querySelector("#xlsx-active-cell").textContent, "A2");
});
await test("Enter commits and Shift+Tab goes backwards", () => {
const area = mount();
const a1 = area.querySelector('td[data-cell="A1"]');
a1.focus();
a1.dispatchEvent(new w.KeyboardEvent("keydown", { key: "Tab", bubbles: true }));
const b1 = area.querySelector('td[data-cell="B1"]');
b1.dispatchEvent(new w.KeyboardEvent("keydown", { key: "Tab", shiftKey: true, bubbles: true }));
assert.equal(area.querySelector("#xlsx-active-cell").textContent, "A1");
});
await test("a saved edit from the bar resets the dirty flag and orig value", async () => {
const area = mount();
const td = area.querySelector('td[data-cell="A1"]');
td.focus();
const input = area.querySelector("#xlsx-formula-input");
input.value = "200";
input.dispatchEvent(new w.Event("input", { bubbles: true }));
area.querySelector("#xlsx-save-btn").click();
await new Promise((r) => setTimeout(r, 5));
assert.equal(area.querySelectorAll("td.xlsx-dirty").length, 0);
assert.equal(td.dataset.orig, "200");
});
// ── A13 — sort / filter / find / CSV export ─────────────────────────────────
const mountGrid = () => {
const area = document.getElementById("content-area");
area.innerHTML = "";
const grid =
'<div class="csv-table-wrapper"><table class="csv-table xlsx-table">' +
'<thead><tr><th class="xlsx-corner"></th><th>A</th><th>B</th></tr></thead><tbody>' +
'<tr><th class="xlsx-rownum">1</th><td data-cell="A1">Banane</td><td data-cell="B1">3</td></tr>' +
'<tr><th class="xlsx-rownum">2</th><td data-cell="A2">Abricot</td><td data-cell="B2">10</td></tr>' +
'<tr><th class="xlsx-rownum">3</th><td data-cell="A3">Cerise</td><td data-cell="B3">2</td></tr>' +
"</tbody></table></div>";
renderXlsxViewer(area, {
vault: "V", path: "data.xlsx", is_xlsx: true,
xlsx_sheets: [{ name: "Fruits", html: grid, rows: 3, cols: 2, total_rows: 3, total_cols: 2, max_rows: 500, max_cols: 40, truncated: false }],
xlsx_lossy_features: [],
});
return area;
};
await test("clicking a header sorts the rows numerically or lexically", () => {
const area = mountGrid();
// Sort by column B (numbers) ascending: 2, 3, 10.
area.querySelector(".xlsx-table thead th:nth-child(3)").click();
let cells = [...area.querySelectorAll("tbody td[data-cell^=\"B\"]")].map((td) => td.textContent);
assert.deepEqual(cells, ["2", "3", "10"]);
// Second click: descending.
area.querySelector(".xlsx-table thead th:nth-child(3)").click();
cells = [...area.querySelectorAll("tbody td[data-cell^=\"B\"]")].map((td) => td.textContent);
assert.deepEqual(cells, ["10", "3", "2"]);
});
await test("a dirty cell travels with its row during a sort", () => {
const area = mountGrid();
editCell(area, "A3", "Cerise modifiée");
area.querySelector(".xlsx-table thead th:nth-child(3)").click();
const aCells = [...area.querySelectorAll("tbody td[data-cell^=\"A\"]")].map((td) => td.textContent);
assert.ok(aCells.includes("Cerise modifiée"), aCells.join("|"));
assert.equal(area.querySelectorAll("td.xlsx-dirty").length, 1);
});
await test("the filter hides the rows that do not match", () => {
const area = mountGrid();
// The filter reuses the find input: type and the rows filter live.
const input = area.querySelector("#xlsx-find-input");
input.value = "abri";
input.dispatchEvent(new w.Event("input", { bubbles: true }));
const visible = [...area.querySelectorAll("tbody tr")].filter((tr) => tr.style.display !== "none");
assert.equal(visible.length, 1);
assert.ok(visible[0].textContent.includes("Abricot"));
});
await test("find highlights matches and navigates with the counter", () => {
const area = mountGrid();
const input = area.querySelector("#xlsx-find-input");
const count = area.querySelector("#xlsx-find-count");
input.value = "cerise"; // lowercase: the default search ignores the case
input.dispatchEvent(new w.Event("input", { bubbles: true }));
assert.equal(area.querySelectorAll("mark.xlsx-find-hit").length, 1);
assert.ok(count.textContent.includes("1/1"), count.textContent);
// The hit is inside the matching cell.
assert.ok(area.querySelector('td[data-cell="A3"] mark.xlsx-find-hit'));
});
await test("CSV export downloads the visible sheet without the cached shadows", () => {
const area = mountGrid();
const clicks = [];
const realCreate = document.createElement.bind(document);
const anchor = realCreate("a");
document.createElement = (tag) => {
if (tag === "a") { clicks.push(1); return anchor; }
return realCreate(tag);
};
let href = "";
Object.defineProperty(anchor, "href", { set(v) { href = v; }, get: () => href });
URL.createObjectURL = () => "blob:x";
URL.revokeObjectURL = () => {};
area.querySelector("#xlsx-csv-btn").click();
document.createElement = realCreate;
assert.equal(clicks.length, 1);
assert.equal(anchor.download, "Fruits.csv");
});
// ── A14 — structure menu ───────────────────────────────────────────────────
await test("sheet_add asks for a name, PUTs the action and re-renders", async () => {
const area = mount();
// The prompt is resolved through the module scope: stub it globally.
const realPrompt = globalThis.prompt;
globalThis.prompt = () => "Feuille 2";
apiQueue.push({ ok: true, status: 200, body: { status: "ok" } }); // PUT
apiQueue.push({
ok: true, status: 200,
body: { is_xlsx: true, vault: "V", path: "data.xlsx", xlsx_sheets: [{ name: "Feuille 2", html: sheetHtml("neuf") }], xlsx_lossy_features: [] },
}); // re-read
area.querySelector("#xlsx-structure-btn").click();
const items = [...area.querySelectorAll(".xlsx-structure-item")];
const addBtn = items.find((b) => b.textContent === FR["xlsx.sheet_add"]);
addBtn.click();
await new Promise((r) => setTimeout(r, 5));
globalThis.prompt = realPrompt;
assert.equal(calls.length, 2);
assert.match(calls[0].url, /\/xlsx\/structure\?path=data\.xlsx/);
assert.deepEqual(calls[0].body.actions, [{ op: "sheet_add", name: "Feuille 2" }]);
assert.equal(calls[0].body.force, false);
// The viewer re-rendered from the server payload (a single sheet → no tabs).
assert.ok(
area.querySelector("#content-area, .xlsx-viewer") || area,
"the viewer was rebuilt",
);
assert.ok(
area.querySelector('td[data-cell="A1"]')?.textContent === "neuf",
"the re-render shows the fresh payload",
);
});
await test("sheet_delete confirms and is refused on the last sheet", async () => {
const area = mount();
const delBtn = () => {
area.querySelector("#xlsx-structure-btn").click();
const items = [...area.querySelectorAll(".xlsx-structure-item")];
const b = items.find((x) => x.textContent === FR["xlsx.sheet_delete"]);
b.click();
};
// One sheet only → blocked before even confirming (no network call).
delBtn();
assert.equal(calls.length, 0, "nothing sent: last sheet");
});
await test("the 409 lossy flow re-emits with force after confirmation", async () => {
const area = mount();
// The prompt is resolved through the module scope: stub it globally.
const realPrompt = globalThis.prompt;
globalThis.prompt = () => "Feuille 2";
apiQueue.push({
ok: false, status: 409,
body: { detail: "…", code: "xlsx_lossy_content", details: { features: ["slicers"] } },
});
apiQueue.push({ ok: true, status: 200, body: { status: "ok" } }); // retry w/ force
apiQueue.push({
ok: true, status: 200,
body: { is_xlsx: true, vault: "V", path: "data.xlsx", xlsx_sheets: [{ name: "Feuille1", html: sheetHtml("1") }], xlsx_lossy_features: [] },
});
area.querySelector("#xlsx-structure-btn").click();
const items = [...area.querySelectorAll(".xlsx-structure-item")];
items.find((b) => b.textContent === FR["xlsx.sheet_add"]).click();
await new Promise((r) => setTimeout(r, 10));
globalThis.prompt = realPrompt;
assert.equal(confirmCalls, 1, "the user is asked about the lossy write");
assert.equal(calls.length, 3);
assert.equal(calls[1].body.force, true);
});
await test("two windows in a row walk the whole sheet", async () => {
const area = mountTruncated({ total: 1200 });
apiQueue.push({
ok: true,
status: 200,
body: { sheet: "Feuille1", offset: 1, limit: 500, rows: 500, total_rows: 1200, truncated: true, has_more: true, html: windowHtml(2, 501) },
});
area.querySelector(".xlsx-load-more").click();
await new Promise((r) => setTimeout(r, 5));
assert.ok(area.querySelector('td[data-cell="A501"]'));
// The footnote still shows, with the updated progress.
let foot = area.querySelector(".xlsx-load-more");
assert.ok(foot, "more rows remain");
assert.ok(foot.textContent.includes("501"), foot.textContent);
apiQueue.push({
ok: true,
status: 200,
body: { sheet: "Feuille1", offset: 501, limit: 500, rows: 200, total_rows: 1200, truncated: true, has_more: false, html: windowHtml(502, 701) },
});
foot.click();
await new Promise((r) => setTimeout(r, 5));
assert.ok(area.querySelector('td[data-cell="A701"]'));
assert.equal(area.querySelector(".xlsx-load-more").classList.contains("done"), true);
});
// ── Report ──────────────────────────────────────────────────────────────────
console.log(`\n${passCount}/${testCount} tests passed\n`);
process.exit(passCount === testCount ? 0 : 1);
+14 -3
View File
@@ -12,7 +12,8 @@ from __future__ import annotations
import pytest
from fastapi.testclient import TestClient
from backend.main import _FALLBACK_MODELS, app
from backend.main import app
from backend.routers.config import _FALLBACK_MODELS # ROADMAP #85 T7 — déménagé depuis backend.main
#: Trimmed-down copy of what api.mistral.ai/v1/models really returns (BUG-044).
MISTRAL_LIVE_PAYLOAD = {
@@ -28,11 +29,12 @@ MISTRAL_LIVE_PAYLOAD = {
@pytest.fixture
def admin_client(tmp_path):
"""Minimal admin client for the /api/config/ai-models endpoint."""
from backend.auth.password import hash_password
import json
import os
from pathlib import Path
from backend.auth.password import hash_password
data_dir = tmp_path / "data"
data_dir.mkdir()
users = {
@@ -69,8 +71,9 @@ def admin_client(tmp_path):
import backend.main
backend.main._load_config = lambda: {"watcher_enabled": False}
from backend.indexer import build_index, index
import asyncio
from backend.indexer import build_index, index
for key in list(index.keys()):
del index[key]
loop = asyncio.new_event_loop()
@@ -232,9 +235,11 @@ class TestListModelsEndpoint:
# in backend.main (which does `from backend.ai import ... get_ai_key`).
import backend.ai as aimod
import backend.main as bmain
import backend.routers.config as rconfig
monkeypatch.setattr(aimod, "get_ai_key", lambda name: "fake-xiaomi-key")
if hasattr(bmain, "get_ai_key"):
monkeypatch.setattr(bmain, "get_ai_key", lambda name: "fake-xiaomi-key")
monkeypatch.setattr(rconfig, "get_ai_key", lambda name: "fake-xiaomi-key")
captured = {}
@@ -316,9 +321,11 @@ class TestListModelsEndpoint:
# Patch BOTH the source module AND the imported reference in backend.main
import backend.ai as aimod
import backend.main as bmain
import backend.routers.config as rconfig
monkeypatch.setattr(aimod, "get_ai_key", lambda name: "fake-key")
if hasattr(bmain, "get_ai_key"):
monkeypatch.setattr(bmain, "get_ai_key", lambda name: "fake-key")
monkeypatch.setattr(rconfig, "get_ai_key", lambda name: "fake-key")
import urllib.request as global_urllib_mod
captured_urls = []
@@ -393,10 +400,12 @@ class TestDeclaredCapabilities:
"""main.py binds get_ai_key at import: patch that binding too."""
import backend.ai as aimod
import backend.main as bmain
import backend.routers.config as rconfig
monkeypatch.setattr(aimod, "get_ai_key", lambda name: "fake-mistral-key")
if hasattr(bmain, "get_ai_key"):
monkeypatch.setattr(bmain, "get_ai_key", lambda name: "fake-mistral-key")
monkeypatch.setattr(rconfig, "get_ai_key", lambda name: "fake-mistral-key")
def test_declared_vision_reaches_both_endpoints(self, admin_client, monkeypatch):
self._fake_key(monkeypatch)
@@ -450,10 +459,12 @@ class TestDeclaredCapabilities:
"""No API key: the curated list must still answer correctly (offline)."""
import backend.ai as aimod
import backend.main as bmain
import backend.routers.config as rconfig
monkeypatch.setattr(aimod, "get_ai_key", lambda name: None)
if hasattr(bmain, "get_ai_key"):
monkeypatch.setattr(bmain, "get_ai_key", lambda name: None)
monkeypatch.setattr(rconfig, "get_ai_key", lambda name: None)
token = _login_admin(admin_client)
resp = admin_client.get(
+19 -19
View File
@@ -589,30 +589,30 @@ class TestHumanizeMtime:
pass
def test_humanize_mtime_now(self):
from backend.main import humanize_mtime
from backend.services.recent import humanize_mtime # ROADMAP #85 T8 — ne plus passer par backend.main
import time
assert "instant" in humanize_mtime(time.time())
def test_humanize_mtime_minutes(self):
from backend.main import humanize_mtime
from backend.services.recent import humanize_mtime # ROADMAP #85 T8 — ne plus passer par backend.main
import time
result = humanize_mtime(time.time() - 120)
assert "min" in result
def test_humanize_mtime_hours(self):
from backend.main import humanize_mtime
from backend.services.recent import humanize_mtime # ROADMAP #85 T8 — ne plus passer par backend.main
import time
result = humanize_mtime(time.time() - 7200)
assert "h" in result or "jour" in result
def test_humanize_mtime_days(self):
from backend.main import humanize_mtime
from backend.services.recent import humanize_mtime # ROADMAP #85 T8 — ne plus passer par backend.main
import time
result = humanize_mtime(time.time() - 172800) # 2 days
assert "j" in result
def test_humanize_mtime_old(self):
from backend.main import humanize_mtime
from backend.services.recent import humanize_mtime # ROADMAP #85 T8 — ne plus passer par backend.main
import time
result = humanize_mtime(time.time() - 86400 * 30)
assert "202" in result or result # Should show formatted date
@@ -624,44 +624,44 @@ class TestHumanizeMtime:
class TestHeadingSlugify:
def test_slugify_simple(self):
from backend.main import _heading_slugify
from backend.render import _heading_slugify # ROADMAP #85 T9 — déménagé depuis backend.main
assert _heading_slugify("Hello World") == "hello-world"
def test_slugify_accented(self):
from backend.main import _heading_slugify
from backend.render import _heading_slugify # ROADMAP #85 T9 — déménagé depuis backend.main
result = _heading_slugify("Café Crème")
assert "cafe" in result or "caf" in result
def test_slugify_strips_symbols(self):
from backend.main import _heading_slugify
from backend.render import _heading_slugify # ROADMAP #85 T9 — déménagé depuis backend.main
result = _heading_slugify("Hello, World! Test?")
assert result.startswith("hello")
class TestRenderMarkdown:
def test_render_basic(self):
from backend.main import _render_markdown
from backend.render import _render_markdown # ROADMAP #85 T9 — déménagé depuis backend.main
result = _render_markdown("# Hello\n\nThis is a test.", "TestVault")
assert "<h1" in result
assert "Hello" in result
def test_render_with_code(self):
from backend.main import _render_markdown
from backend.render import _render_markdown # ROADMAP #85 T9 — déménagé depuis backend.main
result = _render_markdown("```python\nprint('hello')\n```", "TestVault")
assert "code" in result or "highlight" in result
def test_render_with_heading_ids(self):
from backend.main import _render_markdown
from backend.render import _render_markdown # ROADMAP #85 T9 — déménagé depuis backend.main
result = _render_markdown("# Title\n## Subtitle", "TestVault")
assert "id=" in result
def test_render_wikilink(self):
from backend.main import _render_markdown
from backend.render import _render_markdown # ROADMAP #85 T9 — déménagé depuis backend.main
result = _render_markdown("Link [[nonexistent.md]] here", "TestVault")
assert "wikilink" in result
def test_render_image_wikilink(self):
from backend.main import _render_markdown
from backend.render import _render_markdown # ROADMAP #85 T9 — déménagé depuis backend.main
result = _render_markdown("![[image.png]]", "TestVault")
assert "img" in result or "image" in result or "wikilink" in result
@@ -709,31 +709,31 @@ class TestCheckVaultWritable:
class TestConvertWikilinks:
def test_convert_wikilink(self):
from backend.main import _convert_wikilinks
from backend.render import _convert_wikilinks # ROADMAP #85 T9 — déménagé depuis backend.main
# Missing wikilinks render as span.wikilink-missing
result = _convert_wikilinks("See [[Introduction à Python]] for details", "TestVault")
assert "Introduction" in result
assert "wikilink" in result
def test_convert_wikilink_with_alias(self):
from backend.main import _convert_wikilinks
from backend.render import _convert_wikilinks # ROADMAP #85 T9 — déménagé depuis backend.main
result = _convert_wikilinks("See [[file.md|a different name]] here", "TestVault")
assert "a different name" in result
def test_convert_wikilink_image(self):
from backend.main import _convert_wikilinks
from backend.render import _convert_wikilinks # ROADMAP #85 T9 — déménagé depuis backend.main
result = _convert_wikilinks("See ![[image.png]] here", "TestVault")
assert "image" in result or "img" in result
def test_convert_wikilink_anchor(self):
from backend.main import _convert_wikilinks
from backend.render import _convert_wikilinks # ROADMAP #85 T9 — déménagé depuis backend.main
result = _convert_wikilinks("[[#Section importante|voir section]]", "TestVault")
assert 'href="#section-importante"' in result
assert "wikilink-anchor" in result
assert "voir section" in result
def test_convert_wikilink_anchor_without_alias(self):
from backend.main import _convert_wikilinks
from backend.render import _convert_wikilinks # ROADMAP #85 T9 — déménagé depuis backend.main
result = _convert_wikilinks("[[#Claude Code]]", "TestVault")
assert 'href="#claude-code"' in result
assert "Claude Code" in result
@@ -741,7 +741,7 @@ class TestConvertWikilinks:
class TestHeadingSlugifyHtmlStripping:
def test_slugify_strips_html_tags(self):
from backend.main import _heading_slugify
from backend.render import _heading_slugify # ROADMAP #85 T9 — déménagé depuis backend.main
result = _heading_slugify('## 1. Agents installés localement <a href="#table-des-matieres">↩</a>')
assert result == "1-agents-installes-localement"
+80 -1
View File
@@ -402,4 +402,83 @@ class TestAvatar:
"username": "admin", "password": "chab30",
})
assert resp.status_code == 200
assert resp.json()["user"]["avatar"] == TINY_PNG
assert resp.json()["user"]["avatar"] == TINY_PNG
# ═══════════════════════════════════════════════════════════════════
# Secure cookies (#87 T8)
# ═══════════════════════════════════════════════════════════════════
class TestSecureCookies:
"""`Secure` auto par défaut : https → flag, http → pas de flag
(les navigateurs jettent les cookies Secure sur http)."""
@staticmethod
def _req(scheme="http", forwarded_proto=None):
from types import SimpleNamespace
headers = {}
if forwarded_proto is not None:
headers["x-forwarded-proto"] = forwarded_proto
return SimpleNamespace(
url=SimpleNamespace(scheme=scheme),
headers=headers,
)
def test_forced_true(self, monkeypatch):
from backend.auth.router import is_secure_cookies
monkeypatch.setenv("OBSIGATE_SECURE_COOKIES", "true")
assert is_secure_cookies(self._req("http")) is True
def test_forced_false(self, monkeypatch):
from backend.auth.router import is_secure_cookies
monkeypatch.setenv("OBSIGATE_SECURE_COOKIES", "false")
assert is_secure_cookies(self._req("https")) is False
def test_auto_http(self, monkeypatch):
from backend.auth.router import is_secure_cookies
monkeypatch.delenv("OBSIGATE_SECURE_COOKIES", raising=False)
assert is_secure_cookies(self._req("http")) is False
def test_auto_https(self, monkeypatch):
from backend.auth.router import is_secure_cookies
monkeypatch.delenv("OBSIGATE_SECURE_COOKIES", raising=False)
assert is_secure_cookies(self._req("https")) is True
def test_auto_forwarded_proto_trusted(self, monkeypatch):
from backend.auth.router import is_secure_cookies
monkeypatch.delenv("OBSIGATE_SECURE_COOKIES", raising=False)
monkeypatch.setenv("OBSIGATE_TRUST_PROXY", "true")
assert is_secure_cookies(self._req("http", "https")) is True
def test_auto_forwarded_proto_untrusted(self, monkeypatch):
from backend.auth.router import is_secure_cookies
monkeypatch.delenv("OBSIGATE_SECURE_COOKIES", raising=False)
monkeypatch.delenv("OBSIGATE_TRUST_PROXY", raising=False)
assert is_secure_cookies(self._req("http", "https")) is False
def test_login_http_sets_cookie_without_secure(self, auth_client, monkeypatch):
monkeypatch.delenv("OBSIGATE_SECURE_COOKIES", raising=False)
resp = auth_client.post("/api/auth/login", json={
"username": "admin", "password": "chab30",
})
assert resp.status_code == 200
set_cookie = resp.headers.get("set-cookie", "")
assert "access_token" in set_cookie
assert "secure" not in set_cookie.lower()
def test_login_https_sets_secure_cookie(self, auth_client, monkeypatch):
"""Même app servie en https → flag Secure présent."""
from backend.main import app
from fastapi.testclient import TestClient
monkeypatch.delenv("OBSIGATE_SECURE_COOKIES", raising=False)
https_client = TestClient(app, base_url="https://testserver",
raise_server_exceptions=False)
try:
resp = https_client.post("/api/auth/login", json={
"username": "admin", "password": "chab30",
})
assert resp.status_code == 200
assert "secure" in resp.headers.get("set-cookie", "").lower()
finally:
if hasattr(https_client, "close"):
https_client.close()
+115
View File
@@ -0,0 +1,115 @@
"""Garde-fous du workflow CI Gitea (BUG-082, BUG-083).
Sans dépendance (pas de PyYAML) : analyse ligne à ligne de
`.gitea/workflows/ci.yml`, suffisante pour les conventions de ce fichier.
"""
from __future__ import annotations
import re
from pathlib import Path
CI_YML = Path(__file__).resolve().parent.parent / ".gitea" / "workflows" / "ci.yml"
REPO_ROOT = Path(__file__).resolve().parent.parent
def _run_bodies() -> list[tuple[int, str]]:
"""Toutes les lignes shell de chaque bloc `run:` → [(n° ligne, code)]."""
lines = CI_YML.read_text(encoding="utf-8").splitlines()
bodies: list[tuple[int, str]] = []
i = 0
while i < len(lines):
m = re.match(r"^(\s*)run:(?:\s*\|\s*)?$", lines[i])
inline = re.match(r"^(\s*)run:\s+(\S.*)$", lines[i])
if m:
base = len(m.group(1))
i += 1
while i < len(lines):
cur = lines[i]
if not cur.strip():
i += 1
continue
if len(cur) - len(cur.lstrip()) <= base:
break
bodies.append((i + 1, cur.strip()))
i += 1
elif inline:
bodies.append((i + 1, inline.group(2).strip()))
i += 1
else:
i += 1
return bodies
class TestRunnerProofScripts:
def test_no_hash_inside_run_bodies(self):
"""BUG-083 : aucun `#` dans le code shell des `run:`.
Le runner Gitea Act tronque naïvement au premier `#` (même entre
guillemets) : `echo "... see #87)"` devenait une citation non
fermée → `unexpected EOF while looking for matching '"'` (job
`security` rouge). Les lignes-commentaires shell (`# ...`) restent
autorisées : leur troncature est sémantiquement neutre.
"""
offenders = [
f"L{n}: {code}"
for n, code in _run_bodies()
if not code.startswith("#") and "#" in code
]
assert not offenders, (
"BUG-083 : `#` interdit dans le code des `run:` "
f"(tronqué par le runner) :\n" + "\n".join(offenders)
)
class TestSemgrepStep:
def test_semgrep_local_rules_enforced(self):
"""#87 T7 : semgrep bloquant sur règles locales (aucun registre)."""
text = CI_YML.read_text(encoding="utf-8")
assert "semgrep --config semgrep-rules/ backend/" in text, (
"#87 T7 : étape semgrep locale attendue dans le job security"
)
rules = REPO_ROOT / "semgrep-rules" / "obsigate-python.yaml"
assert rules.exists(), "ruleset semgrep manquant"
class TestFrontendStepsHaveTheirDeps:
@staticmethod
def _root_step_files() -> list[str]:
"""Fichiers `node tests/frontend/<f>` de l'étape racine (sans jsdom)."""
text = CI_YML.read_text(encoding="utf-8")
root_part = text.split("Frontend JSDOM tests", 1)[0]
root_steps = root_part.split("Frontend unit tests", 1)[1]
return re.findall(r"node tests/frontend/(\S+\.mjs)", root_steps)
@staticmethod
def _has_static_jsdom_import(rel: str) -> bool:
path = REPO_ROOT / "tests" / "frontend" / rel
return any(
re.match(r"^\s*import\b.*\bfrom\s+['\"]jsdom['\"]", line)
or re.match(r"""\brequire\(\s*['"]jsdom['"]\s*\)""", line)
for line in path.read_text(encoding="utf-8").splitlines()
)
def test_root_step_files_need_no_jsdom(self):
"""BUG-082 : l'étape racine tourne sans `tests/frontend/node_modules`
(installé seulement par l'étape JSDOM) : aucun de ses fichiers ne
doit importer `jsdom` statiquement — sinon `ERR_MODULE_NOT_FOUND`
et `lint` rouge (cas `upload.test.mjs`, puis `config-ai-keys.test.mjs`).
"""
offenders = [f for f in self._root_step_files() if self._has_static_jsdom_import(f)]
assert not offenders, (
"BUG-082 : ces fichiers importent `jsdom` mais tournent dans "
"l'étape racine (sans node_modules) — les déplacer dans l'étape "
f"JSDOM :\n" + "\n".join(offenders)
)
def test_jsdom_dependent_tests_run_in_jsdom_step(self):
"""BUG-082 : les suites à import statique `jsdom` tournent bien dans
l'étape JSDOM (les deux branches)."""
text = CI_YML.read_text(encoding="utf-8")
jsdom_part = text.split("Frontend JSDOM tests", 1)[1]
for suite in ("node upload.test.mjs", "node config-ai-keys.test.mjs"):
assert jsdom_part.count(suite) >= 2, (
f"BUG-082 : `{suite}` attendu dans les deux branches de "
"l'étape JSDOM"
)
+82
View File
@@ -0,0 +1,82 @@
"""Tests — nonces CSP (ROADMAP #87 T5b).
- `inject_csp_nonce` ne touche que les scripts inline exécutables
(`<script>`, `type="module"` / `type="importmap"` sans `src`), jamais les
blocs de données (`type="text/plain"`) ni les scripts externes.
- Chaque page HTML servie avec des scripts inline les porte tous avec un
nonce après injection.
"""
from __future__ import annotations
import re
from pathlib import Path
ROOT = Path(__file__).resolve().parent.parent
NONCE = "TESTNONCE1234567890"
def _read(name: str) -> str:
return (ROOT / "frontend" / name).read_text(encoding="utf-8")
def test_inject_only_bare_executable_scripts():
from backend.csp import inject_csp_nonce
html = (
"<script>var a = 1;</script>"
'<script type="module">import x from "y";</script>'
'<script type="importmap">{"imports": {}}</script>'
'<script type="module" src="/static/js/app.js"></script>'
'<script src="https://cdnjs.cloudflare.com/x.js"></script>'
'<script id="raw-content" type="text/plain">hello</script>'
'<script nonce="OLD">var b = 2;</script>'
)
out = inject_csp_nonce(html, NONCE)
assert out.count(f'nonce="{NONCE}"') == 3
assert '<script src="https://cdnjs.cloudflare.com/x.js">' in out
assert '<script id="raw-content" type="text/plain">' in out
assert '<script nonce="OLD">' in out
def test_new_nonce_unique_per_call():
from backend.csp import new_nonce
assert new_nonce() != new_nonce()
def test_all_pages_fully_nonced():
"""Aucun script inline exécutable sans nonce après injection (sauf src=)."""
from backend.csp import inject_csp_nonce
for name in ("index.html", "popout.html", "admin.html", "editor-poc.html", "excalidraw-editor.html"):
out = inject_csp_nonce(_read(name), NONCE)
bare = re.findall(r"<script>", out)
assert not bare, f"{name} : scripts sans nonce restants"
inline_mods = [m for m in re.findall(r'<script type="(?:module|importmap)">', out)]
assert not inline_mods, f"{name} : modules/importmap sans nonce restants"
def _nonce_of(csp: str) -> str | None:
m = re.search(r"'nonce-([^']+)'", csp or "")
return m.group(1) if m else None
def test_nonce_header_fresh_per_response(client):
"""Chaque réponse porte un nonce frais dans `script-src`."""
r1 = client.get("/")
r2 = client.get("/")
n1, n2 = _nonce_of(r1.headers.get("content-security-policy")), _nonce_of(
r2.headers.get("content-security-policy")
)
assert n1 and n2 and n1 != n2
def test_nonce_matches_injected_html(client):
"""Le nonce de l'en-tête est celui injecté dans le HTML (`/`, excalidraw)."""
for path in ("/", "/excalidraw-editor.html"):
resp = client.get(path)
assert resp.status_code == 200, path
nonce = _nonce_of(resp.headers.get("content-security-policy"))
assert nonce, path
assert f'nonce="{nonce}"' in resp.text, path
+117
View File
@@ -0,0 +1,117 @@
"""Garde-fous anti-blocage du harnais E2E local (BUG-080).
Contexte : un run `npm run test:e2e:ps` est resté pendu toute la nuit —
serveurs orphelins sur le port 2029, `npx` sans `--yes` (prompt interactif
qui attend indéfiniment), installation des navigateurs systématique et suite
Playwright (~130 tests, workers: 1) sans aucun timeout global.
Ces tests statiques vérifient que chaque couche du harnais possède son
garde-fou, afin qu'un run E2E échoue vite au lieu de bloquer indéfiniment.
"""
from __future__ import annotations
import re
from pathlib import Path
REPO_ROOT = Path(__file__).resolve().parent.parent
def _read(rel: str) -> str:
return (REPO_ROOT / rel).read_text(encoding="utf-8")
class TestE2ELocalPs:
SCRIPT = "scripts/run-e2e-local.ps1"
def test_playwright_via_node_no_npx(self):
"""Playwright est lancé via `node` direct, jamais via `npx`.
`Start-Process` ne peut pas exécuter `npx` (ni le `.ps1` ni le
`.cmd` ne sont des applications Win32 directes : "%1 is not a valid
Win32 application"), et `npx` sans `--yes` peut pendre sur un prompt
interactif. Seules les mentions en commentaires/logs sont tolérées.
"""
content = _read(self.SCRIPT)
bare = [
line.strip()
for line in content.splitlines()
if re.match(r"^\s*(?:&\s*)?npx\s", line)
]
assert not bare, f"invocations npx nues : {bare}"
assert "node_modules/@playwright/test/cli.js" in content, (
"CLI Playwright locale attendue (via node)"
)
# `$Args` est une variable automatique PowerShell : un paramètre de
# ce nom serait écrasé (helper lancé sans arguments → exit 0 muet).
# (commentaires `#` exclus : la mise en garde elle-même le cite).
code_lines = [
line for line in content.splitlines()
if not line.strip().startswith("#")
]
assert not re.search(r"\$Args\b", "\n".join(code_lines)), (
"BUG-080 : paramètre `$Args` interdit (shadowing par $args automatique)"
)
def test_browser_install_skippable(self):
"""Install navigateurs sautée si chromium déjà présent (sauf forçage)."""
content = _read(self.SCRIPT)
assert "Test-ChromiumInstalled" in content
assert "E2E_INSTALL_BROWSERS" in content
def test_test_step_has_timeout(self):
"""L'étape `playwright test` est bornée (E2E_TIMEOUT_SEC, défaut 1800).
Le défaut dépasse le globalTimeout Playwright (25 min en local) pour
que ce soit Playwright qui abandonne proprement (avec rapport) en premier.
"""
content = _read(self.SCRIPT)
assert "E2E_TIMEOUT_SEC" in content
assert "Wait-Process -Timeout" in content
assert re.search(r"E2E_TIMEOUT_SEC.*else\s*\{\s*1800\s*\}", content), (
"défaut E2E_TIMEOUT_SEC=1800 attendu"
)
class TestE2ELocalSh:
SCRIPT = "scripts/run-e2e-local.sh"
def test_npx_never_prompts(self):
content = _read(self.SCRIPT)
for line in content.splitlines():
stripped = line.strip()
if stripped.startswith("#") or stripped.startswith("echo") or "npx" not in stripped:
continue
if "playwright" in stripped:
assert "--yes" in stripped, f"appel npx sans --yes : {stripped}"
def test_test_step_has_timeout(self):
content = _read(self.SCRIPT)
assert "E2E_TIMEOUT_SEC" in content
assert "run_with_timeout" in content
class TestE2EServerPs:
SCRIPT = "scripts/e2e-server.ps1"
def test_pidfile_refreshed_with_port_owner(self):
"""Le pidfile est resynchronisé sur le vrai PID d'écoute après READY."""
content = _read(self.SCRIPT)
assert "Get-PortOwner" in content
ready_pos = content.find("[OK] READY")
assert ready_pos != -1
assert "Set-Content -LiteralPath $PidFile" in content[ready_pos - 600:ready_pos]
def test_stop_kills_process_tree(self):
"""`stop` tue aussi les enfants du PID enregistré (pas d'orphelins)."""
content = _read(self.SCRIPT)
assert "ParentProcessId=$srvPid" in content
class TestPlaywrightConfig:
CONFIG = "playwright.config.ts"
def test_global_timeout_set(self):
"""Timeout global : la suite abandonne au lieu de pendre toute la nuit."""
content = _read(self.CONFIG)
assert "globalTimeout" in content
assert "E2E_GLOBAL_TIMEOUT_MS" in content
+198
View File
@@ -0,0 +1,198 @@
"""Tests de durcissement — concurrence users.json + fuzzing regex (ROADMAP #87 T2).
- `users.json` : les read-modify-write sont sérialisés par `_users_lock`
(BUG-029). Ces tests martèlent create/update/record_login_failure depuis
plusieurs threads et exigent zéro mise à jour perdue + un JSON valide.
- Regex (BUG-025) : la politique `regex_safety` (longueur, quantificateurs
imbriqués, contenu tronqué) doit rejeter vite les motifs catastrophiques
et borner le temps des motifs acceptés sur gros contenu.
"""
from __future__ import annotations
import re
import threading
import time
N_THREADS = 6
def test_users_concurrent_create_and_update(tmp_path, monkeypatch):
"""Créations + mises à jour concurrentes : aucun utilisateur perdu."""
from backend.auth import user_store
monkeypatch.setattr(user_store, "USERS_FILE", tmp_path / "users.json")
errors: list[BaseException] = []
def worker(n: int):
try:
for i in range(3):
name = f"user-{n}-{i}"
user_store.create_user(name, "Motdepasse1!", display_name=name)
user_store.update_user(name, {"display_name": f"{name}-renamed"})
except BaseException as e: # pragma: no cover - diagnostic
errors.append(e)
threads = [threading.Thread(target=worker, args=(n,)) for n in range(N_THREADS)]
for t in threads:
t.start()
for t in threads:
t.join()
assert not errors
users = user_store._read()["users"]
assert len(users) == N_THREADS * 3
assert all(u["display_name"].endswith("-renamed") for u in users.values())
def test_users_concurrent_login_failures_no_lost_count(tmp_path, monkeypatch):
"""`record_login_failure` concurrents : compteur exact (pas de lost update)."""
from backend.auth import user_store
monkeypatch.setattr(user_store, "USERS_FILE", tmp_path / "users.json")
user_store.create_user("victim", "Motdepasse1!")
def worker():
for _ in range(10):
try:
user_store.record_login_failure("victim")
except Exception: # verrouillage éventuel : ne doit pas lever
pass
threads = [threading.Thread(target=worker) for _ in range(N_THREADS)]
for t in threads:
t.start()
for t in threads:
t.join()
user = user_store.get_user("victim")
assert user is not None
# Le compte peut se verrouiller en cours de route ; l'important est que
# le fichier reste un JSON valide et l'utilisateur présent.
assert user["username"] == "victim"
def test_users_file_stays_valid_json_under_load(tmp_path, monkeypatch):
"""Le fichier reste lisible à tout moment pendant les écritures."""
import json
from backend.auth import user_store
target = tmp_path / "users.json"
monkeypatch.setattr(user_store, "USERS_FILE", target)
user_store.create_user("base", "Motdepasse1!")
stop = threading.Event()
errors: list[BaseException] = []
def writer(n: int):
i = 0
while not stop.is_set():
try:
user_store.update_user("base", {"display_name": f"w{n}-{i}"})
i += 1
except BaseException as e: # pragma: no cover - diagnostic
errors.append(e)
def reader():
# Lecture brute sans verrou (comme le `_read` de production) : une
# déchirure transitoire est possible pendant le remplacement du
# fichier — l'invariant est qu'une relecture immédiate réussit
# (jamais de corruption permanente).
while not stop.is_set():
try:
raw = target.read_text(encoding="utf-8")
json.loads(raw)
except FileNotFoundError:
pass
except json.JSONDecodeError:
try:
time.sleep(0.01)
json.loads(target.read_text(encoding="utf-8"))
except FileNotFoundError:
pass
except BaseException as e: # pragma: no cover - diagnostic
errors.append(e)
except BaseException as e: # pragma: no cover - diagnostic
errors.append(e)
threads = [threading.Thread(target=writer, args=(n,)) for n in range(4)]
threads.append(threading.Thread(target=reader))
for t in threads:
t.start()
time.sleep(2.0)
stop.set()
for t in threads:
t.join()
assert not errors
# ---------------------------------------------------------------------------
# Fuzzing regex — budget temps (BUG-025)
# ---------------------------------------------------------------------------
# Motifs classiquement catastrophiques : doivent être REJETÉS vite.
CATASTROPHIC = [
"^(a+)+$",
"(a+)+$",
"(.*)*$",
"(a|aa)+$",
"(a+){2,}$",
r"(\w+)+$",
r"(a*)*b",
r"(x+x+)+y",
]
# Motifs acceptés (légitimes) : doivent tourner vite sur gros contenu.
ACCEPTED = [
r"hello",
r"h.llo",
r"\b\w+@\w+\.\w+\b",
r"[A-ZÉÈÊ][a-zéèêàâîôûç]+",
r"(ab|cd)+e",
r"\d{4}-\d{2}-\d{2}",
r"foo|bar|baz",
]
def test_catastrophic_patterns_rejected_fast():
"""Les motifs à backtracking catastrophique sont refusés en < 1 s."""
from backend.services.regex_safety import validate_regex
start = time.perf_counter()
for pattern in CATASTROPHIC:
try:
validate_regex(pattern)
except ValueError:
pass
assert time.perf_counter() - start < 1.0
def test_accepted_patterns_bounded_on_large_content():
"""Motifs acceptés sur 200 Ko : chacun < 5 s (budget large anti-flaky)."""
from backend.services.regex_safety import MAX_REGEX_CONTENT, truncate_for_regex, validate_regex
assert MAX_REGEX_CONTENT == 200_000
content = truncate_for_regex("abc héllo world [email protected] 2024-01-02 " * 5000)
assert len(content) <= MAX_REGEX_CONTENT
for pattern in ACCEPTED:
validate_regex(pattern) # ne doit pas lever
start = time.perf_counter()
re.search(pattern, content)
assert time.perf_counter() - start < 5.0, f"motif lent : {pattern!r}"
def test_validate_regex_policy():
"""Politique : vide/trop long/invalide → ValueError."""
from backend.services.regex_safety import MAX_PATTERN_LENGTH, validate_regex
for bad in ("", "x" * (MAX_PATTERN_LENGTH + 1), "(unclosed"):
try:
validate_regex(bad)
except ValueError:
pass
else: # pragma: no cover - doit lever
raise AssertionError(f"motif accepté à tort : {bad!r}")
assert validate_regex("simple") == "simple"
+34
View File
@@ -0,0 +1,34 @@
"""Parité i18n FR/EN des locales du frontend (#87 T9).
`frontend/locales/fr.json` et `en.json` doivent exposer exactement les mêmes
clés (comparaison profonde) : toute clé manquante fait afficher la clé brute
dans l'UI au lieu du libellé.
"""
from __future__ import annotations
import json
from pathlib import Path
LOCALES = Path(__file__).resolve().parent.parent / "frontend" / "locales"
def _flat(d: dict, prefix: str = "") -> set[str]:
keys = set()
for k, v in d.items():
name = f"{prefix}.{k}" if prefix else str(k)
if isinstance(v, dict):
keys |= _flat(v, name)
else:
keys.add(name)
return keys
def _load(lang: str) -> set[str]:
return _flat(json.loads((LOCALES / f"{lang}.json").read_text(encoding="utf-8")))
class TestI18nParity:
def test_fr_en_same_keys(self):
fr, en = _load("fr"), _load("en")
assert not (fr - en), f"clés sans traduction EN : {sorted(fr - en)[:10]}"
assert not (en - fr), f"clés sans traduction FR : {sorted(en - fr)[:10]}"
+61
View File
@@ -367,3 +367,64 @@ class TestMfaApiEndpoints:
data = login_resp.json()
assert "access_token" in data
assert data.get("mfa_required") is None
# ── BUG-081 : /api/auth/mfa/status avec auth désactivée ──────────────────
@pytest.fixture
def mfa_client_noauth():
"""TestClient avec auth DÉSACTIVÉE (OBSIGATE_AUTH_ENABLED=false)."""
tmp = Path(tempfile.mkdtemp())
data_dir = tmp / "data"
data_dir.mkdir()
orig_cwd = os.getcwd()
test_vault_path = os.path.abspath("test-vault")
os.chdir(str(tmp))
os.environ["VAULT_1_NAME"] = "TestVault"
os.environ["VAULT_1_PATH"] = test_vault_path
os.environ["OBSIGATE_AUTH_ENABLED"] = "false"
os.environ["OBSIGATE_WATCHER_ENABLED"] = "false"
import backend.main
backend.main._load_config = lambda: {"watcher_enabled": False}
from backend.main import app
from backend.indexer import build_index, index
for key in list(index.keys()):
del index[key]
loop = asyncio.new_event_loop()
asyncio.set_event_loop(loop)
loop.run_until_complete(build_index())
from backend.search import init_inverted_index
init_inverted_index()
from fastapi.testclient import TestClient
client = TestClient(app, raise_server_exceptions=False)
yield client
if hasattr(client, 'close'):
client.close()
loop.run_until_complete(asyncio.sleep(0))
os.chdir(orig_cwd)
shutil.rmtree(str(tmp), ignore_errors=True)
for k in ["VAULT_1_NAME", "VAULT_1_PATH", "OBSIGATE_AUTH_ENABLED",
"OBSIGATE_WATCHER_ENABLED"]:
os.environ.pop(k, None)
class TestMfaStatusAuthDisabled:
"""BUG-081 : `GET /api/auth/mfa/status` ne doit pas répondre 500 quand
l'auth est désactivée (pseudo-user `anonymous` sans entrée en store)."""
def test_mfa_status_anonymous_returns_disabled(self, mfa_client_noauth):
resp = mfa_client_noauth.get("/api/auth/mfa/status")
assert resp.status_code == 200, f"BUG-081: {resp.status_code} {resp.text[:200]}"
body = resp.json()
assert body["mfa_enabled"] is False
assert body["totp_enabled"] is False
assert body["webauthn_credentials"] == 0
+143
View File
@@ -0,0 +1,143 @@
"""Tests — rate-limit SQLite optionnel (ROADMAP #85 T10b).
Le store mémoire reste le défaut (comportement inchangé) ; si
``OBSIGATE_RATELIMIT_DB`` pointe vers un fichier SQLite, les compteurs y
sont persistés (partagés entre workers/processus, conservés au redémarrage)
avec une sémantique identique (fenêtre glissante, budgets IP + compte,
reset au succès).
"""
from __future__ import annotations
import sqlite3
import threading
import time
def _use_db(monkeypatch, tmp_path):
monkeypatch.setenv("OBSIGATE_RATELIMIT_DB", str(tmp_path / "ratelimit.db"))
def test_memory_default_unchanged(monkeypatch):
"""Sans la variable d'env : le store mémoire historique est utilisé."""
from backend import ratelimit
monkeypatch.delenv("OBSIGATE_RATELIMIT_DB", raising=False)
assert ratelimit._db_path() is None
ip = "10.9.9.1"
ratelimit._ip_attempts.pop(ip, None)
assert not ratelimit.is_rate_limited(ip)
ratelimit.record_failure(ip)
assert not ratelimit.is_rate_limited(ip)
ratelimit.record_success(ip)
assert not ratelimit.is_rate_limited(ip)
def test_sqlite_failures_and_limit(monkeypatch, tmp_path):
"""Budget IP : N échecs → limité ; succès → reset (SQLite)."""
from backend import ratelimit
_use_db(monkeypatch, tmp_path)
monkeypatch.setattr(ratelimit, "MAX_ATTEMPTS", 3)
ip = "10.8.8.1"
assert not ratelimit.is_rate_limited(ip)
ratelimit.record_failure(ip)
ratelimit.record_failure(ip)
assert not ratelimit.is_rate_limited(ip)
failures, remaining = ratelimit.record_failure(ip)
assert (failures, remaining) == (3, 0)
assert ratelimit.is_rate_limited(ip)
ratelimit.record_success(ip)
assert not ratelimit.is_rate_limited(ip)
def test_sqlite_account_budget_case_insensitive(monkeypatch, tmp_path):
"""Budget par compte : insensible à la casse, indépendant des IP."""
from backend import ratelimit
_use_db(monkeypatch, tmp_path)
monkeypatch.setattr(ratelimit, "ACCOUNT_MAX_ATTEMPTS", 2)
assert not ratelimit.is_account_rate_limited("Alice")
ratelimit.record_account_failure("alice")
assert not ratelimit.is_account_rate_limited("ALICE")
ratelimit.record_account_failure("ALICE")
assert ratelimit.is_account_rate_limited("alice")
# Le budget IP n'est pas affecté par le budget compte.
assert not ratelimit.is_rate_limited("1.2.3.4")
ratelimit.record_account_success("alice")
assert not ratelimit.is_account_rate_limited("alice")
def test_sqlite_window_expiry(monkeypatch, tmp_path):
"""Les tentatives hors fenêtre ne comptent plus (SQLite)."""
from backend import ratelimit
_use_db(monkeypatch, tmp_path)
monkeypatch.setattr(ratelimit, "MAX_ATTEMPTS", 2)
monkeypatch.setattr(ratelimit, "WINDOW_SECONDS", 1)
ip = "10.7.7.1"
ratelimit.record_failure(ip)
ratelimit.record_failure(ip)
assert ratelimit.is_rate_limited(ip)
time.sleep(1.1)
assert not ratelimit.is_rate_limited(ip)
def test_sqlite_persists_across_restart(monkeypatch, tmp_path):
"""Les compteurs survivent au redémarrage (même fichier)."""
import os
from backend import ratelimit
db = tmp_path / "ratelimit.db"
monkeypatch.setenv("OBSIGATE_RATELIMIT_DB", str(db))
monkeypatch.setattr(ratelimit, "MAX_ATTEMPTS", 5)
for _ in range(3):
ratelimit.record_failure("10.6.6.6")
assert os.path.exists(db)
# "Redémarrage" : le module relit le même fichier (connexions courtes).
assert ratelimit.get_status("10.6.6.6")["failures"] == 3
with sqlite3.connect(str(db)) as conn:
(rows,) = conn.execute("SELECT COUNT(*) FROM attempts").fetchone()
assert rows == 3
def test_sqlite_get_status_shapes(monkeypatch, tmp_path):
"""`get_status` garde les mêmes formes qu'en mémoire."""
from backend import ratelimit
_use_db(monkeypatch, tmp_path)
ratelimit.record_failure("10.5.5.5")
ratelimit.record_account_failure("bob")
per_ip = ratelimit.get_status("10.5.5.5")
assert per_ip == {
"ip": "10.5.5.5",
"failures": 1,
"max": ratelimit.MAX_ATTEMPTS,
"limited": False,
"window_seconds": ratelimit.WINDOW_SECONDS,
}
glob = ratelimit.get_status()
assert glob["tracked_ips"] == 1
assert glob["tracked_accounts"] == 1
assert glob["limited_ips"] == 0
assert glob["max_attempts"] == ratelimit.MAX_ATTEMPTS
def test_sqlite_concurrent_writes(monkeypatch, tmp_path):
"""Écritures concurrentes : aucun échec compté perdu (SQLite/WAL)."""
from backend import ratelimit
_use_db(monkeypatch, tmp_path)
monkeypatch.setattr(ratelimit, "MAX_ATTEMPTS", 10_000)
def worker(n: int):
for _ in range(25):
ratelimit.record_failure("10.4.4.4")
threads = [threading.Thread(target=worker, args=(n,)) for n in range(8)]
for t in threads:
t.start()
for t in threads:
t.join()
assert ratelimit.get_status("10.4.4.4")["failures"] == 200
+56
View File
@@ -115,6 +115,25 @@ class TestInvertedIndex:
inv.remove_document("V", "p.md")
assert inv.doc_count == 0 # Skipped
def test_is_ready_tracks_initial_build(self, client):
"""is_ready() is the only freshness signal: no generation counter,
no cooldown, no lazy rebuild (plan.md step 6)."""
inv = InvertedIndex()
assert inv.is_ready() is False
inv.rebuild()
assert inv.is_ready() is True
# The old staleness API is gone for good.
assert not hasattr(inv, "is_stale")
def test_is_ready_survives_incremental_updates(self, client):
"""Incremental add/remove must not flip readiness back (which would
silently push search() onto the O(N) full-scan fallback)."""
self.inv.rebuild()
assert self.inv.is_ready() is True
self.inv.add_document("V", "p.md", {"path": "p.md", "title": "T", "tags": [], "content": "x"})
self.inv.remove_document("V", "p.md")
assert self.inv.is_ready() is True
# ═══════════════════════════════════════════════════════════════════
# Search / Advanced Search integration tests
@@ -192,3 +211,40 @@ class TestSearchFunctions:
def test_suggest_tags_no_match(self, client):
suggestions = suggest_tags("xyznonexistent", vault_filter="all")
assert len(suggestions) == 0
# ═══════════════════════════════════════════════════════════════════
# Vault removal — inverted index must not keep ghost documents
# ═══════════════════════════════════════════════════════════════════
class TestVaultRemovalPurgesInvertedIndex:
"""`remove_vault_from_index()` must notify the inverted-index hook.
Regression: it only cleaned the indexer's own structures, so every document
of the removed vault survived in the inverted index (postings, doc_info,
doc_vault, vault_docs) and kept matching searches for a vault that no
longer exists — a leak that only a manual reindex used to clear.
"""
def test_removing_a_vault_purges_its_documents(self, client):
import asyncio
import backend.indexer as ix
import backend.search as bs
ix.set_index_change_hook(bs._on_index_change_hook)
inv = bs._inverted_index
inv.rebuild()
vault_keys = [k for k in inv.doc_info if k.startswith("TestVault::")]
assert vault_keys, "vault non indexe, test sans valeur"
before = inv.doc_count
asyncio.run(ix.remove_vault_from_index("TestVault"))
ghosts = [k for k in inv.doc_info if k.startswith("TestVault::")]
assert not ghosts, f"documents fantomes dans l'index inverse : {ghosts[:5]}"
assert inv.doc_count == before - len(vault_keys)
assert "TestVault" not in inv.vault_docs
# The index stays usable for the remaining vaults.
assert inv.is_ready() is True
+74
View File
@@ -0,0 +1,74 @@
"""Tests — cookies Secure, CORS same-origin explicite, avertissement bind (ROADMAP #87 T3/T8).
- `is_secure_cookies()` : `OBSIGATE_SECURE_COOKIES=true|false|auto` (défaut
`auto` : Secure si la requête arrive en https, sinon pas de flag — les
navigateurs ignorent les cookies `Secure` en clair).
- CORS same-origin EXPLICITE : `CORSMiddleware(allow_origins=[])` — aucun
`Access-Control-Allow-*` n'est émis même avec un `Origin` cross-origin,
et les preflights sont rejetés (400).
"""
from __future__ import annotations
def test_secure_cookies_default_auto(monkeypatch):
"""Défaut `auto` : sans requête → pas de flag (logins HTTP locaux préservés)."""
from backend.auth.router import is_secure_cookies
monkeypatch.delenv("OBSIGATE_SECURE_COOKIES", raising=False)
assert is_secure_cookies() is False
def test_secure_cookies_forced_values(monkeypatch):
"""`true`/`1`/`yes` → flag ; `false`/`0`/`no` → pas de flag (insensible à la casse)."""
from backend.auth.router import is_secure_cookies
for value in ("true", "True", "TRUE", "1", "yes", "on"):
monkeypatch.setenv("OBSIGATE_SECURE_COOKIES", value)
assert is_secure_cookies() is True
for value in ("false", "False", "FALSE", "0", "no", "off"):
monkeypatch.setenv("OBSIGATE_SECURE_COOKIES", value)
assert is_secure_cookies() is False
def test_no_cors_headers_on_api(client):
"""Pas de `Access-Control-Allow-Origin` : same-origin imposé par le navigateur."""
resp = client.get("/api/health")
assert resp.status_code == 200
assert "access-control-allow-origin" not in {k.lower() for k in resp.headers}
def test_no_cors_headers_on_public_share(client):
"""Idem sur la page publique de partage."""
resp = client.get("/s/jeton-inexistant")
assert resp.status_code == 404
assert "access-control-allow-origin" not in {k.lower() for k in resp.headers}
def test_cross_origin_get_emits_no_acao(client):
"""#87 T8 : même avec un `Origin` cross-origin, aucun ACAO (refus explicite)."""
resp = client.get("/api/health", headers={"Origin": "http://evil.example"})
assert resp.status_code == 200
assert "access-control-allow-origin" not in {k.lower() for k in resp.headers}
def test_cross_origin_preflight_rejected(client):
"""#87 T8 : preflight cross-origin → 400 (origine non autorisée)."""
resp = client.options(
"/api/health",
headers={
"Origin": "http://evil.example",
"Access-Control-Request-Method": "GET",
},
)
assert resp.status_code == 400
def test_security_headers_present(client):
"""En-têtes de durcissement posés par le middleware (non-régression)."""
resp = client.get("/api/health")
assert resp.headers.get("x-content-type-options") == "nosniff"
assert resp.headers.get("x-frame-options") == "SAMEORIGIN"
csp = resp.headers.get("content-security-policy", "")
assert "object-src 'none'" in csp
assert "frame-ancestors 'self'" in csp
+225
View File
@@ -0,0 +1,225 @@
"""Unit tests for the existing-workbook AI tools (#153 A6).
Covers ``list_xlsx_sheets``, ``xlsx_to_markdown``, ``update_xlsx_cells`` and
``append_xlsx_rows`` — risk levels, confirmation gating, vault persistence and
the reuse of the guarded mutation service (P0 guards, formula neutralisation).
"""
from __future__ import annotations
from pathlib import Path
import pytest
from backend.tools.api import (
ToolConfirmationRequired,
ToolContext,
ToolError,
call_tool,
get_tool,
)
from backend.tools.context import ToolRisk
openpyxl = pytest.importorskip("openpyxl")
@pytest.fixture
def vault(tmp_path, monkeypatch):
"""A minimal configured vault (index entry patched, no full build)."""
vault_dir = tmp_path / "Vault"
vault_dir.mkdir()
monkeypatch.setitem(
__import__("backend.indexer", fromlist=["index"]).index,
"Vault",
{"name": "Vault", "path": str(vault_dir), "config": {}},
)
return vault_dir
@pytest.fixture
def workbook(vault: Path) -> Path:
"""A two-sheet workbook: Budget (values + a formula) and Notes."""
path = vault / "classeur.xlsx"
wb = openpyxl.Workbook()
ws = wb.active
ws.title = "Budget"
ws.append(["Poste", "Montant"])
ws.append(["Loyer", 900])
ws.append(["Courses", 250])
notes = wb.create_sheet("Notes")
notes["A1"] = "bonjour"
wb.save(path)
return path
def _ctx() -> ToolContext:
return ToolContext(
user={"username": "tester", "role": "admin", "vaults": ["*"]},
audit_enabled=False,
)
class TestRegistry:
def test_read_tools_are_read_risk(self):
for name in ("list_xlsx_sheets", "xlsx_to_markdown"):
spec = get_tool(name)
assert spec is not None
assert spec.risk == ToolRisk.READ
def test_mutation_tools_require_confirmation(self):
for name in ("update_xlsx_cells", "append_xlsx_rows"):
spec = get_tool(name)
assert spec is not None
assert spec.risk == ToolRisk.WRITE
assert spec.requires_confirmation is True
def test_mutation_tools_raise_without_confirmation(self, vault, workbook):
with pytest.raises(ToolConfirmationRequired):
call_tool("update_xlsx_cells", _ctx(), {
"vault": "Vault", "path": "classeur.xlsx",
"sheet": "Budget", "cells": {"B3": "300"},
})
with pytest.raises(ToolConfirmationRequired):
call_tool("append_xlsx_rows", _ctx(), {
"vault": "Vault", "path": "classeur.xlsx",
"sheet": "Budget", "rows": [["Total", 1150]],
})
class TestListXlsxSheets:
def test_lists_names_and_dimensions(self, vault, workbook):
out = call_tool("list_xlsx_sheets", _ctx(), {
"vault": "Vault", "path": "classeur.xlsx",
})
assert out.ok
data = out.data
assert [s["name"] for s in data["sheets"]] == ["Budget", "Notes"]
budget = data["sheets"][0]
assert budget["total_rows"] == 3 and budget["total_cols"] == 2
assert budget["truncated"] is False
def test_wrong_extension_rejected(self, vault):
with pytest.raises(ToolError):
call_tool("list_xlsx_sheets", _ctx(), {
"vault": "Vault", "path": "note.md",
})
def test_missing_file_rejected(self, vault):
with pytest.raises(ToolError):
call_tool("list_xlsx_sheets", _ctx(), {
"vault": "Vault", "path": "absent.xlsx",
})
class TestXlsxToMarkdown:
def test_renders_a_bounded_markdown_table(self, vault, workbook):
out = call_tool("xlsx_to_markdown", _ctx(), {
"vault": "Vault", "path": "classeur.xlsx",
})
assert out.ok
data = out.data
assert data["sheet"] == "Budget"
assert data["rows"] == 3 and data["truncated"] is False
assert "| Poste | Montant |" in data["markdown"]
assert "| Loyer | 900 |" in data["markdown"]
def test_specific_sheet(self, vault, workbook):
out = call_tool("xlsx_to_markdown", _ctx(), {
"vault": "Vault", "path": "classeur.xlsx", "sheet": "Notes",
})
assert out.ok
data = out.data
assert data["sheet"] == "Notes"
assert "bonjour" in data["markdown"]
def test_unknown_sheet_rejected(self, vault, workbook):
with pytest.raises(ToolError):
call_tool("xlsx_to_markdown", _ctx(), {
"vault": "Vault", "path": "classeur.xlsx", "sheet": "Nope",
})
def test_big_sheet_is_flagged_truncated(self, vault):
path = vault / "gros.xlsx"
wb = openpyxl.Workbook()
ws = wb.active
for i in range(150):
ws.append([f"r{i}", i])
wb.save(path)
out = call_tool("xlsx_to_markdown", _ctx(), {
"vault": "Vault", "path": "gros.xlsx",
})
assert out.ok
data = out.data
assert data["rows"] == 100
assert data["truncated"] is True
class TestUpdateXlsxCells:
def test_edits_cells_and_survives_a_reload(self, vault, workbook):
out = call_tool("update_xlsx_cells", _ctx(), {
"vault": "Vault", "path": "classeur.xlsx",
"sheet": "Budget", "cells": {"B3": "300"},
}, confirm=True)
assert out.ok and out.data["status"] == "ok"
wb = openpyxl.load_workbook(workbook)
assert wb["Budget"]["B3"].value == 300 # coerced like the viewer
wb.close()
def test_formula_stays_text_by_default(self, vault, workbook):
out = call_tool("update_xlsx_cells", _ctx(), {
"vault": "Vault", "path": "classeur.xlsx",
"sheet": "Budget", "cells": {"C1": "=B2+B3"},
}, confirm=True)
assert out.ok
wb = openpyxl.load_workbook(workbook)
assert wb["Budget"]["C1"].data_type == "s" # A4 guard inherited
wb.close()
def test_empty_cells_rejected(self, vault, workbook):
with pytest.raises(ToolError):
call_tool("update_xlsx_cells", _ctx(), {
"vault": "Vault", "path": "classeur.xlsx",
"sheet": "Budget", "cells": {},
})
class TestAppendXlsxRows:
def test_appends_below_the_last_row(self, vault, workbook):
out = call_tool("append_xlsx_rows", _ctx(), {
"vault": "Vault", "path": "classeur.xlsx",
"sheet": "Budget", "rows": [["Total", 1150]],
}, confirm=True)
assert out.ok
assert out.data["first_row"] == 4
wb = openpyxl.load_workbook(workbook)
ws = wb["Budget"]
assert ws["A4"].value == "Total"
assert ws["B4"].value == 1150 and isinstance(ws["B4"].value, int)
wb.close()
def test_values_are_coerced(self, vault, workbook):
out = call_tool("append_xlsx_rows", _ctx(), {
"vault": "Vault", "path": "classeur.xlsx",
"sheet": "Budget", "rows": [["VRAI", "01/02/2026", 12.5]],
}, confirm=True)
assert out.ok
wb = openpyxl.load_workbook(workbook)
ws = wb["Budget"]
assert ws["A4"].value is True
assert (ws["B4"].value.month, ws["B4"].value.day) == (2, 1)
assert ws["C4"].value == 12.5
wb.close()
def test_unknown_sheet_rejected(self, vault, workbook):
with pytest.raises(ToolError):
call_tool("append_xlsx_rows", _ctx(), {
"vault": "Vault", "path": "classeur.xlsx",
"sheet": "Nope", "rows": [["x"]],
})
def test_empty_rows_rejected(self, vault, workbook):
with pytest.raises(ToolError):
call_tool("append_xlsx_rows", _ctx(), {
"vault": "Vault", "path": "classeur.xlsx",
"sheet": "Budget", "rows": [],
})
+123
View File
@@ -0,0 +1,123 @@
"""Tests de non-régression — verrous des stores JSON (ROADMAP #85 T10a).
Sans verrou, les read-modify-write concurrents (créations de partages,
révocations de jetons) perdent des mises à jour : deux threads lisent le
même état, chacun écrit le sien, la première écriture est écrasée. Ces
tests martèlent les stores depuis plusieurs threads et exigent un compte
exact à la fin (aucune mise à jour perdue).
"""
from __future__ import annotations
import threading
N_THREADS = 8
N_OPS = 25
def test_concurrent_share_creations_lose_nothing(tmp_path, monkeypatch):
"""N threads × N partages → le store final contient tout."""
from backend import share as share_mod
monkeypatch.setattr(share_mod, "SHARES_FILE", tmp_path / "shares.json")
errors: list[BaseException] = []
def worker(n: int):
try:
for i in range(N_OPS):
share_mod.create_share("V", f"doc-{n}-{i}.md", "tester")
except BaseException as e: # pragma: no cover - diagnostic
errors.append(e)
threads = [threading.Thread(target=worker, args=(n,)) for n in range(N_THREADS)]
for t in threads:
t.start()
for t in threads:
t.join()
assert not errors
assert len(share_mod.list_shares()) == N_THREADS * N_OPS
def test_concurrent_share_access_and_revoke(tmp_path, monkeypatch):
"""Accès + révocations concurrents : compteurs et suppressions cohérents."""
from backend import share as share_mod
monkeypatch.setattr(share_mod, "SHARES_FILE", tmp_path / "shares.json")
tokens = [share_mod.create_share("V", f"doc-{i}.md", "tester")["token"] for i in range(20)]
def worker(n: int):
share_mod.record_access(tokens[2 * n])
share_mod.record_access(tokens[2 * n + 1])
share_mod.revoke_share(tokens[2 * n])
threads = [threading.Thread(target=worker, args=(n,)) for n in range(10)]
for t in threads:
t.start()
for t in threads:
t.join()
# Les 10 tokens pairs ont été révoqués ; les impairs subsistent avec
# leurs compteurs d'accès (aucune écriture perdue).
remaining = {s["token"] for s in share_mod.list_shares()}
assert len(remaining) == 10
assert all(share_mod.get_share_by_token(t)["access_count"] >= 1 for t in remaining)
def test_concurrent_token_revokes_lose_nothing(tmp_path, monkeypatch):
"""N threads × N révocations → toutes les JTIs sont persistées."""
import json
from backend.auth import jwt_handler
revoked_file = tmp_path / "revoked_tokens.json"
monkeypatch.setattr(jwt_handler, "REVOKED_TOKENS_FILE", revoked_file)
monkeypatch.setattr(jwt_handler, "_revoked_map", {})
monkeypatch.setattr(jwt_handler, "_revoked_loaded", True)
errors: list[BaseException] = []
def worker(n: int):
try:
for i in range(N_OPS):
jwt_handler.revoke_token(f"jti-{n}-{i}")
except BaseException as e: # pragma: no cover - diagnostic
errors.append(e)
threads = [threading.Thread(target=worker, args=(n,)) for n in range(N_THREADS)]
for t in threads:
t.start()
for t in threads:
t.join()
assert not errors
assert len(jwt_handler._revoked_map) == N_THREADS * N_OPS
# Le fichier reflète l'état mémoire (aucune écriture perdue).
on_disk = json.loads(revoked_file.read_text(encoding="utf-8"))
assert len(on_disk) == N_THREADS * N_OPS
assert jwt_handler.is_token_revoked("jti-0-0")
assert not jwt_handler.is_token_revoked("jti-absent")
def test_concurrent_webhook_and_tool_key_writes(tmp_path, monkeypatch):
"""Créations de webhooks + clés d'outils concurrentes : rien de perdu."""
from backend import webhooks as wh_mod
from backend.tools import secrets as sec_mod
monkeypatch.setattr(wh_mod, "WEBHOOKS_FILE", tmp_path / "webhooks.json")
monkeypatch.setattr(wh_mod, "WEBHOOK_SECRETS_FILE", tmp_path / "webhook_secrets.json")
monkeypatch.setattr(sec_mod, "_keys_file", lambda: tmp_path / "api_keys.json")
def worker(n: int):
for i in range(N_OPS):
wh_mod.create_webhook(f"hook-{n}-{i}", "https://example.com/hook", ["file_created"])
sec_mod.set_tool_key("OBSIGATE_GITHUB_TOKEN", f"tok-{n}-{i}")
threads = [threading.Thread(target=worker, args=(n,)) for n in range(N_THREADS)]
for t in threads:
t.start()
for t in threads:
t.join()
assert len(wh_mod.get_webhooks()) == N_THREADS * N_OPS
+188
View File
@@ -0,0 +1,188 @@
"""Structural mutations of an .xlsx workbook (#153 A14): service + endpoint.
Covers sheet add/rename/delete/duplicate and row/col insert/delete, the
atomicity of the batch (one locked rewrite) and the shared P0 guards
(lossy 409 gate, backup, path safety).
"""
from __future__ import annotations
import zipfile
from pathlib import Path
import pytest
openpyxl = pytest.importorskip("openpyxl")
VAULT = "TestVault"
@pytest.fixture
def book(test_vault_dir: str) -> str:
path = Path(test_vault_dir) / "struct.xlsx"
wb = openpyxl.Workbook()
ws = wb.active
ws.title = "Data"
ws.append(["Nom", "Valeur"])
ws.append(["a", 1])
ws.append(["b", 2])
wb.create_sheet("Vide")
wb.save(path)
return str(path)
def _put(client, path="struct.xlsx", actions=None, **extra):
return client.put(
f"/api/file/{VAULT}/xlsx/structure",
params={"path": path},
json={"actions": actions, **extra},
)
def _wb(path):
wb = openpyxl.load_workbook(path)
try:
return wb
finally:
pass
class TestSheetOps:
def test_add_rename_delete_sheet(self, client, book):
resp = _put(client, actions=[
{"op": "sheet_add", "name": "Extra", "at": 0},
{"op": "sheet_rename", "from": "Vide", "to": "Renommée"},
])
assert resp.status_code == 200
wb = openpyxl.load_workbook(book)
assert wb.sheetnames[0] == "Extra" # inserted at position 0
assert "Renommée" in wb.sheetnames and "Vide" not in wb.sheetnames
wb.close()
resp = _put(client, actions=[{"op": "sheet_delete", "name": "Extra"}])
assert resp.status_code == 200
wb = openpyxl.load_workbook(book)
assert "Extra" not in wb.sheetnames
wb.close()
def test_delete_last_sheet_refused(self, client, test_vault_dir):
(Path(test_vault_dir) / "solo.xlsx").write_bytes(book_bytes("Solo"))
resp = _put(client, path="solo.xlsx", actions=[
{"op": "sheet_delete", "name": "Solo"},
])
assert resp.status_code == 400
def test_duplicate_copies_values(self, client, book):
resp = _put(client, actions=[
{"op": "sheet_duplicate", "name": "Data", "as": "Data copie"},
])
assert resp.status_code == 200
wb = openpyxl.load_workbook(book)
assert wb["Data copie"]["A1"].value == "Nom"
assert wb["Data copie"]["B3"].value == 2
wb.close()
def book_bytes(sheet_name: str) -> bytes:
import io
wb = openpyxl.Workbook()
wb.active.title = sheet_name
buf = io.BytesIO()
wb.save(buf)
return buf.getvalue()
class TestRowColOps:
def test_row_insert_shifts_and_delete_removes(self, client, book):
resp = _put(client, actions=[
{"op": "row_insert", "sheet": "Data", "at": 2, "count": 1},
])
assert resp.status_code == 200
wb = openpyxl.load_workbook(book)
ws = wb["Data"]
assert ws["A2"].value is None # the new blank row
assert ws["A3"].value == "a" # shifted down
wb.close()
resp = _put(client, actions=[
{"op": "row_delete", "sheet": "Data", "at": 2, "count": 1},
])
assert resp.status_code == 200
wb = openpyxl.load_workbook(book)
assert wb["Data"]["A2"].value == "a"
wb.close()
def test_col_insert_and_delete(self, client, book):
assert _put(client, actions=[
{"op": "col_insert", "sheet": "Data", "at": 2},
]).status_code == 200
wb = openpyxl.load_workbook(book)
assert wb["Data"]["B1"].value is None
assert wb["Data"]["C1"].value == "Valeur"
wb.close()
assert _put(client, actions=[
{"op": "col_delete", "sheet": "Data", "at": 2},
]).status_code == 200
wb = openpyxl.load_workbook(book)
assert wb["Data"]["B1"].value == "Valeur"
wb.close()
class TestGuards:
def test_unknown_sheet_is_400(self, client, book):
resp = _put(client, actions=[{"op": "row_insert", "sheet": "Nope", "at": 1}])
assert resp.status_code == 400
def test_unknown_op_is_400(self, client, book):
resp = _put(client, actions=[{"op": "sheet_explode", "name": "X"}])
assert resp.status_code == 400
def test_bad_position_is_400(self, client, book):
resp = _put(client, actions=[
{"op": "row_insert", "sheet": "Data", "at": "deux"},
])
assert resp.status_code == 400
def test_empty_actions_is_400(self, client, book):
assert _put(client, actions=[]).status_code == 400
def test_lossy_workbook_refused_without_force(self, client, test_vault_dir):
"""Same 409 gate as the cell edits (A1)."""
path = Path(test_vault_dir) / "lossy-struct.xlsx"
wb = openpyxl.Workbook()
wb.active.title = "S"
wb["S"]["A1"] = "=A2" # no cached value -> add one via the raw XML
wb.save(path)
with zipfile.ZipFile(path) as zf:
items = {n: zf.read(n) for n in zf.namelist()}
sheet = next(n for n in items if n.startswith("xl/worksheets/sheet"))
items[sheet] = items[sheet].decode("utf-8").replace(
"<f>A2</f>", "<f>A2</f><v>7</v>"
).encode("utf-8")
with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as zf:
for name, blob in items.items():
zf.writestr(name, blob)
resp = _put(client, path="lossy-struct.xlsx", actions=[
{"op": "sheet_add", "name": "X"},
])
assert resp.status_code == 409
assert resp.json()["code"] == "xlsx_lossy_content"
resp = _put(client, path="lossy-struct.xlsx", actions=[
{"op": "sheet_add", "name": "X"},
], force=True)
assert resp.status_code == 200
def test_atomicity_one_bad_action_writes_nothing(self, client, book):
"""A batch with a valid action followed by a bad one writes nothing."""
resp = _put(client, actions=[
{"op": "sheet_add", "name": "Temp"},
{"op": "sheet_delete", "name": "Inexistante"},
])
assert resp.status_code == 400
wb = openpyxl.load_workbook(book)
assert "Temp" not in wb.sheetnames
wb.close()
+99
View File
@@ -0,0 +1,99 @@
"""Style metadata in the .xlsx viewer (#153 A15): bold, colors, number formats,
merged ranges and freeze panes — plus the endpoint wiring."""
from __future__ import annotations
from pathlib import Path
import pytest
openpyxl = pytest.importorskip("openpyxl")
VAULT = "TestVault"
@pytest.fixture
def styled_book(test_vault_dir: str) -> str:
"""A workbook with a bold red-font header, a filled cell, a merge and a freeze."""
from openpyxl.styles import Font, PatternFill
path = Path(test_vault_dir) / "styles.xlsx"
wb = openpyxl.Workbook()
ws = wb.active
ws.title = "Data"
ws["A1"] = "En-tête"
ws["A1"].font = Font(bold=True, italic=True, color="FFCC0000")
ws["B1"] = "Total"
ws["B1"].fill = PatternFill("solid", fgColor="FFFFEE99")
ws["B2"] = 1234.5
ws["B2"].number_format = "#,##0.00"
ws["C3"] = "fusionnée"
ws.merge_cells("C3:D3")
ws["C3"].value = "fusionnée"
ws.freeze_panes = "A2"
wb.save(path)
return str(path)
class TestStyleReading:
def test_bold_italic_and_font_color(self, styled_book):
from backend.xlsx_reader import read_sheet_styles
styles = read_sheet_styles(Path(styled_book), "Data")
a1 = styles["A1"]
assert "font-weight:600" in a1["style"]
assert "font-style:italic" in a1["style"]
assert a1["style"].startswith("color:#cc0000")
def test_background_fill_is_read(self, styled_book):
from backend.xlsx_reader import read_sheet_styles
styles = read_sheet_styles(Path(styled_book), "Data")
assert "background:#ffee99" in styles["B1"]["style"]
def test_number_format_is_signalled(self, styled_book):
from backend.xlsx_reader import read_sheet_styles
styles = read_sheet_styles(Path(styled_book), "Data")
assert "font-family" in styles["B2"]["style"]
def test_plain_cell_has_no_entry(self, styled_book):
from backend.xlsx_reader import read_sheet_styles
styles = read_sheet_styles(Path(styled_book), "Data")
assert "A2" not in styles # untouched cell
def test_merges_and_freeze(self, styled_book):
from backend.xlsx_reader import read_sheet_freeze, read_sheet_merges
assert read_sheet_merges(Path(styled_book), "Data") == ["C3:D3"]
assert read_sheet_freeze(Path(styled_book), "Data") == "A2"
def test_unknown_sheet_yields_empty(self, styled_book):
from backend.xlsx_reader import read_sheet_styles
assert read_sheet_styles(Path(styled_book), "Nope") == {}
def test_broken_file_yields_empty(self, test_vault_dir):
from backend.xlsx_reader import read_sheet_styles
bad = Path(test_vault_dir) / "broken-styles.xlsx"
bad.write_bytes(b"not a zip")
assert read_sheet_styles(bad, "Data") == {}
class TestEndpointWiring:
def test_read_response_carries_styles_merges_freeze(self, client, styled_book):
resp = client.get(f"/api/file/{VAULT}", params={"path": "styles.xlsx"})
assert resp.status_code == 200
sheet = resp.json()["xlsx_sheets"][0]
assert sheet["styles"]["A1"].startswith("color:#cc0000")
assert sheet["aligns"] == {} or isinstance(sheet["aligns"], dict)
assert sheet["merges"] == ["C3:D3"]
assert sheet["freeze"] == "A2"
def test_rendered_html_carries_the_inline_style(self, client, styled_book):
resp = client.get(f"/api/file/{VAULT}", params={"path": "styles.xlsx"})
html = resp.json()["xlsx_sheets"][0]["html"]
assert 'data-cell="A1" style="' in html
assert "font-weight:600" in html
+571 -3
View File
@@ -1,7 +1,13 @@
"""Display / edit / download for .xlsx files (viewer + PUT xlsx/save)."""
"""Display / edit / download for .xlsx files (viewer + PUT xlsx/save).
Covers #152 (affichage / édition) and #153 P0 : A1 alerte de fidélité avant
écriture, A2 écriture atomique, A3 verrou par fichier, A4 neutralisation de
l'injection de formule.
"""
from __future__ import annotations
import zipfile
from pathlib import Path
import pytest
@@ -29,6 +35,43 @@ def xlsx_file(test_vault_dir: str) -> str:
return str(path)
def _add_lossy_parts(path: Path, parts: dict[str, bytes]) -> None:
"""Re-pack *path* with extra OPC parts openpyxl cannot write back."""
with zipfile.ZipFile(path) as zf:
items = {n: zf.read(n) for n in zf.namelist()}
items.update(parts)
with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as zf:
for name, blob in items.items():
zf.writestr(name, blob)
@pytest.fixture
def lossy_xlsx(test_vault_dir: str) -> str:
"""Workbook with a slicer + a formula carrying its cached result."""
from openpyxl import Workbook
path = Path(test_vault_dir) / "risky.xlsx"
wb = Workbook()
ws = wb.active
ws.title = "Data"
ws["A1"] = 3
ws["A2"] = "=A1*3"
wb.save(path)
# <f>…</f><v>…</v> : openpyxl keeps the formula, drops the cached result.
with zipfile.ZipFile(path) as zf:
items = {n: zf.read(n) for n in zf.namelist()}
sheet = next(n for n in items if n.startswith("xl/worksheets/sheet"))
xml = items[sheet].decode("utf-8").replace(
"<f>A1*3</f>", "<f>A1*3</f><v>9</v>"
)
items[sheet] = xml.encode("utf-8")
items["xl/slicers/slicer1.xml"] = b"<slicer/>"
with zipfile.ZipFile(path, "w", zipfile.ZIP_DEFLATED) as zf:
for name, blob in items.items():
zf.writestr(name, blob)
return str(path)
# ── Display ───────────────────────────────────────────────────────────────
@@ -63,16 +106,208 @@ class TestXlsxIndexing:
assert ".xlsx" in SUPPORTED_EXTENSIONS
def test_xlsx_indexed_metadata_only(self, test_vault_dir, xlsx_file):
def test_xlsx_indexes_sheet_names_and_headers(self, test_vault_dir, xlsx_file):
"""#153 A5 — a workbook is searchable by its cell values."""
from backend.indexer import _index_single_file_sync
info = _index_single_file_sync(VAULT, test_vault_dir, xlsx_file)
assert info is not None
assert info["extension"] == ".xlsx"
assert info["content"] == "" # binary: never read into TF-IDF
# Sheet names + header rows reach TF-IDF (was metadata-only before A5).
assert "Budget" in info["content"]
assert "Poste" in info["content"]
assert info["content_preview"]
assert info["title"] # filename-derived title
class TestXlsxCachedValues:
"""#153 A12 — show what Excel last computed next to each formula."""
def test_cached_result_is_shown_beside_the_formula(self, client, lossy_xlsx):
from backend.xlsx_reader import render_sheets
html = render_sheets(Path(lossy_xlsx))[0]["html"]
# A2 is "=A1*3" with <v>9</v> in the fixture.
assert 'data-cell="A2"' in html
assert "=A1*3" in html
assert "xlsx-cached" in html
assert ">9<" in html # the cached result Excel computed
def test_no_shadow_when_no_formula_carries_a_result(self, client, xlsx_file):
from backend.xlsx_reader import render_sheets
html = render_sheets(Path(xlsx_file))[0]["html"]
assert "xlsx-cached" not in html # budget.xlsx has no <v> at all
def test_plain_cells_are_never_duplicated(self, client, lossy_xlsx):
from backend.xlsx_reader import render_sheets
html = render_sheets(Path(lossy_xlsx))[0]["html"]
# A1 is the literal 3: the two reads agree, so only one value shows.
assert 'data-cell="A1"' in html
assert html.count("xlsx-cached") == 1 # only the formula cell
class TestXlsxValueCoercion:
"""#153 A10 — a typed value comes back with the type Excel would infer."""
def _write(self, client, xlsx_file, ref, value):
return client.put(
f"/api/file/{VAULT}/xlsx/save",
params={"path": "budget.xlsx"},
json={"sheet": "Budget", "cells": {ref: value}, "force": True},
)
def test_number_and_bool_are_stored_as_typed(self, client, xlsx_file):
resp = self._write(
client, xlsx_file, "D1", "42"
)
assert resp.status_code == 200
resp = self._write(client, xlsx_file, "D2", "VRAI")
assert resp.status_code == 200
resp = self._write(client, xlsx_file, "D3", "12/03/2026")
assert resp.status_code == 200
from openpyxl import load_workbook
wb = load_workbook(xlsx_file)
ws = wb["Budget"]
assert ws["D1"].value == 42 and isinstance(ws["D1"].value, int)
assert ws["D2"].value is True
assert ws["D3"].value.year == 2026 and ws["D3"].value.month == 3
assert ws["D3"].value.day == 12 # FR day-first, not 3 December
wb.close()
def test_day_first_date_is_not_read_as_us(self, client, xlsx_file):
"""'01/02/2026' is 1 February in French, not 2 January."""
assert self._write(client, xlsx_file, "E1", "01/02/2026").status_code == 200
from openpyxl import load_workbook
wb = load_workbook(xlsx_file)
d = wb["Budget"]["E1"].value
wb.close()
assert (d.month, d.day) == (2, 1)
def test_ambiguous_text_is_left_alone(self, client, xlsx_file):
"""A version string or a partial date stays text, never a date."""
assert self._write(client, xlsx_file, "F1", "3.14.2").status_code == 200
assert self._write(client, xlsx_file, "F2", "Ref 12/34").status_code == 200
assert self._write(client, xlsx_file, "F3", "12/2026").status_code == 200
from openpyxl import load_workbook
wb = load_workbook(xlsx_file)
ws = wb["Budget"]
assert ws["F1"].value == "3.14.2"
assert ws["F2"].value == "Ref 12/34"
assert ws["F3"].value == "12/2026"
wb.close()
def test_plain_integer_text_becomes_a_number(self, client, xlsx_file):
"""Typing a bare number yields a number, as it did before #153 A10."""
assert self._write(client, xlsx_file, "H1", "75001").status_code == 200
from openpyxl import load_workbook
wb = load_workbook(xlsx_file)
assert wb["Budget"]["H1"].value == 75001
wb.close()
def test_formula_looking_date_stays_text(self, client, xlsx_file):
"""A formula is never mistaken for a date (BUG-088 must not regress)."""
assert self._write(client, xlsx_file, "G1", "=12/03/2026").status_code == 200
from openpyxl import load_workbook
wb = load_workbook(xlsx_file)
c = wb["Budget"]["G1"]
wb.close()
assert c.data_type == "s"
assert c.value == "=12/03/2026"
class TestXlsxSearchable:
"""#153 A5 — a keyword living in a CELL must make the file findable."""
def test_search_finds_a_word_stored_in_a_cell(self, test_vault_dir):
"""A word typed in a CELL must make the workbook findable.
Deliberately hermetic: it drives the indexer and the inverted index
directly instead of going through the HTTP reload, because both are
process-wide singletons that other test modules mutate (some reload the
module outright), which would make this test order-dependent.
"""
from openpyxl import Workbook
import backend.indexer as indexer
import backend.search as search_mod
path = Path(test_vault_dir) / "fournisseurs.xlsx"
wb = Workbook()
ws = wb.active
ws.title = "Contacts"
ws["A1"] = "Fournisseur"
ws["A2"] = "Menuiserie Beaulieu"
wb.save(path)
# Index exactly this vault, from scratch.
indexer.index[VAULT] = {
"files": [indexer._index_single_file_sync(VAULT, test_vault_dir, str(path))],
"tags": {},
"paths": [],
"config": {},
}
entries = [f for f in indexer.index[VAULT]["files"] if f]
assert entries, "le tableur n'a pas ete indexe"
assert "Menuiserie" in entries[0]["content"], (
f"contenu indexe : {entries[0]['content'][:80]!r}"
)
search_mod.init_inverted_index()
hits = [r["path"] for r in search_mod.search("Menuiserie", "all")]
assert "fournisseurs.xlsx" in hits
def test_indexable_text_is_capped(self, tmp_path):
"""A data dump must not flood the index."""
from openpyxl import Workbook
from backend.xlsx_reader import MAX_INDEX_CHARS, extract_indexable_text
path = tmp_path / "huge.xlsx"
wb = Workbook()
ws = wb.active
ws.title = "Big"
for r in range(1, 400):
ws.cell(row=r, column=1, value=f"ligne {r} " + "x" * 60)
wb.save(path)
text = extract_indexable_text(path)
assert len(text) <= MAX_INDEX_CHARS
assert "Big" in text # the sheet name survives
def test_corrupt_workbook_indexes_as_empty_not_a_crash(self, tmp_path):
from backend.xlsx_reader import extract_indexable_text
path = tmp_path / "broken.xlsx"
path.write_bytes(b"not a zip at all")
assert extract_indexable_text(path) == ""
def test_blank_rows_are_skipped(self, tmp_path):
from openpyxl import Workbook
from backend.xlsx_reader import extract_indexable_text
path = tmp_path / "sparse.xlsx"
wb = Workbook()
ws = wb.active
ws.title = "S"
ws["A1"] = "Alpha"
ws["A50"] = "Omega" # beyond the indexed prefix -> ignored on purpose
wb.save(path)
text = extract_indexable_text(path)
assert "Alpha" in text
assert "Omega" not in text
assert "\t\t" not in text # no run of empty columns
# ── Edit ──────────────────────────────────────────────────────────────────
@@ -150,3 +385,336 @@ class TestXlsxSave:
def test_missing_sheet_field_400(self, client, xlsx_file):
resp = self._save(client, {"cells": {"A1": "x"}})
assert resp.status_code == 400
def test_non_boolean_flag_400(self, client, xlsx_file):
for flag in ("force", "allow_formula"):
resp = self._save(client, {"sheet": "Budget", "cells": {"A1": "x"}, flag: "yes"})
assert resp.status_code == 400, flag
# ── #153 A1 — lossy-write guard ──────────────────────────────────────────
class TestXlsxLossyGuard:
def _save(self, client, body, path):
return client.put(
f"/api/file/{VAULT}/xlsx/save", params={"path": path}, json=body,
)
def test_read_reports_lossy_features(self, client, lossy_xlsx):
data = client.get(f"/api/file/{VAULT}", params={"path": "risky.xlsx"}).json()
assert data["is_xlsx"] is True
assert "slicers" in data["xlsx_lossy_features"]
assert "cached_values" in data["xlsx_lossy_features"]
def test_read_reports_nothing_for_a_plain_workbook(self, client, xlsx_file):
data = client.get(f"/api/file/{VAULT}", params={"path": "budget.xlsx"}).json()
# B2 holds "=B1*2" but openpyxl wrote no cached <v> for it.
assert data["xlsx_lossy_features"] == []
def test_save_refuses_without_force(self, client, lossy_xlsx):
resp = self._save(client, {"sheet": "Data", "cells": {"B1": "hello"}}, "risky.xlsx")
assert resp.status_code == 409
body = resp.json()
assert body["code"] == "xlsx_lossy_content"
assert "slicers" in body["details"]["features"]
def test_refused_save_leaves_the_file_untouched(self, client, lossy_xlsx):
before = Path(lossy_xlsx).read_bytes()
self._save(client, {"sheet": "Data", "cells": {"B1": "hello"}}, "risky.xlsx")
assert Path(lossy_xlsx).read_bytes() == before
def test_save_with_force_succeeds(self, client, lossy_xlsx):
resp = self._save(
client,
{"sheet": "Data", "cells": {"B1": "hello"}, "force": True},
"risky.xlsx",
)
assert resp.status_code == 200
assert openpyxl.load_workbook(lossy_xlsx)["Data"]["B1"].value == "hello"
def test_inspect_flags_every_known_family(self, lossy_xlsx):
from backend.xlsx_reader import LOSSY_PARTS, inspect_workbook
for key, prefixes in LOSSY_PARTS.items():
_add_lossy_parts(
Path(lossy_xlsx),
{f"{prefixes[0]}probe.xml": b"<x/>" for _ in [0]},
)
assert key in inspect_workbook(Path(lossy_xlsx)), key
def test_inspect_is_quiet_on_a_corrupt_archive(self, test_vault_dir):
from backend.xlsx_reader import inspect_workbook
bad = Path(test_vault_dir) / "broken.xlsx"
bad.write_bytes(b"not a zip at all")
assert inspect_workbook(bad) == []
# ── #153 A2 — atomic write ───────────────────────────────────────────────
class TestXlsxAtomicWrite:
def test_failed_save_keeps_the_original(self, client, xlsx_file, monkeypatch):
from openpyxl.workbook.workbook import Workbook
before = Path(xlsx_file).read_bytes()
def boom(self, *args, **kwargs):
raise OSError("disk full")
monkeypatch.setattr(Workbook, "save", boom)
# TestClient re-raises the server exception (in production: 500).
with pytest.raises(OSError):
client.put(
f"/api/file/{VAULT}/xlsx/save",
params={"path": "budget.xlsx"},
json={"sheet": "Budget", "cells": {"A1": "perdu"}},
)
# The workbook on disk is byte-identical : the write never reached it.
assert Path(xlsx_file).read_bytes() == before
# No temporary file left behind in the vault.
assert list(Path(xlsx_file).parent.glob("*.tmp")) == []
def test_no_tmp_left_after_a_successful_save(self, client, xlsx_file):
client.put(
f"/api/file/{VAULT}/xlsx/save",
params={"path": "budget.xlsx"},
json={"sheet": "Budget", "cells": {"A1": "ok"}},
)
assert list(Path(xlsx_file).parent.glob("*.tmp")) == []
# ── #153 A3 — per-file write lock ────────────────────────────────────────
class TestXlsxWriteLock:
def test_concurrent_write_returns_409(self, client, xlsx_file):
from backend.services import mutations
key = str(Path(xlsx_file).resolve())
with mutations._xlsx_write_lock(key):
resp = client.put(
f"/api/file/{VAULT}/xlsx/save",
params={"path": "budget.xlsx"},
json={"sheet": "Budget", "cells": {"A1": "concurrent"}},
)
assert resp.status_code == 409
assert resp.json()["code"] == "conflict"
# The blocked call wrote nothing.
assert openpyxl.load_workbook(xlsx_file)["Budget"]["A1"].value == "Poste"
def test_lock_is_released_after_a_normal_save(self, client, xlsx_file):
from backend.services import mutations
key = str(Path(xlsx_file).resolve())
client.put(
f"/api/file/{VAULT}/xlsx/save",
params={"path": "budget.xlsx"},
json={"sheet": "Budget", "cells": {"A1": "premier"}},
)
# The lock must be free again once the request returned.
with mutations._xlsx_write_lock(key):
pass
assert openpyxl.load_workbook(xlsx_file)["Budget"]["A1"].value == "premier"
@pytest.fixture
def wide_xlsx(test_vault_dir: str) -> str:
"""Workbook whose sheet exceeds BOTH render caps (501 rows x 45 cols).
Sparse on purpose: a cell in A501 and one in AS1 are enough for openpyxl
to declare those dimensions, without writing 20 000 cells to disk.
"""
from openpyxl import Workbook
path = Path(test_vault_dir) / "grand.xlsx"
wb = Workbook()
ws = wb.active
ws.title = "Data"
ws["A1"] = "tête"
ws["A501"] = "dernière ligne"
ws["AS1"] = "colonne 45"
wb.save(path)
return str(path)
@pytest.fixture
def edge_xlsx(test_vault_dir: str) -> str:
"""Sheet exactly on the caps (500 rows x 40 cols) — must NOT be truncated."""
from openpyxl import Workbook
path = Path(test_vault_dir) / "limite.xlsx"
wb = Workbook()
ws = wb.active
ws.title = "Data"
ws["A1"] = "bord"
ws["A500"] = "ligne 500"
ws["AN1"] = "colonne 40"
wb.save(path)
return str(path)
# ── #153 A8 — silent truncation made visible ─────────────────────────────
class TestXlsxTruncationNotice:
"""A sheet bigger than the caps must SAY so instead of looking complete."""
def test_render_reports_the_real_dimensions(self, client, wide_xlsx):
resp = client.get(f"/api/file/{VAULT}", params={"path": "grand.xlsx"})
sheet = resp.json()["xlsx_sheets"][0]
assert (sheet["total_rows"], sheet["total_cols"]) == (501, 45)
assert sheet["truncated"] is True
# The caps are the coverage the banner announces — `rows`/`cols` are
# post-trim and would understate it on a sparse sheet.
assert (sheet["max_rows"], sheet["max_cols"]) == (500, 40)
assert (sheet["rows"], sheet["cols"]) == (1, 1) # only 3 filled cells
def test_a_sheet_on_the_caps_is_not_flagged(self, client, edge_xlsx):
"""Boundary: 500x40 is exactly what the renderer supports."""
resp = client.get(f"/api/file/{VAULT}", params={"path": "limite.xlsx"})
sheet = resp.json()["xlsx_sheets"][0]
assert sheet["truncated"] is False
assert (sheet["total_rows"], sheet["total_cols"]) == (500, 40)
def test_a_normal_sheet_is_not_flagged(self, client, xlsx_file):
resp = client.get(f"/api/file/{VAULT}", params={"path": "budget.xlsx"})
assert all(not s["truncated"] for s in resp.json()["xlsx_sheets"])
def test_blank_tail_is_not_reported_as_truncation(self, client, test_vault_dir):
"""A sheet with empty rows below its data fits in the caps."""
from openpyxl import Workbook
path = Path(test_vault_dir) / "blanc.xlsx"
wb = Workbook()
ws = wb.active
ws.title = "Data"
ws["A1"] = "seule ligne"
ws["A300"] = None # formatted-but-empty row inside the caps
wb.save(path)
resp = client.get(f"/api/file/{VAULT}", params={"path": "blanc.xlsx"})
sheet = resp.json()["xlsx_sheets"][0]
assert sheet["truncated"] is False
assert sheet["rows"] == 1 # trailing blanks dropped by _trim
# ── #153 A9 — lazy per-sheet loading ─────────────────────────────────────
class TestXlsxSheetWindow:
"""GET /api/file/{vault}/xlsx/sheet — one window of one sheet."""
def _get(self, client, path="budget.xlsx", **params):
return client.get(
f"/api/file/{VAULT}/xlsx/sheet", params={"path": path, **params}
)
def test_window_returns_rows_and_totals(self, client, xlsx_file):
resp = self._get(client, sheet="Budget", offset=0, limit=10)
assert resp.status_code == 200
data = resp.json()
assert data["sheet"] == "Budget"
assert (data["offset"], data["limit"]) == (0, 10)
assert data["rows"] == 2 and data["cols"] == 2
assert data["total_rows"] == 2 and data["truncated"] is False
assert (data["max_rows"], data["max_cols"]) == (500, 40)
assert data["has_more"] is False
assert 'data-cell="A1"' in data["html"]
def test_window_keeps_the_real_a1_coordinates(self, client, xlsx_file):
"""A window must be indistinguishable from a full render: the A1
references and the row numbers have to be the sheet's, not the
window's, or an edit would land on the wrong cell. The CONTENT matters
as much as the label — row 2 of "Budget" is "Total", not "Poste"."""
data = self._get(client, sheet="Budget", offset=1, limit=1).json()
assert data["rows"] == 1
assert 'data-cell="A2"' in data["html"]
assert 'data-cell="A1"' not in data["html"]
assert "<th class=\"xlsx-rownum\">2</th>" in data["html"]
assert "Total" in data["html"] and "Poste" not in data["html"]
def test_window_offsets_walk_the_whole_sheet(self, client, wide_xlsx):
first = self._get(client, path="grand.xlsx", sheet="Data", offset=0, limit=10).json()
last = self._get(client, path="grand.xlsx", sheet="Data", offset=500, limit=10).json()
assert first["truncated"] is True and first["has_more"] is True
assert 'data-cell="A501"' in last["html"] # the row the caps used to hide
assert "dernière ligne" in last["html"]
assert "dernière ligne" not in first["html"]
assert last["rows"] == 1 and last["has_more"] is False
def test_offset_past_the_end_is_empty_not_an_error(self, client, xlsx_file):
resp = self._get(client, sheet="Budget", offset=9999, limit=10)
assert resp.status_code == 200
data = resp.json()
assert data["rows"] == 0
assert data["has_more"] is False
def test_cached_result_survives_the_window(self, client, lossy_xlsx):
"""A12 must not be lost on the lazy path."""
data = self._get(client, path="risky.xlsx", sheet="Data", offset=0, limit=10).json()
assert "xlsx-cached" in data["html"]
def test_unknown_sheet_is_404(self, client, xlsx_file):
resp = self._get(client, sheet="Nope")
assert resp.status_code == 404
assert "Feuille introuvable" in resp.json()["detail"]
def test_missing_file_is_404(self, client, test_vault_dir):
assert self._get(client, path="absent.xlsx", sheet="Budget").status_code == 404
def test_non_xlsx_file_is_415(self, client, test_vault_dir):
(Path(test_vault_dir) / "note.md").write_text("# hi", encoding="utf-8")
resp = self._get(client, path="note.md", sheet="Budget")
assert resp.status_code == 415
def test_limit_above_the_server_cap_is_rejected(self, client, xlsx_file):
"""The cap is a contract, not a silent truncation of the request."""
assert self._get(client, sheet="Budget", limit=100_000).status_code == 422
def test_reader_clamps_a_hostile_limit(self, xlsx_file):
"""Belt and braces: the reader caps too, whoever calls it."""
from backend.xlsx_reader import MAX_WINDOW_ROWS, read_sheet_window
window = read_sheet_window(Path(xlsx_file), "Budget", offset=0, limit=10**9)
assert window["limit"] == MAX_WINDOW_ROWS
def test_reader_rejects_a_negative_offset(self, xlsx_file):
from backend.xlsx_reader import read_sheet_window
window = read_sheet_window(Path(xlsx_file), "Budget", offset=-5, limit=10)
assert window["offset"] == 0
# ── #153 A4 — formula injection ──────────────────────────────────────────
class TestXlsxFormulaGuard:
def _save(self, client, cells, **extra):
return client.put(
f"/api/file/{VAULT}/xlsx/save",
params={"path": "budget.xlsx"},
json={"sheet": "Budget", "cells": cells, **extra},
)
def test_formula_like_value_is_stored_as_text(self, client, xlsx_file):
resp = self._save(client, {"A3": "=cmd|'/c calc'!A1"})
assert resp.status_code == 200
cell = openpyxl.load_workbook(xlsx_file)["Budget"]["A3"]
assert cell.value == "=cmd|'/c calc'!A1"
assert cell.data_type == "s" # pas de <f> dans l'archive
def test_at_prefix_is_stored_as_text(self, client, xlsx_file):
self._save(client, {"A4": "@SUM(A1:A2)"})
assert openpyxl.load_workbook(xlsx_file)["Budget"]["A4"].data_type == "s"
def test_allow_formula_keeps_a_real_formula(self, client, xlsx_file):
resp = self._save(client, {"A3": "=B1+5"}, allow_formula=True)
assert resp.status_code == 200
assert openpyxl.load_workbook(xlsx_file)["Budget"]["A3"].data_type == "f"
def test_numbers_are_unaffected(self, client, xlsx_file):
self._save(client, {"B3": "42", "B4": "-3.5"})
ws = openpyxl.load_workbook(xlsx_file)["Budget"]
assert ws["B3"].value == 42 and isinstance(ws["B3"].value, int)
assert ws["B4"].value == -3.5