fix(export): v4.7.2 contenu des documents absent des exports MD/HTML/PDF

Le service d'export ne lisait que les pages content_format='blocks'. Les docs stockees autrement sortaient avec le seul titre: - content_format='file' (.md/code uploades): content=JSON meta, le vrai texte est sur disque (/data/uploads/workspace_*) -> n'etait jamais lu. - content_format='markdown': HTML/PDF enveloppait chaque ligne en <p> (headings et listes aplatis). Resolution de la vraie source pour les 3 formats (app/services/export.py): - lit le fichier upload sur disque pour les pages file (repertoire via FLOWDECK_DATA_DIR, defaut /data), - rend les pages markdown / fichiers .md en blocs (headings, listes, code, quote, todo) pour un HTML/PDF riche, - fichiers texte non-markdown -> bloc de code, - binaires (PDF/images) ignores. 195/195 tests (5 nouveaux v4.7.2). Verifie en reel via HTTP sur README.md et l'arborescence Base de Connaissances (contenu complet dans les 3 formats).
2026-09-03 01:29:35 -04:00
parent 5390a6ceab
commit fa97f07ec8
6 changed files with 417 additions and 21 deletions
@@ -1,5 +1,18 @@
 # Changelog — FlowDeck

+## v4.7.2 (2026-09-03) — Fix export : le contenu des documents était absent
+
+### Fixed
+- **Export Markdown / HTML / PDF : seuls le nom de page et le titre étaient inclus, jamais le contenu.** Cause : le service d'export ne lisait que les pages en `content_format='blocks'`. Or les documents peuvent être stockés autrement :
+  - `content_format='file'` (fichiers `.md`/code uploadés) : le `content` de la page n'est que des métadonnées JSON — le vrai texte est un fichier sur disque (`/data/uploads/workspace_*`). L'export n'allait jamais le lire → uniquement le titre.
+  - `content_format='markdown'` : le HTML/PDF se contentait d'envelopper chaque ligne en `<p>` (titres et listes aplatis).
+- Le service résout désormais la vraie source pour les 3 formats : lit le fichier sur disque pour les pages `file`, rend les pages `markdown`/fichiers `.md` en blocs (headings, listes, code…), et traite les fichiers texte non-markdown comme bloc de code. Les fichiers binaires (PDF/images) sont laissés de côté (rien d'exploitable).
+- Chemin des uploads rendu configurable pour les tests via `FLOWDECK_DATA_DIR` (défaut `/data`, identique à `dashboard.py`).
+
+### Tests
+- **195 tests au total** (5 nouveaux v4.7.2) : page markdown → corps exporté ; page markdown → HTML avec headings/listes ; fichier `.md` uploadé → contenu réel dans Markdown/HTML/PDF ; fichier code → exporté ; fichier binaire → non déversé.
+- Vérifié en réel sur `/data` (page `README.md` et arborescence « Base de Connaissances » → fichiers `.md`) : le corps complet est maintenant présent dans les 3 formats.
+
 ## v4.7.1 (2026-09-03) — Fix UI éditeur : menus Share / More invisibles

 ### Fixed
@@ -291,6 +291,10 @@ Détails livrés :
 - [x] **Cache CSS** — query string app.css bumpé v4.7.1
 - [x] Vérifié Playwright headless + 190/190 tests

+### v4.7.2 — Fix Export : contenu absent ✅ (2026-09-03)
+- [x] **Export MD/HTML/PDF des documents** — le service ne lisait que les pages `blocks` ; les pages `file` (upload `.md`/code, contenu sur disque) et `markdown` sortaient avec le seul titre. Résolution de la vraie source pour les 3 formats + rendu HTML/PDF correct des headings/listes.
+- [x] 5 nouveaux tests → 195/195 ; vérifié sur les vraies données
+
 ### v4.8.0 — Collaboration
 - [ ] **Inline comments** — commentaires sur sélection de texte
 - [ ] **@mentions** — notifier un utilisateur → page/commentaire
@@ -443,4 +447,4 @@ v4.0.2 ✅    v4.1.0 ✅    v4.2.0 ✅    v4.3.0 ✅    v4.4.0 ✅    v4.5.0 ✅
 Quality     Data Sources  Templates +  10 Views     Tasks &      Sprints &     Content      Export       Collab +     Pro + Agent
 & Tests     & Linked DB   Dashboards   complets     Dependencies  My Tasks     Blocks       MD/PDF/HTML  Agent IA     (futur)

-*Dernière mise à jour: 2026-09-03 — v4.7.1 Fix UI Share/More/Export ✅*
+*Dernière mise à jour: 2026-09-03 — v4.7.2 Export fix contenu ✅*
@@ -1 +1 @@
-4.7.1
+4.7.2
@@ -39,13 +39,13 @@ async def lifespan(_app: FastAPI):
            (admin_hash,)
        )
        conn.commit()
-    logger.info("FlowDeck v4.7.1 started on port %d", settings.app_port)
+    logger.info("FlowDeck v4.7.2 started on port %d", settings.app_port)
    yield


 app = FastAPI(
    title="FlowDeck",
-    version="4.7.1",
+    version="4.7.2",
    docs_url="/docs" if settings.log_level == "DEBUG" else None,
    redoc_url=None,
    lifespan=lifespan,
@@ -1,16 +1,26 @@
-"""FlowDeck — Export service (v4.7.0).
+"""FlowDeck — Export service (v4.7.2).

 Four types of export, all generated server-side:
  - Markdown  (``page_to_markdown``): title + blocks + récursif sous-pages
  - HTML      (``page_to_standalone_html``): document autonome (styles inline)
  - PDF       (``page_to_pdf_bytes``): convertit un HTML print-friendly
  - Site      (``build_static_site``): site statique multi-pages (zip)
+
+Supports the three ways a page's content can be stored:
+  - content_format == "blocks"   -> JSON list of blocks in ``content``
+  - content_format == "markdown" -> raw Markdown in ``content``
+  - content_format == "file"     -> ``content`` is JSON metadata; the real text
+    lives in an uploaded file on disk (uploads/workspace_*). We read it back so
+    an exported document carries its actual content, not just its title.
 """
 from __future__ import annotations

 import io
 import json
+import os
+import re
 import zipfile
+from pathlib import Path
 from urllib.parse import quote

 from app.db import get_conn
@@ -54,6 +64,242 @@ def _block_text(b: dict) -> str:
    return _text(b.get("content"), escape=False)


+# ── Source resolution: read real textual content for ANY page type ──
+
+# Extensions whose content is plain text / code / markdown (textual exportable).
+_TEXTUAL_EXTS = {
+    "md", "markdown", "txt", "log", "text",
+    "py", "js", "ts", "jsx", "tsx", "html", "htm", "css", "json", "xml",
+    "yaml", "yml", "toml", "ini", "cfg", "conf", "env", "sh", "bash", "zsh",
+    "ps1", "bat", "cmd", "rb", "go", "rs", "java", "c", "cpp", "h", "hpp",
+    "php", "swift", "kt", "scala", "sql", "r", "vue", "svelte", "astro",
+    "properties", "gitignore", "dockerfile", "makefile",
+}
+_CODE_LANG = {
+    "py": "python", "js": "javascript", "ts": "typescript", "jsx": "javascript",
+    "tsx": "typescript", "html": "html", "htm": "html", "css": "css",
+    "json": "json", "xml": "xml", "yaml": "yaml", "yml": "yaml",
+    "toml": "toml", "ini": "ini", "cfg": "ini", "conf": "ini", "env": "ini",
+    "sh": "bash", "bash": "bash", "zsh": "bash", "ps1": "powershell",
+    "bat": "batch", "cmd": "batch", "rb": "ruby", "go": "go", "rs": "rust",
+    "java": "java", "c": "c", "cpp": "cpp", "h": "c", "hpp": "cpp",
+    "php": "php", "swift": "swift", "kt": "kotlin", "scala": "scala",
+    "sql": "sql", "r": "r", "vue": "html", "svelte": "html",
+    "astro": "html", "properties": "ini", "md": "markdown",
+    "markdown": "markdown", "txt": "plaintext", "log": "plaintext",
+    "text": "plaintext",
+}
+_MARKDOWN_MIMES = {"text/markdown", "text/x-markdown", "application/octet-stream"}
+
+
+def _data_root() -> Path:
+    """Directory that contains ``uploads/`` (mirrors dashboard.py /data)."""
+    return Path(os.environ.get("FLOWDECK_DATA_DIR", "/data"))
+
+
+def _file_meta(page: dict) -> dict:
+    try:
+        meta = json.loads(page.get("content") or "{}")
+        return meta if isinstance(meta, dict) else {}
+    except (json.JSONDecodeError, TypeError):
+        return {}
+
+
+def _file_text(page: dict) -> str | None:
+    """Return the textual content of an uploaded ``file`` page, or None.
+
+    Only reads plain-text / code / markdown files. Binary (PDF, images…)
+    returns None and is skipped by exporters (nothing meaningful to include).
+    """
+    if (page.get("content_format") or "") != "file":
+        return None
+    meta = _file_meta(page)
+    rel = (meta.get("file_path") or "").replace("\\", "/").strip()
+    if not rel or ".." in rel.replace("\\", "/").split("/") or not rel.startswith("uploads/"):
+        return None
+    name = (rel.rsplit("/", 1)[-1] or "").lower()
+    ext = name.rsplit(".", 1)[-1] if "." in name else ""
+    mime = (meta.get("mime_type") or "").lower()
+    if not (ext in _TEXTUAL_EXTS or mime.startswith("text/")):
+        return None
+    try:
+        full = (_data_root() / rel).resolve()
+        root = _data_root().resolve()
+        if root not in full.parents:
+            return None
+        return full.read_text(encoding="utf-8", errors="replace")
+    except (OSError, ValueError):
+        return None
+
+
+def _page_source(page: dict):
+    """Return (kind, payload) describing where the page's real content lives.
+
+    kind ∈ {"blocks", "md", "code"}:
+      - "blocks": payload is the block list (block editor pages)
+      - "md"    : payload is raw Markdown text
+      - "code"  : payload is (text, language)
+    An empty/unsupported page yields ("blocks", []).
+    """
+    fmt = (page.get("content_format") or "blocks")
+    content = page.get("content") or ""
+
+    if fmt == "blocks":
+        return "blocks", _blocks_of(page)
+
+    if fmt == "markdown":
+        if content.strip():
+            return "md", content
+        return "blocks", []
+
+    if fmt == "file":
+        text = _file_text(page)
+        if text is None:
+            return "blocks", []
+        meta = _file_meta(page)
+        name = (meta.get("file_path") or "").replace("\\", "/").rsplit("/", 1)[-1].lower()
+        ext = name.rsplit(".", 1)[-1] if "." in name else ""
+        mime = (meta.get("mime_type") or "").lower()
+        if ext in ("md", "markdown") or mime in _MARKDOWN_MIMES or mime.startswith("text/markdown"):
+            return "md", text
+        lang = _CODE_LANG.get(ext, "plaintext")
+        return "code", (text, lang)
+
+    # Unknown format (e.g. legacy) -> try to dump as raw text
+    if content.strip():
+        return "md", content
+    return "blocks", []
+
+
+# ── Markdown renderer (raw markdown → exportable fragments) ──
+
+def _md_to_blocks(md: str) -> list:
+    """Convert raw Markdown text into the same lightweight block list the
+    editor produces (headings, lists, to-do, quote, code, divider, paragraph).
+
+    Kept intentionally simple: inline formatting (bold/links) is preserved as
+    literal text, matching how the block editor treats imported .md files.
+    """
+    blocks: list = []
+    buf = md.replace("\r\n", "\n").replace("\r", "\n")
+    lines = buf.split("\n")
+    i = 0
+    n = len(lines)
+    para: list[str] = []
+
+    def flush_para():
+        nonlocal para
+        if para:
+            blocks.append({"type": "paragraph", "content": "\n".join(para).strip()})
+            para = []
+
+    while i < n:
+        line = lines[i].rstrip()
+        stripped = line.strip()
+        if not stripped:
+            flush_para()
+            i += 1
+            continue
+        if stripped.startswith("```") or stripped.startswith("~~~"):
+            flush_para()
+            fence = stripped[0:3]
+            lang = stripped[3:].strip()
+            i += 1
+            code: list[str] = []
+            while i < n and not lines[i].strip().startswith(fence):
+                code.append(lines[i])
+                i += 1
+            if i < n:
+                i += 1  # closing fence
+            blocks.append({"type": "code", "content": "\n".join(code), "language": lang})
+            continue
+        m = re.match(r"^(#{1,6})\s+(.*)$", stripped)
+        if m and line == stripped:  # ATX heading must be whole line
+            level = len(m.group(1))
+            flush_para()
+            blocks.append({"type": f"heading_{min(level, 4)}", "content": m.group(2).strip()})
+            i += 1
+            continue
+        if stripped == "---" or stripped == "***" or stripped == "___":
+            flush_para()
+            blocks.append({"type": "divider", "content": ""})
+            i += 1
+            continue
+        if re.match(r"^\s*[-*+]\s+", line):
+            flush_para()
+            while i < n:
+                s = lines[i].strip()
+                m2 = re.match(r"^[-*+]\s+(.*)$", s)
+                if not m2:
+                    break
+                blocks.append({"type": "bulleted_list", "content": m2.group(1).strip()})
+                i += 1
+            continue
+        if re.match(r"^\s*\d+[.)]\s+", line):
+            flush_para()
+            while i < n:
+                s = lines[i].strip()
+                m2 = re.match(r"^\d+[.)]\s+(.*)$", s)
+                if not m2:
+                    break
+                blocks.append({"type": "numbered_list", "content": m2.group(1).strip()})
+                i += 1
+            continue
+        mtodo = re.match(r"^\s*[-*+]\s+\[([ xX])\]\s+(.*)$", stripped)
+        if mtodo:
+            flush_para()
+            while i < n:
+                s = lines[i].strip()
+                m2 = re.match(r"^[-*+]\s+\[([ xX])\]\s+(.*)$", s)
+                if not m2:
+                    break
+                blocks.append({
+                    "type": "to_do",
+                    "content": m2.group(2).strip(),
+                    "checked": m2.group(1).lower() == "x",
+                })
+                i += 1
+            continue
+        mq = re.match(r"^>\s?(.*)$", stripped)
+        if mq and line == stripped:
+            flush_para()
+            while i < n:
+                s = lines[i].strip()
+                m2 = re.match(r"^>\s?(.*)$", s)
+                if not m2:
+                    break
+                para.append(m2.group(1))
+                i += 1
+            blocks.append({"type": "quote", "content": "\n".join(para)})
+            para = []
+            continue
+        para.append(stripped)
+        i += 1
+    flush_para()
+    return blocks
+
+
+def _page_blocks(page: dict) -> list:
+    """Blocks used for HTML/PDF rendering regardless of storage format."""
+    kind, payload = _page_source(page)
+    if kind == "blocks":
+        return payload
+    if kind == "code":
+        text, lang = payload
+        return [{"type": "code", "content": text, "language": lang}] if text else []
+    if kind == "md":
+        return _md_to_blocks(payload)
+    return []
+
+
+def _page_markdown_source(page: dict) -> str:
+    """Raw markdown when the page IS markdown-sourced, else empty string."""
+    kind, payload = _page_source(page)
+    if kind == "md":
+        return payload
+    return ""
+
+
 # ═══════════════ Markdown ═══════════════

 def blocks_to_markdown(blocks: list) -> str:
@@ -117,11 +363,11 @@ def _child_pages(page: dict) -> list:
 def page_to_markdown(page: dict, *, include_children: bool = True) -> str:
    """Markdown for a single page, with optional sub-pages appended."""
    parts = [f"# {_page_title(page)}", ""]
-    content_inner = _blocks_of(page)
-    if page.get("content_format") == "markdown" and (page.get("content") or "").strip():
-        parts.append((page["content"] or "").strip())
+    md_source = _page_markdown_source(page)
+    if md_source:
+        parts.append(md_source.strip())
    else:
-        md = blocks_to_markdown(content_inner)
+        md = blocks_to_markdown(_page_blocks(page))
        if md:
            parts.append(md)
    md = "\n\n".join(filter(None, parts)).rstrip()
@@ -253,11 +499,7 @@ def page_to_standalone_html(
 ) -> str:
    """Return a standalone, self-contained HTML document for a page."""
    title = _page_title(page)
-    content_inner = _blocks_of(page)
-    if page.get("content_format") == "markdown" and (page.get("content") or "").strip():
-        body = "\n".join(f"<p>{_text(line)}</p>" for line in (page.get("content") or "").splitlines())
-    else:
-        body = blocks_to_html(content_inner)
+    body = blocks_to_html(_page_blocks(page))

    meta_updated = page.get("updated_at") or ""
    footer = f"<div class='footer'><span>FlowDeck · {_page_title(page)}</span><span>{meta_updated}</span></div>"
@@ -293,11 +535,7 @@ def page_to_standalone_html(
 def _pdf_html(page: dict) -> str:
    """A print-friendly, minimal-CSS HTML for PDF conversion."""
    title = _page_title(page)
-    content_inner = _blocks_of(page)
-    if page.get("content_format") == "markdown" and (page.get("content") or "").strip():
-        body = "\n".join(f"<p>{_text(line)}</p>" for line in (page.get("content") or "").splitlines())
-    else:
-        body = blocks_to_html(content_inner)
+    body = blocks_to_html(_page_blocks(page))
    return f"""<html><head><meta charset="utf-8"><title>{_text(title)}</title>
 <style>
 body{{font-family:Helvetica,Arial,sans-serif;color:#1f2328;font-size:12px;line-height:1.5;}}
@@ -339,7 +577,6 @@ def _site_index_html(pages: list[dict]) -> str:
    """Build the index.html of the static site (list of all pages)."""
    def link(p: dict) -> str:
        title = _page_title(p)
-        slug = quote(f"{title}", safe="")
        return f'<li><a href="{quote(title, safe="")}.html">{_text(title)}</a></li>'

    items = "".join(link(p) for p in pages)
@@ -3024,4 +3024,146 @@ def test_v470_export_site_zip(client):
 def test_v470_export_404(client):
    """Export endpoints return 404 for missing pages."""
    resp = client.get("/api/export/markdown/999999")
-    assert resp.status_code == 404
+    assert resp.status_code == 404
+
+
+# ── v4.7.2: Export resolves file & markdown page content (not just title) ──
+# Tested at the service level: the export HTTP endpoints run behind an in-memory
+# rate limiter whose store persists across the whole test process, so dozens of
+# additional HTTP hits at the end of the suite trip 429. These assert on the
+# service functions directly, which is where the content-resolution lives.
+
+def _make_src_page(raw_md=None, file_info=None, title="Src Page", parent_id=None):
+    """Insert a user + page storing raw markdown OR file metadata.
+
+    file_info = (rel_path_under_data_dir, mime). The real file is written to a
+    temp data dir whose path is exposed through ``FLOWDECK_DATA_DIR``.
+    """
+    import uuid
+    import json as _json
+    from app.db import get_conn
+    login = "v472_" + uuid.uuid4().hex[:10]
+    with get_conn() as conn:
+        conn.execute("INSERT INTO users (login, full_name, email) VALUES (?, 'V472', ?)",
+                     (login, login + "@t.com"))
+        uid = conn.execute("SELECT id FROM users WHERE login=?", (login,)).fetchone()["id"]
+        if file_info:
+            rel, mime = file_info
+            payload = _json.dumps({"file_path": rel, "mime_type": mime, "size": 100})
+            fmt = "file"
+        else:
+            payload = raw_md
+            fmt = "markdown"
+        cur = conn.execute(
+            "INSERT INTO pages (workspace, title, content, content_format, parent_id) "
+            "VALUES (?, ?, ?, ?, ?)",
+            (login, title, payload, fmt, parent_id),
+        )
+        pid = cur.lastrowid
+        conn.commit()
+    return pid, uid
+
+
+def _cleanup_src(pid, uid):
+    from app.db import get_conn
+    with get_conn() as conn:
+        conn.execute("DELETE FROM pages WHERE parent_id=? OR id=?", (pid, pid))
+        conn.execute("DELETE FROM users WHERE id=?", (uid,))
+        conn.commit()
+
+
+def _load(pid):
+    from app.db import get_conn
+    with get_conn() as conn:
+        return dict(conn.execute("SELECT * FROM pages WHERE id=?", (pid,)).fetchone())
+
+
+def test_v472_markdown_sourced_page_exports_body(client):
+    """content_format='markdown' page exports its body, not just the title."""
+    from app.services.export import page_to_markdown
+    raw = "# Intro\n\nCeci est le contenu réel de la page.\n\n- point un\n- point deux\n"
+    pid, uid = _make_src_page(raw_md=raw, title="Page MD")
+    try:
+        md = page_to_markdown(_load(pid))
+        assert "# Page MD" in md
+        assert "Ceci est le contenu réel de la page." in md
+        assert "point un" in md
+    finally:
+        _cleanup_src(pid, uid)
+
+
+def test_v472_markdown_sourced_page_renders_headings_to_html(client):
+    """Raw-markdown page renders headings/lists in HTML (not line-wrapped)."""
+    from app.services.export import page_to_standalone_html
+    raw = "# Titre Principal\n\nParagraphe de contenu.\n\n## Sous section\n\n- a\n- b\n"
+    pid, uid = _make_src_page(raw_md=raw, title="Page HTML")
+    try:
+        body = page_to_standalone_html(_load(pid), include_children=False)
+        assert "<h1" in body and "Titre Principal" in body
+        assert "<h2" in body and "Sous section" in body
+        assert "<li>" in body and "Paragraphe de contenu" in body
+    finally:
+        _cleanup_src(pid, uid)
+
+
+def test_v472_file_page_exports_uploaded_content(client, monkeypatch, tmp_path):
+    """Uploaded markdown file page exports its real disk content (v4.7.2 fix)."""
+    from app.services.export import page_to_markdown, page_to_standalone_html, page_to_pdf_bytes
+    monkeypatch.setenv("FLOWDECK_DATA_DIR", str(tmp_path))
+    rel = "uploads/workspace_1/note.md"
+    disk = tmp_path / rel
+    disk.parent.mkdir(parents=True, exist_ok=True)
+    disk.write_text("# Note technique\n\ncontenu du fichier sur disque\n", encoding="utf-8")
+
+    pid, uid = _make_src_page(file_info=(rel, "text/markdown"), title="note.md")
+    try:
+        md = page_to_markdown(_load(pid), include_children=False)
+        assert "# note.md" in md
+        assert "Note technique" in md
+        assert "contenu du fichier sur disque" in md
+
+        html = page_to_standalone_html(_load(pid), include_children=False)
+        assert "Note technique" in html
+        assert "contenu du fichier sur disque" in html
+
+        pdf = page_to_pdf_bytes(_load(pid))
+        assert pdf[:5] == b"%PDF-"
+    finally:
+        _cleanup_src(pid, uid)
+
+
+def test_v472_code_file_page_exported_as_code(client, monkeypatch, tmp_path):
+    """A non-markdown text file page exports its content, not blank."""
+    from app.services.export import page_to_markdown, page_to_standalone_html
+    monkeypatch.setenv("FLOWDECK_DATA_DIR", str(tmp_path))
+    rel = "uploads/workspace_1/app.py"
+    disk = tmp_path / rel
+    disk.parent.mkdir(parents=True, exist_ok=True)
+    disk.write_text("def hello():\n    return 'world'\n", encoding="utf-8")
+
+    pid, uid = _make_src_page(file_info=(rel, "text/x-python"), title="app.py")
+    try:
+        md = page_to_markdown(_load(pid), include_children=False)
+        assert "def hello():" in md and "world" in md
+        html = page_to_standalone_html(_load(pid), include_children=False)
+        assert "def hello():" in html
+    finally:
+        _cleanup_src(pid, uid)
+
+
+def test_v472_binary_file_page_not_exported(client, monkeypatch, tmp_path):
+    """Non-textual files (e.g. PDF uploads) export only the title, no garbage."""
+    from app.services.export import page_to_markdown
+    monkeypatch.setenv("FLOWDECK_DATA_DIR", str(tmp_path))
+    rel = "uploads/workspace_1/manual.pdf"
+    disk = tmp_path / rel
+    disk.parent.mkdir(parents=True, exist_ok=True)
+    disk.write_bytes(b"%PDF-1.4\nfake binary content")
+
+    pid, uid = _make_src_page(file_info=(rel, "application/pdf"), title="manual.pdf")
+    try:
+        md = page_to_markdown(_load(pid), include_children=False)
+        assert "# manual.pdf" in md
+        assert "fake binary" not in md  # never dump binary into markdown
+    finally:
+        _cleanup_src(pid, uid)
@@ -1 +1 @@
 .7.1
 .7.2