ObsiGate

Ultra-light web gateway for your Obsidian vaults — Access, browse, and search all your Obsidian notes from any device via a modern, responsive web interface.

┌─────────────────────────────────────────────────────────┐
│  [🔍 Search...]          [☀/🌙 Theme]   ObsiGate      │
├──────────────┬──────────────────────────────────────────┤
│  SIDEBAR     │           CONTENT AREA                   │
│  ▼ Recipes   │   📄 File Title                          │
│    📁 Soups  │   Tags: #recipe #quick                  │
│    📄 Pizza  │   [Rendered Markdown Content]            │
│  ▼ IT        │                                          │
│    📁 Docker │                                          │
│  Tags Cloud  │                                          │
└──────────────┴──────────────────────────────────────────┘

---

📋 Table of Contents

• Features
• Architecture
• Prerequisites
• Quick Installation
• Detailed Configuration
• Environment Variables
• 🔒 Authentication
• Adding a New Vault
• Build & Deployment with build.sh
• Desktop (Tauri) — Native Application
• Usage
• API
• Performance
• Troubleshooting
• Tech Stack
• Changelog

---

✨ Features

• 🤖 Integrated AI Editor — CodeMirror 6 editor with AI toolbar: improve, correct, translate, generate, custom rewrite, toolbox (list, table, frontmatter, canvas) — multi-provider DeepSeek/OpenRouter/Gemini
• 🧩 MCP Server & AI Agent — Built-in Model Context Protocol server (/mcp) and tool-calling assistant: read, search and edit your vaults from Claude Desktop, Cursor… with two-step confirmations, per-vault permissions, rate limiting and secret redaction (guide)
• 👥 Real-time Collaboration — Simultaneous editing of the same document (Yjs/CRDT): colored remote cursors, presence indicator, conflict-free merge, automatic reconnection and server-side persistence (details)
• 📱 Native Mobile Editor — Touch-optimised editing: floating Markdown toolbar (bold/italic/code/list/link), persistent Paste button (iOS workaround), pinch-zoom font & adjustable height, swipe shortcuts (backlinks / table of contents) and a full-screen reading mode with page navigation (details)
• 🗺️ Interactive Graph View — Canvas force-directed with Barnes-Hut O(n log n), filters (tag, type), depth, focus mode, navigation history ←→↑, export PNG, preview on hover (Ctrl+click)
• 🗂️ Multi-vault : View multiple Obsidian vaults simultaneously
• 🌳 Tree Navigation : Browse your folders and files in the sidebar
• 🔍 Advanced Search : TF-IDF search engine with French stemming, accent normalization, highlighted snippets, facets, pagination, and sorting — plus an optional semantic search (embeddings via all-MiniLM-L6-v2, hybrid TF-IDF + RRF fusion) toggled with ~ (details)
• 💡 Smart Autocomplete : Suggestions for files, tags, and history with keyboard navigation
• 🧩 Query Syntax : Operators tag:, #, vault:, title:, path:, ext: with visual chips
• 📜 Search History : Persisted in localStorage (max 50 entries, LIFO, deduplicated)
• 🏷️ Tag Cloud : Filtering by tags extracted from YAML frontmatters
• 🔗 Wikilinks : [[internal links]] from Obsidian are clickable
• 🖼️ Obsidian Images : Full support for all Obsidian image syntaxes with intelligent resolution
• 🎨 Excalidraw Diagrams : Native viewer/editor for .excalidraw and .excalidraw.md files (sandboxed iframe, autosave, dark/light theme, diagram text indexed for search)
• 🎨 Syntax Highlight : Syntax highlighting for code blocks
• 🌓 Light/Dark Theme : Toggle persisted in localStorage
• 📡 Real-time Sync : Automatic file monitoring via watchdog with incremental index updates
• 📡 Server-Sent Events : SSE notifications for index changes with automatic reconnection
• ➕ Dynamic Vault Management : Add/remove vaults via API without restart
• 🐳 Multi-platform Docker : linux/amd64, linux/arm64, linux/arm/v7, linux/386
• 🔒 Authentication : JWT + Argon2id, persistent sessions, per-vault access control
• 🛡️ Security : Rate limiting, audit log, automatic backup, secret redaction, CSP headers, path traversal protection, non-root user
• ⚡ Performance : GZip compression, Cache-Control immutable, incremental inverted index, search with no disk I/O
• ❤️ Healthcheck : /api/health endpoint integrated for Docker and monitoring

---

🚀 Prerequisites

System Requirements

• Docker >= 20.10
• docker-compose >= 2.0
• Disk space : ~200MB for the Docker image

Supported Systems

• Linux (Ubuntu, Debian, CentOS, etc.)
• macOS (Intel and Apple Silicon)
• Windows (with Docker Desktop)
• Docker-compatible NAS (Synology, QNAP, etc.)

---

⚡ Quick Installation

1. Clone the Repository

git clone https://git.dracodev.net/Projets/ObsiGate.git
cd ObsiGate

2. Configure Your Vaults and Secrets

Edit the docker-compose.yml file to add your Obsidian vaults:

volumes:
  - /absolute/path/to/your/vault:/vaults/YourVaultName:ro

Important

: The path must be absolute and the volume must be read-only (:ro)

Create your .env file for authentication and secrets:

cp .env.example .env
# Edit .env to configure your passwords and options

Never commit .env! It is in .gitignore. Use .env.example as a reference.

3. Launch the Application

# Make the script executable (once)
chmod +x build.sh

# Build + deploy in one command
./build.sh

That's it! build.sh checks Docker, validates your volumes, builds the image, and starts the container automatically.

Useful options: ./build.sh --help for help, ./build.sh --cache for a faster rebuild, ./build.sh --build-only to build without starting.

4. Access the Interface

Open your browser at: http://localhost:2020

---

⚙️ Detailed Configuration

Step 1: Prepare Your Vaults

1. Locate your Obsidian vaults on your system
2. Note the absolute paths to each .obsidian folder
3. Check permissions : Docker must be able to read these folders

Step 2: Configure docker-compose.yml

Here's a complete example:

services:
  obsigate:
    build:
      context: .
    image: obsigate:latest
    container_name: obsigate
    restart: unless-stopped
    ports:
      - "2020:8080"  # Local port 2020 → Container port 8080
    volumes:
      # Vault examples (adapt to your paths)
      - /home/user/Documents/Obsidian-Recipes:/vaults/Recipes:ro
      - /home/user/Documents/Obsidian-IT:/vaults/IT:ro
      - /home/user/Documents/Obsidian-Personal:/vaults/Personal:ro
      # Auth data persistence
      - ./data:/app/data
    environment:
      # Vault configuration
      - VAULT_1_NAME=Recipes
      - VAULT_1_PATH=/vaults/Recipes
      - VAULT_2_NAME=IT
      - VAULT_2_PATH=/vaults/IT
      - VAULT_3_NAME=Personal
      - VAULT_3_PATH=/vaults/Personal
      # Auth (secrets are in .env)
      - OBSIGATE_AUTH_ENABLED=true
      - OBSIGATE_ADMIN_USER=admin
    env_file:
      - .env  # Contains OBSIGATE_ADMIN_PASSWORD and other secrets

Step 3: Build & Deploy

# Use the automated build script (recommended)
chmod +x build.sh
./build.sh

The build.sh script handles everything: prerequisite checks, volume validation, image building, and startup.

Manual alternative:

docker compose build --no-cache
docker compose up -d

Docker Compatibility : The image uses the minimal uvicorn variant and a compatible fastapi version (0.110.3) to avoid certain optional native dependencies (watchfiles, uvloop, httptools, fastapi-cli, etc.) that may fail to build on some platforms like Alpine, ARM, or i386.

---

🌍 Environment Variables

Vaults are configured using pairs of VAULT_N_NAME / VAULT_N_PATH variables (N = 1, 2, 3…):

Variable	Description	Example
VAULT_1_NAME	Display name of the vault	Recipes
VAULT_1_PATH	Path inside the container	/vaults/Obsidian-RECIPES
VAULT_1_ATTACHMENTS_PATH	Relative path to the attachments folder (optional)	06_Toolbox/6.2_Attachments
VAULT_1_SCAN_ATTACHMENTS	Enable image scanning on startup (optional, default: true)	true
VAULT_2_NAME	Display name of the vault	IT
VAULT_2_PATH	Path inside the container	/vaults/Obsidian_IT

Naming rules:

• Use only letters, numbers, and hyphens
• No spaces or special characters
• The name must match the path inside the container

---

🔒 Authentication

Disabled by default — Compatible with all existing installations.

ObsiGate supports an optional authentication system based on JWT + Argon2id with per-vault access control.

Enable Authentication

1. Copy the .env template :

   cp .env.example .env
   

2. Edit .env and uncomment/configure the variables :

   OBSIGATE_AUTH_ENABLED=true
   OBSIGATE_ADMIN_USER=admin
   OBSIGATE_ADMIN_PASSWORD=your_password  # Leave empty = auto-generated (see logs)
   # OBSIGATE_SECURE_COOKIES=false              # true if behind HTTPS
   

3. In docker-compose.yml, make sure you have :

   env_file:
     - .env
   

Never put passwords in docker-compose.yml! Always use .env.

First Startup

If no user exists, ObsiGate automatically creates an admin account and displays the password in the logs :

docker-compose logs obsigate | grep -A4 "FIRST"

============================================================
FIRST STARTUP — Admin account created automatically
  Username : admin
  Password : xK9mQ3pLr7wN2jT5
CHANGE THIS PASSWORD on first login!
============================================================

User Management via CLI

# Create a user
docker exec obsigate python backend/create_admin.py create alice MyPassword --role user --vaults Recipes IT

# Create an admin with full access
docker exec obsigate python backend/create_admin.py create bob SecretPass --role admin --vaults "*"

# List users
docker exec obsigate python backend/create_admin.py list

# Delete a user
docker exec obsigate python backend/create_admin.py delete alice

Admin Interface

When an admin account is logged in, a 🛡️ icon appears in the header. Clicking it opens the admin panel allowing you to:

• List all users
• Create / edit / delete users
• Assign accessible vaults per user
• Enable/disable accounts

Per-Vault Access Control

vaults value	Access
["*"]	All vaults (including future ones) — admin default
["Recipes", "IT"]	Only these vaults
[]	No access

Auth Environment Variables

Variable	Description	Default
OBSIGATE_AUTH_ENABLED	Enable authentication	false
OBSIGATE_ADMIN_USER	Auto-created admin username	admin
OBSIGATE_ADMIN_PASSWORD	Admin password (empty = auto-generated)	(auto)
OBSIGATE_SECURE_COOKIES	Secure cookie (HTTPS only)	false
OBSIGATE_ACCESS_TOKEN_TTL	JWT token lifetime (seconds)	3600
OBSIGATE_REFRESH_TOKEN_TTL	Refresh token lifetime (seconds)	2592000
OBSIGATE_LOGIN_MAX_ATTEMPTS	Max login attempts per IP	10
OBSIGATE_ACCOUNT_MAX_ATTEMPTS	Max login attempts per account	10
OBSIGATE_LOGIN_WINDOW_SECONDS	Rate limiting window (seconds)	900
OBSIGATE_TRUST_PROXY	Trust X-Forwarded-For for the client IP (reverse proxy)	false
OBSIGATE_WEBHOOK_ALLOW_HTTP	Allow non-HTTPS webhook targets	false
OBSIGATE_WEBHOOK_ALLOW_PRIVATE	Allow webhooks to private/loopback addresses	false
OBSIGATE_PDF_MAX_SIZE_MB	Max PDF size for text extraction	50
OBSIGATE_PDF_EXTRACT_TIMEOUT	PDF extraction timeout (seconds)	30
OBSIGATE_TAVILY_API_KEY / OBSIGATE_BRAVE_API_KEY / OBSIGATE_SERPAPI_API_KEY / OBSIGATE_EXA_API_KEY	Keyed web-search providers (tried before SearXNG)	—
OBSIGATE_WEB_PROVIDERS	Search provider order (e.g. brave,searxng)	—
OBSIGATE_WEB_RETRY	Web tools network retries (house-made backoff)	1
OBSIGATE_WEB_CACHE_TTL	SQLite cache TTL for web results (seconds, 0 = off)	900
OBSIGATE_GITEA_URL / OBSIGATE_GITEA_TOKEN	Gitea connected source (git_list_repos…)	—
OBSIGATE_GITHUB_TOKEN	GitHub token (git_list_repos…)	—

These keys can also be entered from the UI (menu → Configurations → "Connected sources & search"): the stored value goes to data/api_keys.json and takes precedence over the environment variable.

All these variables are documented in .env.example.

Volume for Persistence

Auth data (users.json, secret.key) is stored in /app/data. Add a volume to persist it:

volumes:
  # vaults...
  - ./data:/app/data  # Persist users and JWT key

---

➕ Adding a New Vault

Method 1: Direct Editing

1. Stop the container :

   docker-compose down
   

2. Add a volume in docker-compose.yml :

   volumes:
     - /new/vault/path:/vaults/NewVault:ro
   

3. Add the environment variables :

   environment:
     - VAULT_4_NAME=NewVault
     - VAULT_4_PATH=/vaults/NewVault
   

4. Restart :

   ./build.sh
   

Method 2: Hot-reload (recommended)

1. Add the volume and variables as above
2. Apply the changes :

   ./build.sh
   

3. Reload the index via the interface or the API :

   curl http://localhost:2020/api/index/reload
   

Method 3: Dynamic API (without restart)

Add a vault on the fly via the API (the volume must already be mounted):

curl -X POST http://localhost:2020/api/vaults/add \
  -H "Content-Type: application/json" \
  -d '{"name": "NewVault", "path": "/vaults/NewVault"}'

Remove a vault:

curl -X DELETE http://localhost:2020/api/vaults/NewVault

---

🔨 Build & Deployment with build.sh

The build.sh script is the recommended way to build and deploy ObsiGate.

Basic Usage

chmod +x build.sh   # once
./build.sh          # build from scratch + start

Available Options

Option	Description
--help, -h	Display full help
--build-only	Build the image without starting the container
--no-cache	Full rebuild without Docker cache (default)
--cache	Use Docker cache (faster if few changes)
--progress=plain	Verbose output (recommended for debugging)
--progress=tty	Interactive output with progress bars

What the Script Does

1. Checks that Docker and Docker Compose are installed (with version numbers)
2. Validates the presence and syntax of the docker-compose.yml file
3. Checks each mounted volume : warns if a source directory does not exist
4. Builds the Docker image via docker compose build (multi-stage, ~180MB)
5. Starts the container via docker compose up -d
6. Displays the container status then real-time logs

Examples

# Clean build + start (recommended after code changes)
./build.sh

# Fast rebuild with cache (minor frontend changes only)
./build.sh --cache

# Test the build without starting (useful for pre-prod testing)
./build.sh --build-only

# Verbose output to debug a build failure
./build.sh --progress=plain

Stopping / Restarting

docker compose down      # Stop the container
docker compose up -d     # Restart without rebuild
docker compose logs -f   # View logs

---

🖼️ Obsidian Image Rendering

ObsiGate supports all Obsidian image syntaxes with an intelligent multi-strategy resolution system.

Supported Syntaxes

1. Standard Markdown with HTML attributes (Obsidian-compatible) :

   [<img width="180" height="60" src="path/to/image.svg"/>](https://example.com)
   

2. Wiki-link embed with full path :

   ![[06_Toolbox/6.2_Attachments/image.svg]]
   

3. Wiki-link embed with filename only :

   ![[image.svg]]
   

4. Standard Markdown :

   ![alt text](path/to/image.png)
   

Intelligent Path Resolution

ObsiGate uses 7 resolution strategies in order of priority:

1. Absolute path : If the path is absolute and exists
2. Configured attachments folder : Via VAULT_N_ATTACHMENTS_PATH
3. Startup index (unique match) : Search by filename in the index
4. Same directory : Relative to the current markdown file
5. Vault root : Relative to the vault root
6. Startup index (closest match) : If multiple files have the same name
7. Fallback : Display a styled placeholder [image not found: filename.ext]

Configuration

To optimize resolution, configure the attachments folder for each vault:

environment:
  - VAULT_1_NAME=MyVault
  - VAULT_1_PATH=/vaults/MyVault
  - VAULT_1_ATTACHMENTS_PATH=Assets/Images  # Relative path
  - VAULT_1_SCAN_ATTACHMENTS=true           # Enable scanning (default)

Manual Rescan

To rescan images in a vault after adding/removing:

curl -X POST http://localhost:2020/api/attachments/rescan/MyVault

---

🖥️ Desktop (Tauri) — Native Application

ObsiGate Desktop is a native application built with Tauri (Rust + system webview). It embeds the Python backend and frontend in a standalone executable — zero Docker, zero command line.

🚧 Version 2.0.0 — binaries are being stabilized. For now, building from source is recommended.

Native Desktop Features

Feature	Web	Desktop
Local file access	Via upload	Native (folder picker)
System theme	Manual	Auto (follows OS dark/light)
Notifications	Service Worker	Native OS
.md associations	❌	✅ "Open with ObsiGate"
Tray icon	❌	✅ Taskbar
Auto-update	❌	✅ Checks Gitea releases
Offline mode	Limited	Full (local backend)

Download (Pre-built Binaries)

Releases are published on Gitea :

Platform	Format
Linux	.deb + .AppImage
Windows	.msi + .exe (NSIS)

Linux

# .deb (Debian/Ubuntu/Deepin)
sudo dpkg -i obsigate_2.0.0_amd64.deb
# Launch : ObsiGate from the applications menu or `obsigate-desktop`

# .AppImage (any distro)
chmod +x ObsiGate_2.0.0_amd64.AppImage
./ObsiGate_2.0.0_amd64.AppImage

Windows

:: Double-click ObsiGate_2.0.0_x64.msi
:: Or launch ObsiGate from the Start menu

Getting Started

1. Launch the application from the menu or command line
2. The Python backend starts automatically on 127.0.0.1:17890
3. The window opens and loads the ObsiGate interface
4. First launch : select your Obsidian vaults folder via the native picker
5. To close : tray icon → Quit (clean backend shutdown)

Build from Source

Detailed guide: desktop/README.md.

Common Prerequisites

Tool	Version	Installation
Rust (cargo)	≥ 1.75	rustup
Tauri CLI	≥ 2.0	cargo install tauri-cli
Git	—	—
Linux system dependencies	—	sudo apt install libwebkit2gtk-4.1-dev libgtk-3-dev libayatana-appindicator3-dev

Important — staging : tauri.conf.json embeds backend/** and frontend/** from the desktop/ folder. The build scripts automatically copy ../backend and ../frontend into desktop/ before cargo tauri build. Without this staging, the build fails with "glob pattern backend/**/* path not found".

🪟 Windows — build-windows.bat

REM Prerequisites (via Scoop) : rustup, curl, git
scoop install rustup curl git
rustup default stable
cargo install tauri-cli

cd desktop
build-windows.bat

Script steps:

1. Kills residual Python processes (taskkill /F /IM python.exe)
2. Downloads Python 3.11 embed (python.org) → desktop\python-embed\ + enables pip (python311._pth)
3. pip install -r ..\backend\requirements.txt in the embed
4. Staging : copies ..\backend and ..\frontend to desktop\
5. cargo tauri build --target x86_64-pc-windows-msvc --bundles nsis
6. Copies python-embed next to the executable (target\x86_64-pc-windows-msvc\release\) for local dev mode
7. Cleans the staged folders

→ Artifact : desktop\target\x86_64-pc-windows-msvc\release\bundle\nsis\ObsiGate_2.0.0_x64-setup.exe

🐧 Linux — build-linux.sh

cd desktop
chmod +x build-linux.sh
./build-linux.sh

Script steps:

1. Checks Rust + Tauri CLI, installs system dependencies (apt)
2. Creates a Python venv desktop/python-embed/venv + pip install -r ../backend/requirements.txt
3. Staging : copies ../backend and ../frontend to desktop/
4. cargo tauri build --target x86_64-unknown-linux-gnu --bundles deb,appimage
5. Copies the runtime (python-embed/, backend/, frontend/) next to the executable

→ Artifacts :

• desktop/target/x86_64-unknown-linux-gnu/release/bundle/deb/obsigate_2.0.0_amd64.deb
• desktop/target/x86_64-unknown-linux-gnu/release/bundle/appimage/ObsiGate_2.0.0_amd64.AppImage

🤖 CI/CD Builds — Automatic Artifacts

Yes — the workflow .gitea/workflows/desktop-build.yml builds desktop binaries on every push to main touching desktop/**, frontend/**, or backend/** (and manually via workflow_dispatch), on self-hosted runners :

Job	Runner	Artifacts (retained 30 days)
build-windows	[self-hosted, windows, desktop]	desktop/target/release/bundle/msi/*.msi
build-linux	[self-hosted, linux, desktop]	*.AppImage + *.deb

• Artifacts are downloadable from the Actions page of the Gitea run.
• Publishing to Gitea Release is planned for tags v* (Publish to Gitea Release step).
• The web workflow .gitea/workflows/ci.yml separately handles lint → tests → security → Docker build → e2e Playwright.

Desktop Architecture

┌────────────────────────────────────────────┐
│ Tauri (Rust)                               │
│  ├─ Webview (system webview)              │
│  │   └─ Frontend (HTML/JS/CSS)            │
│  └─ Sidecar Python                         │
│      └─ uvicorn backend.main:app          │
│           └─ port 127.0.0.1:17890         │
└────────────────────────────────────────────┘

Lifecycle: Tauri spawns the Python backend → health check → opens the webview. On close: SIGTERM → clean shutdown → cleanup.

---

📖 Usage

Web Interface

1. Navigation : Click on vaults in the sidebar to expand them
2. Search : Use the search bar to search across all vaults
3. Tags : Click on tags to filter content
4. Wikilinks : [[page]] links are clickable and navigable
5. Images : All Obsidian image syntaxes are rendered automatically
6. Theme : Toggle between light/dark theme with the 🌙/☀️ icon

Keyboard Shortcuts

Action	Shortcut
Search	Ctrl + K or /
Toggle theme	Ctrl + T
Focus search	Esc

---

👥 Real-time Collaboration

Multiple users can edit the same markdown document simultaneously (Google Docs style):

• Conflict-free merge via Yjs (CRDT): two people can type in the same place, no change is lost.
• Colored remote cursors and visible selections in CodeMirror, labelled with each user's name.
• Presence indicator in the editor header (avatars + connection status).
• Automatic reconnection (exponential backoff): state is merged on return.
• Server-side persistence: the document is written to disk 2 s after the last change.
• Transport: WebSocket ws(s)://<host>/ws/collab/{vault}/{path}, authenticated via the access_token cookie (or ?token=) and subject to per-vault access control.

No configuration is required: open the same file in two browsers (or two windows) to see it live.

---

🔌 API

ObsiGate exposes a complete REST API :

Endpoint	Description	Method	Auth
/api/health	Health check (status, version, stats)	GET	No
/api/auth/status	Auth status (enabled, users present)	GET	No
/api/auth/login	Login (returns access token + refresh cookie)	POST	No
/api/auth/refresh	Refresh access token via refresh cookie	POST	Cookie
/api/auth/logout	Logout + revoke refresh token	POST	Yes
/api/auth/me	Current user info	GET	Yes
/api/auth/change-password	Change password	POST	Yes
/api/auth/admin/users	List / create users	GET/POST	Admin
/api/auth/admin/users/{u}	Modify / delete a user	PATCH/DELETE	Admin
/api/vaults	List vaults (filtered by permissions)	GET	Yes
/api/browse/{vault}?path=	Browse folders	GET	Yes
/api/file/{vault}?path=	Rendered content of a file	GET	Yes
/api/file/{vault}/raw?path=	Raw content of a file	GET	Yes
/api/file/{vault}/download?path=	Download a file	GET	Yes
/api/file/{vault}/save?path=	Save a file	PUT	Yes
/api/file/{vault}?path=	Delete a file	DELETE	Yes
/api/search/advanced	Advanced TF-IDF search (+ semantic=true for hybrid)	GET	Yes
/api/suggest / /api/tags/suggest	Autocomplete	GET	Yes
/api/tags?vault=	Unique tags with counters	GET	Yes
/api/index/reload	Force a rescan of vaults	GET	Admin
/api/events	Real-time SSE stream	GET	Yes
/api/vaults/add / /api/vaults/{name}	Dynamic vault management	POST/DELETE	Admin
/api/image/{vault}?path=	Serve an image	GET	Yes
/api/config	Read / write configuration	GET/POST	Yes/Admin
/api/diagnostics	Index and memory statistics	GET	Admin

When OBSIGATE_AUTH_ENABLED=false, all endpoints are accessible without a token.

All endpoints expose documented Pydantic schemas. Interactive docs are available at /docs (Swagger UI).

Usage Example :

# Health check
curl http://localhost:2020/api/health

# List vaults
curl http://localhost:2020/api/vaults

# Simple search (legacy)
curl "http://localhost:2020/api/search?q=recipe&vault=all"

# Advanced search with TF-IDF, facets, and pagination
curl "http://localhost:2020/api/search/advanced?q=recipe%20tag:cuisine&vault=all&limit=20&offset=0&sort=relevance"

# Title autocomplete
curl "http://localhost:2020/api/suggest?q=piz&vault=all"

# Tag autocomplete
curl "http://localhost:2020/api/tags/suggest?q=rec&vault=all"

# Get a file
curl "http://localhost:2020/api/file/Recipes?path=pizza.md"

🔍 Advanced Search

Query Syntax

Operator	Description	Example
tag:<name>	Filter by tag	tag:recipe docker
#<name>	Tag shortcut	#linux server
vault:<name>	Filter by vault	vault:IT kubernetes
title:<text>	Filter by title	title:pizza
path:<text>	Filter by path	path:recipes/soups
ext:<type>	Filter by file type	ext:md kubernetes
"exact phrase"	Phrase search	tag:"multiple words"

Extension filter examples: ext:sh for bash scripts, ext:py for Python scripts, ext:md for Markdown files, ext:pdf for PDF documents, ext:excalidraw for Excalidraw diagrams (text-extracted content is indexed).

PDF support

PDF files in your vaults are rendered inline in the browser via the native PDF viewer (iframe + <embed>). The viewer streams the file over HTTP Range requests (206 Partial Content), so large PDFs load progressively. Text is extracted on indexing (pypdf / pymupdf) so PDF content is searchable via the full-text search. Filter with ext:pdf to restrict results to PDFs. PDF metadata (pages, title, author) is available via GET /api/file/{vault}/pdf/info without transferring the document.

Limitations: no OCR (scanned PDFs aren't searchable), no annotation, no editing of the PDF itself.

Excalidraw diagrams

.excalidraw and .excalidraw.md files open in a full visual Excalidraw editor embedded in a sandboxed iframe — draw, edit and save without leaving ObsiGate. Changes are saved automatically (2s debounce) or with Ctrl+S; the editor follows the light/dark theme. Text labels inside the diagram elements are extracted on indexing, so diagram content is searchable via the full-text search (ext:excalidraw). Files created with the Obsidian Excalidraw plugin (including the compressed .excalidraw.md format) are compatible.

Operators are combinable: tag:linux vault:IT ext:md server web searches for "server web" in Markdown files of the IT vault with the linux tag.

Keyboard Shortcuts

Shortcut	Action
Ctrl+K / Cmd+K	Focus search bar
/	Focus search (outside text fields)
↑ / ↓	Navigate suggestions
Enter	Select active suggestion or launch search
Escape	Close suggestions / exit search

Features

• TF-IDF : Scoring based on term frequency weighted by inverse document frequency
• Title boost : Title matches receive 3× higher score
• Accent normalization : resume finds résumé, elephant finds éléphant
• Highlighted snippets : Found terms are wrapped in <mark> in excerpts
• Facets : Counters by vault and by tag in results
• Pagination : Page navigation with 50 results per page
• Sorting : By relevance (TF-IDF) or modification date
• Visual chips : Active filters are shown as removable colored chips
• History : Last 50 searches are stored in localStorage
• Semantic search (optional) : Toggle ~ (or Alt+S) fuses the TF-IDF ranking with an embedding-based ranking (RRF). Works out of the box with a dependency-free hashing embedder; install backend/requirements-semantic.txt and/or set OBSIGATE_EMBEDDING_* for real all-MiniLM-L6-v2 embeddings. See docs/features/semantic-search.md.

---

🔧 Troubleshooting

Common Issues

Port already in use:

# Check who is using the port
sudo netstat -tulpn | grep 2020
# Change the port in docker-compose.yml
ports:
  - "2021:8080"

Vault not found:

• Verify paths are absolute
• Ensure read permissions
• Restart the container after modifications

Build fails:

# Clean Docker cache and rebuild
docker system prune -f
docker compose down
./build.sh --progress=plain

# If failure persists, check build logs
./build.sh --progress=plain 2>&1 | tee build.log

Logs for debugging:

# Real-time logs
docker compose logs -f obsigate

# Detailed logs
docker compose logs --tail=100 obsigate

---

⚡ Performance

Metric	Estimate
Indexing	~1–2s for 1,000 markdown files
Advanced Search	< 10ms for most queries (inverted index + TF-IDF)
Wikilink Resolution	O(1) via lookup table
Memory	~80–150MB per 1,000 files (content capped at 100 KB/file)
Docker Image	~180MB (multi-stage, no build tools)
CPU	Non-blocking; search offloaded to dedicated thread pool

Recommended Settings by Vault Size

Size	Files	search_workers	prefix_max_expansions	max_content_size
Small	< 500	1	50	100,000
Medium	500–5,000	2	50	100,000
Large	5,000+	4	30	50,000

These parameters are configurable via the interface (Settings) or the /api/config API.

Key Optimizations (v1.2.0)

• Inverted index with set-intersection : Search uses posting lists for O(k × postings) retrieval instead of O(N) full scan
• Prefix matching by binary search : O(log V + k) instead of O(V) linear vocabulary scan
• ThreadPoolExecutor : CPU-bound search functions are offloaded from the asyncio event loop
• Race condition guard : currentSearchId + AbortController prevent stale result rendering
• Progress bar : Animated progress bar during search
• Search timeout : Automatic abort after 30s (configurable)
• Query time display : Server time shown in results (query_time_ms)
• Staleness detection fix : Generation counter instead of id(index) to detect index changes

Optimizations v1.5.0 (Quick Wins 2026-05-27)

• GZip Compression : FastAPI middleware compresses all responses >1KB (~70% bandwidth saved)
• Cache-Control immutable : Static assets (/static/*) are cached by the browser for 1 year
• .dockerignore : Minimal Docker build context (no .git, docs/, *.md, __pycache__)
• Incremental InvertedIndex : add_document/remove_document hooks, no more O(N) rebuild on every mutation

Optimizations v1.1.0

• Search without I/O : File content is cached in memory index
• Multi-factor scoring : exact title (+20), partial title (+10), path (+5), tag (+3), content frequency (x1 per occurrence, capped at 10)
• Markdown renderer singleton : The mistune renderer is instantiated once
• AbortController : Stale search requests are aborted client-side
• Debounced icon rendering : lucide.createIcons() is batched via requestAnimationFrame

---

🛡️ Security

• Path traversal : All file endpoints validate that the resolved path stays within the vault
• Rate limiting : 10 login attempts max per IP over 15 minutes + per-account lockout (5 attempts)
• Audit log : All writes, deletions, and config changes are logged in data/audit.log (JSON lines, 10 MB rotation)
• Automatic backup : Every file modification or deletion is saved in .obsigate-backup/ with timestamp
• Secret redaction : Automatic masking of JWTs, API keys, tokens, and connection strings in previews
• Non-root user : The Docker container runs under user obsigate (UID 1000)
• Read-only volumes : Vaults are mounted as :ro by default in docker-compose
• Secrets in .env : Passwords and tokens are never in docker-compose.yml
• Atomic writes : Data files (users.json, shares.json, webhooks.json) use tmp+replace

---

🏗️ Tech Stack

• Backend : Python 3.11 + FastAPI 0.110 + Uvicorn
• Auth : python-jose (JWT HS256) + argon2-cffi (Argon2id)
• Security : Rate limiting (IP + account lockout), audit logging, secret redaction, atomic writes
• File Watcher : watchdog 4.x (native inotify + fallback polling)
• Frontend : Vanilla JS + HTML + CSS (zero framework, zero build)
• Markdown Rendering : mistune 3.x
• PDF Export : WeasyPrint 60+
• Compression : GZip middleware (FastAPI), Cache-Control immutable for assets
• Docker Image : python:3.11-slim (multi-stage), .dockerignore for minimal context
• User Storage : Local JSON (data/users.json) — no database
• Architecture : SPA + REST API + SSE

---

🏠 Architecture

┌─────────────────┐     ┌─────────────────────────────────────────┐
│   Browser       │◄───►│  FastAPI (backend/main.py)             │
│   (SPA)         │ REST │                                         │
│                 │      │  ┌──────────────┐  ┌──────────────┐  │
│  app.js         │      │  │ indexer.py  │  │  search.py    │  │
│  style.css      │      │  │ (scan+cache)│  │ (in-memory)  │  │
│  index.html     │      │  └───────┬──────┘  └──────┬───────┘  │
└─────────────────┘      │        │              │              │
                         │        └──────┬───────┘              │
                         │             │                        │
                         │    ┌────────┴─────────┐            │
                         │    │ In-Memory Index │            │
                         │    │ (files, tags,   │            │
                         │    │  content, lookup)│            │
                         │    └──────────────────┘            │
                         └─────────────────────────────────────────┘
         ┌───────────────────────────────┐
         │  Filesystem (mounted vaults)  │
         │  /vaults/Recipes  (ro)        │
         │  /vaults/IT         (ro)      │
         └───────────────────────────────┘

Data Flow:

1. On startup, indexer.py scans all vaults in parallel (thread pool)
2. Content, tags (YAML + inline), and metadata are cached in memory
3. An O(1) lookup table is built for wikilink resolution
4. watcher.py starts file monitoring (native watchdog or polling)
5. Detected changes trigger incremental index updates
6. Changes are notified to the frontend via Server-Sent Events (SSE)
7. Search queries use the in-memory index (zero disk I/O)
8. The SPA frontend communicates via REST + SSE and manages state client-side

---

📝 Development

🧪 Tests and CI — Mandatory Rules (Before Each Push)

⚠️ Absolute rule — human contributors and AI agents: never git push without having locally executed the entire sequence below with 100 % success. Push is not meant to discover failures: CI is a safety net, not a diagnostic tool.

Local Checks Before Commit

Step	Command	Equivalent CI Job
Backend lint	ruff check backend/	lint
Frontend import validation	node tests/frontend/validate-imports.mjs	lint
Frontend unit tests	node tests/frontend/unit.test.mjs	lint
Backend tests	pytest tests/ -q	test
E2E Playwright	npm run test:e2e (~5 min)	e2e

Local E2E Tests (npm run test:e2e)

The scripts/run-e2e-local.sh script faithfully reproduces the e2e CI job conditions (.gitea/workflows/ci.yml):

• backend started natively (venv .venv-e2e, Python 3.11 via uv) ;
• authentication disabled (OBSIGATE_AUTH_ENABLED=false) ;
• fixtures test_vault (TestVault) and test_dir (TestDir) ;
• port 2029 (baseURL from playwright.config.ts) ;
• Playwright project chromium-desktop only (same as CI).

Prerequisites: uv, Node.js ≥ 20, Playwright browsers (npx playwright install chromium). The venv is created automatically on first run. Docker is not required.

npm run test:e2e                                  # full suite
bash scripts/run-e2e-local.sh --headed            # visible browser
bash scripts/run-e2e-local.sh -g "reset panes"    # filter on a test

The suite must end with all tests passing (60 currently), with no failure or reliance on retries. In case of failure: fix and re-run locally until 100 %, then only commit.

Test Writing Rules (Lesson Learned)

• Check every selector in the actual DOM before using it in a test. Real bug encountered: a test was targeting #cp-input while the command palette input only has the .cp-input class (no id) — the test could not pass in any environment.
• Any new or modified test must be run locally (at least with -g "test name") before push.
• No workarounds that mask flakiness (arbitrary waitForTimeout, silent fallbacks): fix the root cause.

Commits and Push

• Commit message: imperative subject ≤ 50 chars; body (72 col.) only if it adds useful information.
• One commit = one logical change (do not mix fix and refactor).
• Push only when all 5 local steps are green. CI pipeline: lint → test → security → build → e2e.

Project Structure

ObsiGate/
├── backend/              # FastAPI API
│   ├── main.py          # Endpoints, Pydantic models, markdown rendering
│   ├── indexer.py       # Vault scanning, in-memory index, lookup table
│   ├── search.py        # Full-text search engine with scoring
│   ├── watcher.py       # File monitoring (watchdog + debounce)
│   ├── auth/            # Authentication module
│   │   ├── password.py  # Argon2id hashing
│   │   ├── jwt_handler.py # JWT tokens + revocation
│   │   ├── user_store.py  # users.json CRUD (atomic)
│   │   ├── middleware.py  # FastAPI auth dependencies
│   │   └── router.py    # /api/auth/* endpoints
│   ├── create_admin.py  # User management CLI
│   └── requirements.txt
├── frontend/            # Web interface (Vanilla JS, zero framework)
│   ├── index.html       # SPA page + login screen
│   ├── app.js           # SPA logic, AuthManager, AdminPanel
│   └── style.css        # Styles (CSS variables, themes, responsive)
├── data/                # Persistent data (created on startup)
│   ├── users.json       # Users (Argon2id hashed)
│   └── secret.key       # JWT secret key (512 bits)
├── Dockerfile           # Multi-stage, healthcheck, non-root
├── docker-compose.yml   # Deployment with healthcheck and auth env vars
├── build.sh             # Automated build & deployment (docker compose build + up)
└── docs/CONTRIBUTING.md # Contribution guide

Contributing

See CONTRIBUTING.md for code standards and docs/DELIVERY_WORKFLOW.md for the mandatory delivery process.

---

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

---

🤝 Support

• Issues : git.dracodev.net/Projets/ObsiGate/issues
• Documentation : git.dracodev.net/Projets/ObsiGate/wiki
• Author : Bruno Beloeil

---

📝 Changelog

See CHANGELOG.md for the complete version history (v1.0.0 → v2.11.2).

---

Project: ObsiGate | Version: 2.11.2 | Last updated: May 2026