ObsiGate/README.md

# ObsiGate

**Ultra-light web gateway for your Obsidian vaults** — Access, browse, and search all your Obsidian notes from any device via a modern, responsive web interface.

[![Version](https://img.shields.io/badge/Version-1.7.0-blue.svg)]()
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
[![Docker](https://img.shields.io/badge/Docker-Ready-blue.svg)](https://www.docker.com/)
[![Python](https://img.shields.io/badge/Python-3.11+-green.svg)](https://www.python.org/)
[![CI/CD](https://img.shields.io/badge/CI%2FCD-Gitea_Actions-green.svg)](https://git.dracodev.net/Projets/ObsiGate/actions)

```
┌─────────────────────────────────────────────────────────┐
│  [🔍 Search...]          [☀/🌙 Theme]   ObsiGate      │
├──────────────┬──────────────────────────────────────────┤
│  SIDEBAR     │           CONTENT AREA                   │
│  ▼ Recipes   │   📄 File Title                          │
│    📁 Soups  │   Tags: #recipe #quick                  │
│    📄 Pizza  │   [Rendered Markdown Content]            │
│  ▼ IT        │                                          │
│    📁 Docker │                                          │
│  Tags Cloud  │                                          │
└──────────────┴──────────────────────────────────────────┘
```

---

## 📋 Table of Contents

- [Features](#features)
- [Architecture](#architecture)
- [Prerequisites](#prerequisites)
- [Quick Installation](#quick-installation)
- [Detailed Configuration](#detailed-configuration)
- [Environment Variables](#environment-variables)
- [🔒 Authentication](#authentication)
- [Adding a New Vault](#adding-a-new-vault)
- [Build & Deployment with build.sh](#build--deployment-with-buildsh)
- [Desktop (Tauri) — Native Application](#desktop-tauri--native-application)
- [Usage](#usage)
- [API](#api)
- [Performance](#performance)
- [Troubleshooting](#troubleshooting)
- [Tech Stack](#tech-stack)
- [Changelog](#changelog)

---

## ✨ Features

- **🤖 Integrated AI Editor** — CodeMirror 6 editor with AI toolbar: improve, correct, translate, generate, custom rewrite, toolbox (list, table, frontmatter, canvas) — multi-provider DeepSeek/OpenRouter/Gemini
- **🧩 MCP Server & AI Agent** — Built-in Model Context Protocol server (`/mcp`) and tool-calling assistant: read, search and edit your vaults from Claude Desktop, Cursor… with two-step confirmations, per-vault permissions, rate limiting and secret redaction ([guide](docs/MCP_GUIDE.md))
- **👥 Real-time Collaboration** — Simultaneous editing of the same document (Yjs/CRDT): colored remote cursors, presence indicator, conflict-free merge, automatic reconnection and server-side persistence ([details](docs/features/collaboration.md))
- **📱 Native Mobile Editor** — Touch-optimised editing: floating Markdown toolbar (bold/italic/code/list/link), persistent Paste button (iOS workaround), pinch-zoom font & adjustable height, swipe shortcuts (backlinks / table of contents) and a full-screen reading mode with page navigation ([details](docs/features/mobile-editor.md))
- **🗺️ Interactive Graph View** — Canvas force-directed with Barnes-Hut O(n log n), filters (tag, type), depth, focus mode, navigation history ←→↑, export PNG, preview on hover (Ctrl+click)
- **🗂️ Multi-vault** : View multiple Obsidian vaults simultaneously
- **🌳 Tree Navigation** : Browse your folders and files in the sidebar
- **🔍 Advanced Search** : TF-IDF search engine with French stemming, accent normalization, highlighted snippets, facets, pagination, and sorting — plus an optional **semantic search** (embeddings via `all-MiniLM-L6-v2`, hybrid TF-IDF + RRF fusion) toggled with `~` ([details](docs/features/semantic-search.md))
- **💡 Smart Autocomplete** : Suggestions for files, tags, and history with keyboard navigation
- **🧩 Query Syntax** : Operators `tag:`, `#`, `vault:`, `title:`, `path:`, `ext:` with visual chips
- **📜 Search History** : Persisted in localStorage (max 50 entries, LIFO, deduplicated)
- **🏷️ Tag Cloud** : Filtering by tags extracted from YAML frontmatters
- **🔗 Wikilinks** : `[[internal links]]` from Obsidian are clickable
- **🖼️ Obsidian Images** : Full support for all Obsidian image syntaxes with intelligent resolution
- **🎨 Excalidraw Diagrams** : Native viewer/editor for `.excalidraw` and `.excalidraw.md` files (sandboxed iframe, autosave, dark/light theme, diagram text indexed for search)
- **🎨 Syntax Highlight** : Syntax highlighting for code blocks
- **🌓 Light/Dark Theme** : Toggle persisted in localStorage
- **📡 Real-time Sync** : Automatic file monitoring via watchdog with incremental index updates
- **📡 Server-Sent Events** : SSE notifications for index changes with automatic reconnection
- **➕ Dynamic Vault Management** : Add/remove vaults via API without restart
- **🐳 Multi-platform Docker** : linux/amd64, linux/arm64, linux/arm/v7, linux/386
- **🔒 Authentication** : JWT + Argon2id, persistent sessions, per-vault access control
- **🛡️ Security** : Rate limiting, audit log, automatic backup, secret redaction, CSP headers, path traversal protection, non-root user
- **⚡ Performance** : GZip compression, Cache-Control immutable, incremental inverted index, search with no disk I/O
- **❤️ Healthcheck** : `/api/health` endpoint integrated for Docker and monitoring

---

## 🚀 Prerequisites

### System Requirements
- **Docker** >= 20.10
- **docker-compose** >= 2.0
- **Disk space** : ~200MB for the Docker image

### Supported Systems
- Linux (Ubuntu, Debian, CentOS, etc.)
- macOS (Intel and Apple Silicon)
- Windows (with Docker Desktop)
- Docker-compatible NAS (Synology, QNAP, etc.)

---

## ⚡ Quick Installation

### 1. Clone the Repository

```bash
git clone https://git.dracodev.net/Projets/ObsiGate.git
cd ObsiGate
```

### 2. Configure Your Vaults and Secrets

Edit the `docker-compose.yml` file to add your Obsidian vaults:

```yaml
volumes:
  - /absolute/path/to/your/vault:/vaults/YourVaultName:ro
```

> **Important** : The path must be absolute and the volume must be read-only (`:ro`)

Create your `.env` file for authentication and secrets:

```bash
cp .env.example .env
# Edit .env to configure your passwords and options
```

> **Never commit `.env`!** It is in `.gitignore`. Use `.env.example` as a reference.

### 3. Launch the Application

```bash
# Make the script executable (once)
chmod +x build.sh

# Build + deploy in one command
./build.sh
```

> **That's it!** `build.sh` checks Docker, validates your volumes, builds the image, and starts the container automatically.
>
> Useful options: `./build.sh --help` for help, `./build.sh --cache` for a faster rebuild, `./build.sh --build-only` to build without starting.

### 4. Access the Interface

Open your browser at: **http://localhost:2020**

---

## ⚙️ Detailed Configuration

### Step 1: Prepare Your Vaults

1. **Locate your Obsidian vaults** on your system
2. **Note the absolute paths** to each `.obsidian` folder
3. **Check permissions** : Docker must be able to read these folders

### Step 2: Configure docker-compose.yml

Here's a complete example:

```yaml
services:
  obsigate:
    build:
      context: .
    image: obsigate:latest
    container_name: obsigate
    restart: unless-stopped
    ports:
      - "2020:8080"  # Local port 2020 → Container port 8080
    volumes:
      # Vault examples (adapt to your paths)
      - /home/user/Documents/Obsidian-Recipes:/vaults/Recipes:ro
      - /home/user/Documents/Obsidian-IT:/vaults/IT:ro
      - /home/user/Documents/Obsidian-Personal:/vaults/Personal:ro
      # Auth data persistence
      - ./data:/app/data
    environment:
      # Vault configuration
      - VAULT_1_NAME=Recipes
      - VAULT_1_PATH=/vaults/Recipes
      - VAULT_2_NAME=IT
      - VAULT_2_PATH=/vaults/IT
      - VAULT_3_NAME=Personal
      - VAULT_3_PATH=/vaults/Personal
      # Auth (secrets are in .env)
      - OBSIGATE_AUTH_ENABLED=true
      - OBSIGATE_ADMIN_USER=admin
    env_file:
      - .env  # Contains OBSIGATE_ADMIN_PASSWORD and other secrets
```

### Step 3: Build & Deploy

  ```bash
  # Use the automated build script (recommended)
  chmod +x build.sh
  ./build.sh
  ```

  The `build.sh` script handles everything: prerequisite checks, volume validation, image building, and startup.

  **Manual alternative:**
  ```bash
  docker compose build --no-cache
  docker compose up -d
  ```

> **Docker Compatibility** : The image uses the minimal `uvicorn` variant and a compatible `fastapi` version (`0.110.3`) to avoid certain optional native dependencies (`watchfiles`, `uvloop`, `httptools`, `fastapi-cli`, etc.) that may fail to build on some platforms like Alpine, ARM, or i386.

---

## 🌍 Environment Variables

Vaults are configured using pairs of `VAULT_N_NAME` / `VAULT_N_PATH` variables (N = 1, 2, 3…):

| Variable | Description | Example |
|----------|-------------|---------|
| `VAULT_1_NAME` | Display name of the vault | `Recipes` |
| `VAULT_1_PATH` | Path inside the container | `/vaults/Obsidian-RECIPES` |
| `VAULT_1_ATTACHMENTS_PATH` | Relative path to the attachments folder (optional) | `06_Toolbox/6.2_Attachments` |
| `VAULT_1_SCAN_ATTACHMENTS` | Enable image scanning on startup (optional, default: true) | `true` |
| `VAULT_2_NAME` | Display name of the vault | `IT` |
| `VAULT_2_PATH` | Path inside the container | `/vaults/Obsidian_IT` |

**Naming rules:**
- Use only letters, numbers, and hyphens
- No spaces or special characters
- The name must match the path inside the container

---

## 🔒 Authentication

> **Disabled by default** — Compatible with all existing installations.

ObsiGate supports an optional authentication system based on **JWT + Argon2id** with per-vault access control.

### Enable Authentication

1. **Copy the `.env` template** :
   ```bash
   cp .env.example .env
   ```

2. **Edit `.env`** and uncomment/configure the variables :
   ```bash
   OBSIGATE_AUTH_ENABLED=true
   OBSIGATE_ADMIN_USER=admin
   OBSIGATE_ADMIN_PASSWORD=your_password  # Leave empty = auto-generated (see logs)
   # OBSIGATE_SECURE_COOKIES=false              # true if behind HTTPS
   ```

3. **In `docker-compose.yml`**, make sure you have :
   ```yaml
   env_file:
     - .env
   ```

> **Never put passwords in `docker-compose.yml`!** Always use `.env`.

### First Startup

If no user exists, ObsiGate automatically creates an admin account and **displays the password in the logs** :

```bash
docker-compose logs obsigate | grep -A4 "FIRST"
```

```
============================================================
FIRST STARTUP — Admin account created automatically
  Username : admin
  Password : xK9mQ3pLr7wN2jT5
CHANGE THIS PASSWORD on first login!
============================================================
```

### User Management via CLI

```bash
# Create a user
docker exec obsigate python backend/create_admin.py create alice MyPassword --role user --vaults Recipes IT

# Create an admin with full access
docker exec obsigate python backend/create_admin.py create bob SecretPass --role admin --vaults "*"

# List users
docker exec obsigate python backend/create_admin.py list

# Delete a user
docker exec obsigate python backend/create_admin.py delete alice
```

### Admin Interface

When an **admin** account is logged in, a 🛡️ icon appears in the header. Clicking it opens the admin panel allowing you to:

- List all users
- Create / edit / delete users
- Assign accessible vaults per user
- Enable/disable accounts

### Per-Vault Access Control

| vaults value | Access |
|--------------|--------|
| `["*"]` | All vaults (including future ones) — admin default |
| `["Recipes", "IT"]` | Only these vaults |
| `[]` | No access |

### Auth Environment Variables

| Variable | Description | Default |
|----------|-------------|---------|
| `OBSIGATE_AUTH_ENABLED` | Enable authentication | `false` |
| `OBSIGATE_ADMIN_USER` | Auto-created admin username | `admin` |
| `OBSIGATE_ADMIN_PASSWORD` | Admin password (empty = auto-generated) | *(auto)* |
| `OBSIGATE_SECURE_COOKIES` | `Secure` cookie (HTTPS only) | `false` |
| `OBSIGATE_ACCESS_TOKEN_TTL` | JWT token lifetime (seconds) | `3600` |
| `OBSIGATE_REFRESH_TOKEN_TTL` | Refresh token lifetime (seconds) | `2592000` |
| `OBSIGATE_LOGIN_MAX_ATTEMPTS` | Max login attempts per IP | `10` |
| `OBSIGATE_ACCOUNT_MAX_ATTEMPTS` | Max login attempts per account | `10` |
| `OBSIGATE_LOGIN_WINDOW_SECONDS` | Rate limiting window (seconds) | `900` |
| `OBSIGATE_TRUST_PROXY` | Trust `X-Forwarded-For` for the client IP (reverse proxy) | `false` |
| `OBSIGATE_WEBHOOK_ALLOW_HTTP` | Allow non-HTTPS webhook targets | `false` |
| `OBSIGATE_WEBHOOK_ALLOW_PRIVATE` | Allow webhooks to private/loopback addresses | `false` |
| `OBSIGATE_PDF_MAX_SIZE_MB` | Max PDF size for text extraction | `50` |
| `OBSIGATE_PDF_EXTRACT_TIMEOUT` | PDF extraction timeout (seconds) | `30` |

>All these variables are documented in `.env.example`.

### Volume for Persistence

Auth data (`users.json`, `secret.key`) is stored in `/app/data`. Add a volume to persist it:

```yaml
volumes:
  # vaults...
  - ./data:/app/data  # Persist users and JWT key
```

---

## ➕ Adding a New Vault

### Method 1: Direct Editing

1. **Stop the container** :
   ```bash
   docker-compose down
   ```

2. **Add a volume** in `docker-compose.yml` :
   ```yaml
   volumes:
     - /new/vault/path:/vaults/NewVault:ro
   ```

3. **Add the environment variables** :
   ```yaml
   environment:
     - VAULT_4_NAME=NewVault
     - VAULT_4_PATH=/vaults/NewVault
   ```

4. **Restart** :
   ```bash
   ./build.sh
   ```

### Method 2: Hot-reload (recommended)

1. **Add the volume and variables** as above
2. **Apply the changes** :
   ```bash
   ./build.sh
   ```
3. **Reload the index** via the interface or the API :
   ```bash
   curl http://localhost:2020/api/index/reload
   ```

### Method 3: Dynamic API (without restart)

Add a vault on the fly via the API (the volume must already be mounted):
```bash
curl -X POST http://localhost:2020/api/vaults/add \
  -H "Content-Type: application/json" \
  -d '{"name": "NewVault", "path": "/vaults/NewVault"}'
```

Remove a vault:
```bash
curl -X DELETE http://localhost:2020/api/vaults/NewVault
```

---

## 🔨 Build & Deployment with build.sh

The `build.sh` script is the recommended way to build and deploy ObsiGate.

### Basic Usage

```bash
chmod +x build.sh   # once
./build.sh          # build from scratch + start
```

### Available Options

| Option | Description |
|--------|-------------|
| `--help`, `-h` | Display full help |
| `--build-only` | Build the image without starting the container |
| `--no-cache` | Full rebuild without Docker cache **(default)** |
| `--cache` | Use Docker cache (faster if few changes) |
| `--progress=plain` | Verbose output (recommended for debugging) |
| `--progress=tty` | Interactive output with progress bars |

### What the Script Does

1. **Checks** that Docker and Docker Compose are installed (with version numbers)
2. **Validates** the presence and syntax of the `docker-compose.yml` file
3. **Checks each mounted volume** : warns if a source directory does not exist
4. **Builds** the Docker image via `docker compose build` (multi-stage, ~180MB)
5. **Starts** the container via `docker compose up -d`
6. **Displays** the container status then real-time logs

### Examples

```bash
# Clean build + start (recommended after code changes)
./build.sh

# Fast rebuild with cache (minor frontend changes only)
./build.sh --cache

# Test the build without starting (useful for pre-prod testing)
./build.sh --build-only

# Verbose output to debug a build failure
./build.sh --progress=plain
```

### Stopping / Restarting

```bash
docker compose down      # Stop the container
docker compose up -d     # Restart without rebuild
docker compose logs -f   # View logs
```

---

## 🖼️ Obsidian Image Rendering

ObsiGate supports **all Obsidian image syntaxes** with an intelligent multi-strategy resolution system.

### Supported Syntaxes

1. **Standard Markdown with HTML attributes** (Obsidian-compatible) :
   ```markdown
   [<img width="180" height="60" src="path/to/image.svg"/>](https://example.com)
   ```

2. **Wiki-link embed with full path** :
   ```markdown
   ![[06_Toolbox/6.2_Attachments/image.svg]]
   ```

3. **Wiki-link embed with filename only** :
   ```markdown
   ![[image.svg]]
   ```

4. **Standard Markdown** :
   ```markdown
   ![alt text](path/to/image.png)
   ```

### Intelligent Path Resolution

ObsiGate uses 7 resolution strategies in order of priority:

1. **Absolute path** : If the path is absolute and exists
2. **Configured attachments folder** : Via `VAULT_N_ATTACHMENTS_PATH`
3. **Startup index (unique match)** : Search by filename in the index
4. **Same directory** : Relative to the current markdown file
5. **Vault root** : Relative to the vault root
6. **Startup index (closest match)** : If multiple files have the same name
7. **Fallback** : Display a styled placeholder `[image not found: filename.ext]`

### Configuration

To optimize resolution, configure the attachments folder for each vault:

```yaml
environment:
  - VAULT_1_NAME=MyVault
  - VAULT_1_PATH=/vaults/MyVault
  - VAULT_1_ATTACHMENTS_PATH=Assets/Images  # Relative path
  - VAULT_1_SCAN_ATTACHMENTS=true           # Enable scanning (default)
```

### Manual Rescan

To rescan images in a vault after adding/removing:

```bash
curl -X POST http://localhost:2020/api/attachments/rescan/MyVault
```

---

## 🖥️ Desktop (Tauri) — Native Application

ObsiGate Desktop is a native application built with [Tauri](https://tauri.app/) (Rust + system webview). It embeds the Python backend and frontend in a standalone executable — zero Docker, zero command line.

> 🚧 **Version 2.0.0 — binaries are being stabilized.** For now, building from source is recommended.

### Native Desktop Features

| Feature | Web | Desktop |
|---|---|---|
| Local file access | Via upload | Native (folder picker) |
| System theme | Manual | Auto (follows OS dark/light) |
| Notifications | Service Worker | Native OS |
| `.md` associations | ❌ | ✅ "Open with ObsiGate" |
| Tray icon | ❌ | ✅ Taskbar |
| Auto-update | ❌ | ✅ Checks Gitea releases |
| Offline mode | Limited | Full (local backend) |

### Download (Pre-built Binaries)

Releases are published on [Gitea](https://git.dracodev.net/Projets/ObsiGate/releases) :

| Platform | Format |
|---|---|
| **Linux** | `.deb` + `.AppImage` |
| **Windows** | `.msi` + `.exe` (NSIS) |

#### Linux

```bash
# .deb (Debian/Ubuntu/Deepin)
sudo dpkg -i obsigate_2.0.0_amd64.deb
# Launch : ObsiGate from the applications menu or `obsigate-desktop`

# .AppImage (any distro)
chmod +x ObsiGate_2.0.0_amd64.AppImage
./ObsiGate_2.0.0_amd64.AppImage
```

#### Windows

```cmd
:: Double-click ObsiGate_2.0.0_x64.msi
:: Or launch ObsiGate from the Start menu
```

### Getting Started

1. **Launch the application** from the menu or command line
2. The Python backend starts automatically on `127.0.0.1:17890`
3. The window opens and loads the ObsiGate interface
4. **First launch** : select your Obsidian vaults folder via the native picker
5. To close : tray icon → Quit (clean backend shutdown)

### Build from Source

Detailed guide: [desktop/README.md](./desktop/README.md).

#### Common Prerequisites

| Tool | Version | Installation |
|---|---|---|
| Rust (cargo) | ≥ 1.75 | `rustup` |
| Tauri CLI | ≥ 2.0 | `cargo install tauri-cli` |
| Git | — | — |
| Linux system dependencies | — | `sudo apt install libwebkit2gtk-4.1-dev libgtk-3-dev libayatana-appindicator3-dev` |

> **Important — staging :** `tauri.conf.json` embeds `backend/**` and `frontend/**` **from the `desktop/` folder**.
> The build scripts automatically copy `../backend` and `../frontend` into `desktop/` before `cargo tauri build`.
> Without this staging, the build fails with "glob pattern backend/**/* path not found".

#### 🪟 Windows — `build-windows.bat`

```cmd
REM Prerequisites (via Scoop) : rustup, curl, git
scoop install rustup curl git
rustup default stable
cargo install tauri-cli

cd desktop
build-windows.bat
```

Script steps:

1. Kills residual Python processes (`taskkill /F /IM python.exe`)
2. Downloads **Python 3.11 embed** (python.org) → `desktop\python-embed\` + enables pip (`python311._pth`)
3. `pip install -r ..\backend\requirements.txt` in the embed
4. **Staging** : copies `..\backend` and `..\frontend` to `desktop\`
5. `cargo tauri build --target x86_64-pc-windows-msvc --bundles nsis`
6. Copies `python-embed` next to the executable (`target\x86_64-pc-windows-msvc\release\`) for local dev mode
7. Cleans the staged folders

→ **Artifact :** `desktop\target\x86_64-pc-windows-msvc\release\bundle\nsis\ObsiGate_2.0.0_x64-setup.exe`

#### 🐧 Linux — `build-linux.sh`

```bash
cd desktop
chmod +x build-linux.sh
./build-linux.sh
```

Script steps:

1. Checks Rust + Tauri CLI, installs system dependencies (apt)
2. Creates a Python venv `desktop/python-embed/venv` + `pip install -r ../backend/requirements.txt`
3. **Staging** : copies `../backend` and `../frontend` to `desktop/`
4. `cargo tauri build --target x86_64-unknown-linux-gnu --bundles deb,appimage`
5. Copies the runtime (`python-embed/`, `backend/`, `frontend/`) next to the executable

→ **Artifacts :**

- `desktop/target/x86_64-unknown-linux-gnu/release/bundle/deb/obsigate_2.0.0_amd64.deb`
- `desktop/target/x86_64-unknown-linux-gnu/release/bundle/appimage/ObsiGate_2.0.0_amd64.AppImage`

### 🤖 CI/CD Builds — Automatic Artifacts

**Yes** — the workflow [`.gitea/workflows/desktop-build.yml`](./.gitea/workflows/desktop-build.yml) builds desktop binaries on every push to `main` touching `desktop/**`, `frontend/**`, or `backend/**` (and manually via `workflow_dispatch`), on **self-hosted runners** :

| Job | Runner | Artifacts (retained 30 days) |
|---|---|---|
| `build-windows` | `[self-hosted, windows, desktop]` | `desktop/target/release/bundle/msi/*.msi` |
| `build-linux` | `[self-hosted, linux, desktop]` | `*.AppImage` + `*.deb` |

- Artifacts are downloadable from the **Actions** page of the Gitea run.
- Publishing to **Gitea Release** is planned for tags `v*` (`Publish to Gitea Release` step).
- The web workflow [`.gitea/workflows/ci.yml`](./.gitea/workflows/ci.yml) separately handles lint → tests → security → Docker build → e2e Playwright.

### Desktop Architecture

```
┌────────────────────────────────────────────┐
│ Tauri (Rust)                               │
│  ├─ Webview (system webview)              │
│  │   └─ Frontend (HTML/JS/CSS)            │
│  └─ Sidecar Python                         │
│      └─ uvicorn backend.main:app          │
│           └─ port 127.0.0.1:17890         │
└────────────────────────────────────────────┘
```

Lifecycle: Tauri spawns the Python backend → health check → opens the webview. On close: SIGTERM → clean shutdown → cleanup.

---

## 📖 Usage

### Web Interface

1. **Navigation** : Click on vaults in the sidebar to expand them
2. **Search** : Use the search bar to search across all vaults
3. **Tags** : Click on tags to filter content
4. **Wikilinks** : `[[page]]` links are clickable and navigable
5. **Images** : All Obsidian image syntaxes are rendered automatically
6. **Theme** : Toggle between light/dark theme with the 🌙/☀️ icon

### Keyboard Shortcuts

| Action | Shortcut |
|--------|----------|
| Search | `Ctrl + K` or `/` |
| Toggle theme | `Ctrl + T` |
| Focus search | `Esc` |

---

## 👥 Real-time Collaboration

Multiple users can edit the same markdown document simultaneously (Google Docs style):

- **Conflict-free merge** via Yjs (CRDT): two people can type in the same place, no change is lost.
- **Colored remote cursors** and visible selections in CodeMirror, labelled with each user's name.
- **Presence indicator** in the editor header (avatars + connection status).
- **Automatic reconnection** (exponential backoff): state is merged on return.
- **Server-side persistence**: the document is written to disk 2 s after the last change.
- **Transport**: WebSocket `ws(s)://<host>/ws/collab/{vault}/{path}`, authenticated via the
  `access_token` cookie (or `?token=`) and subject to per-vault access control.

No configuration is required: open the same file in two browsers (or two windows) to see it live.

---

## 🔌 API

ObsiGate exposes a complete REST API :

| Endpoint | Description | Method | Auth |
|----------|-------------|--------|------|
| `/api/health` | Health check (status, version, stats) | GET | No |
| `/api/auth/status` | Auth status (enabled, users present) | GET | No |
| `/api/auth/login` | Login (returns access token + refresh cookie) | POST | No |
| `/api/auth/refresh` | Refresh access token via refresh cookie | POST | Cookie |
| `/api/auth/logout` | Logout + revoke refresh token | POST | Yes |
| `/api/auth/me` | Current user info | GET | Yes |
| `/api/auth/change-password` | Change password | POST | Yes |
| `/api/auth/admin/users` | List / create users | GET/POST | Admin |
| `/api/auth/admin/users/{u}` | Modify / delete a user | PATCH/DELETE | Admin |
| `/api/vaults` | List vaults (filtered by permissions) | GET | Yes |
| `/api/browse/{vault}?path=` | Browse folders | GET | Yes |
| `/api/file/{vault}?path=` | Rendered content of a file | GET | Yes |
| `/api/file/{vault}/raw?path=` | Raw content of a file | GET | Yes |
| `/api/file/{vault}/download?path=` | Download a file | GET | Yes |
| `/api/file/{vault}/save?path=` | Save a file | PUT | Yes |
| `/api/file/{vault}?path=` | Delete a file | DELETE | Yes |
| `/api/search/advanced` | Advanced TF-IDF search (+ `semantic=true` for hybrid) | GET | Yes |
| `/api/suggest` / `/api/tags/suggest` | Autocomplete | GET | Yes |
| `/api/tags?vault=` | Unique tags with counters | GET | Yes |
| `/api/index/reload` | Force a rescan of vaults | GET | Admin |
| `/api/events` | Real-time SSE stream | GET | Yes |
| `/api/vaults/add` / `/api/vaults/{name}` | Dynamic vault management | POST/DELETE | Admin |
| `/api/image/{vault}?path=` | Serve an image | GET | Yes |
| `/api/config` | Read / write configuration | GET/POST | Yes/Admin |
| `/api/diagnostics` | Index and memory statistics | GET | Admin |

> When `OBSIGATE_AUTH_ENABLED=false`, all endpoints are accessible without a token.

> All endpoints expose documented Pydantic schemas. Interactive docs are available at `/docs` (Swagger UI).

**Usage Example :**
```bash
# Health check
curl http://localhost:2020/api/health

# List vaults
curl http://localhost:2020/api/vaults

# Simple search (legacy)
curl "http://localhost:2020/api/search?q=recipe&vault=all"

# Advanced search with TF-IDF, facets, and pagination
curl "http://localhost:2020/api/search/advanced?q=recipe%20tag:cuisine&vault=all&limit=20&offset=0&sort=relevance"

# Title autocomplete
curl "http://localhost:2020/api/suggest?q=piz&vault=all"

# Tag autocomplete
curl "http://localhost:2020/api/tags/suggest?q=rec&vault=all"

# Get a file
curl "http://localhost:2020/api/file/Recipes?path=pizza.md"
```

## 🔍 Advanced Search

### Query Syntax

| Operator | Description | Example |
|----------|-------------|---------|
| `tag:<name>` | Filter by tag | `tag:recipe docker` |
| `#<name>` | Tag shortcut | `#linux server` |
| `vault:<name>` | Filter by vault | `vault:IT kubernetes` |
| `title:<text>` | Filter by title | `title:pizza` |
| `path:<text>` | Filter by path | `path:recipes/soups` |
| `ext:<type>` | Filter by file type | `ext:md kubernetes` |
| `"exact phrase"` | Phrase search | `tag:"multiple words"` |

Extension filter examples: `ext:sh` for bash scripts, `ext:py` for Python scripts, `ext:md` for Markdown files, `ext:pdf` for PDF documents, `ext:excalidraw` for Excalidraw diagrams (text-extracted content is indexed).

### PDF support

PDF files in your vaults are rendered inline in the browser via the native PDF viewer (iframe + `<embed>`).
The viewer streams the file over HTTP Range requests (206 Partial Content), so large PDFs load progressively.
Text is extracted on indexing (pypdf / pymupdf) so PDF content is searchable via the full-text search.
Filter with `ext:pdf` to restrict results to PDFs.
PDF metadata (pages, title, author) is available via `GET /api/file/{vault}/pdf/info` without transferring the document.

**Limitations:** no OCR (scanned PDFs aren't searchable), no annotation, no editing of the PDF itself.

### Excalidraw diagrams

`.excalidraw` and `.excalidraw.md` files open in a full visual Excalidraw editor embedded in a sandboxed iframe — draw, edit and save without leaving ObsiGate.
Changes are saved automatically (2s debounce) or with `Ctrl+S`; the editor follows the light/dark theme.
Text labels inside the diagram elements are extracted on indexing, so diagram content is searchable via the full-text search (`ext:excalidraw`).
Files created with the **Obsidian Excalidraw plugin** (including the compressed `.excalidraw.md` format) are compatible.

Operators are combinable: `tag:linux vault:IT ext:md server web` searches for "server web" in Markdown files of the IT vault with the linux tag.

### Keyboard Shortcuts

| Shortcut | Action |
|----------|--------|
| `Ctrl+K` / `Cmd+K` | Focus search bar |
| `/` | Focus search (outside text fields) |
| `↑` / `↓` | Navigate suggestions |
| `Enter` | Select active suggestion or launch search |
| `Escape` | Close suggestions / exit search |

### Features

- **TF-IDF** : Scoring based on term frequency weighted by inverse document frequency
- **Title boost** : Title matches receive 3× higher score
- **Accent normalization** : `resume` finds `résumé`, `elephant` finds `éléphant`
- **Highlighted snippets** : Found terms are wrapped in `<mark>` in excerpts
- **Facets** : Counters by vault and by tag in results
- **Pagination** : Page navigation with 50 results per page
- **Sorting** : By relevance (TF-IDF) or modification date
- **Visual chips** : Active filters are shown as removable colored chips
- **History** : Last 50 searches are stored in localStorage
- **Semantic search** (optional) : Toggle `~` (or `Alt+S`) fuses the TF-IDF ranking with an
  embedding-based ranking (RRF). Works out of the box with a dependency-free hashing embedder;
  install `backend/requirements-semantic.txt` and/or set `OBSIGATE_EMBEDDING_*` for real
  `all-MiniLM-L6-v2` embeddings. See [docs/features/semantic-search.md](docs/features/semantic-search.md).

---

## 🔧 Troubleshooting

### Common Issues

**Port already in use:**
```bash
# Check who is using the port
sudo netstat -tulpn | grep 2020
# Change the port in docker-compose.yml
ports:
  - "2021:8080"
```

**Vault not found:**
- Verify paths are absolute
- Ensure read permissions
- Restart the container after modifications

**Build fails:**
```bash
# Clean Docker cache and rebuild
docker system prune -f
docker compose down
./build.sh --progress=plain

# If failure persists, check build logs
./build.sh --progress=plain 2>&1 | tee build.log
```

**Logs for debugging:**
```bash
# Real-time logs
docker compose logs -f obsigate

# Detailed logs
docker compose logs --tail=100 obsigate
```

---

## ⚡ Performance

| Metric | Estimate |
|--------|----------|
| **Indexing** | ~1–2s for 1,000 markdown files |
| **Advanced Search** | < 10ms for most queries (inverted index + TF-IDF) |
| **Wikilink Resolution** | O(1) via lookup table |
| **Memory** | ~80–150MB per 1,000 files (content capped at 100 KB/file) |
| **Docker Image** | ~180MB (multi-stage, no build tools) |
| **CPU** | Non-blocking; search offloaded to dedicated thread pool |

### Recommended Settings by Vault Size

| Size | Files | `search_workers` | `prefix_max_expansions` | `max_content_size` |
|------|-------|------------------|--------------------------|---------------------|
| Small | < 500 | 1 | 50 | 100,000 |
| Medium | 500–5,000 | 2 | 50 | 100,000 |
| Large | 5,000+ | 4 | 30 | 50,000 |

These parameters are configurable via the interface (Settings) or the `/api/config` API.

### Key Optimizations (v1.2.0)

- **Inverted index with set-intersection** : Search uses posting lists for O(k × postings) retrieval instead of O(N) full scan
- **Prefix matching by binary search** : O(log V + k) instead of O(V) linear vocabulary scan
- **ThreadPoolExecutor** : CPU-bound search functions are offloaded from the asyncio event loop
- **Race condition guard** : `currentSearchId` + `AbortController` prevent stale result rendering
- **Progress bar** : Animated progress bar during search
- **Search timeout** : Automatic abort after 30s (configurable)
- **Query time display** : Server time shown in results (`query_time_ms`)
- **Staleness detection fix** : Generation counter instead of `id(index)` to detect index changes

### Optimizations v1.5.0 (Quick Wins 2026-05-27)

- **GZip Compression** : FastAPI middleware compresses all responses >1KB (~70% bandwidth saved)
- **Cache-Control immutable** : Static assets (`/static/*`) are cached by the browser for 1 year
- **`.dockerignore`** : Minimal Docker build context (no `.git`, `docs/`, `*.md`, `__pycache__`)
- **Incremental InvertedIndex** : `add_document`/`remove_document` hooks, no more O(N) rebuild on every mutation

### Optimizations v1.1.0

- **Search without I/O** : File content is cached in memory index
- **Multi-factor scoring** : exact title (+20), partial title (+10), path (+5), tag (+3), content frequency (x1 per occurrence, capped at 10)
- **Markdown renderer singleton** : The mistune renderer is instantiated once
- **AbortController** : Stale search requests are aborted client-side
- **Debounced icon rendering** : `lucide.createIcons()` is batched via `requestAnimationFrame`

---

## 🛡️ Security

- **Path traversal** : All file endpoints validate that the resolved path stays within the vault
- **Rate limiting** : 10 login attempts max per IP over 15 minutes + per-account lockout (5 attempts)
- **Audit log** : All writes, deletions, and config changes are logged in `data/audit.log` (JSON lines, 10 MB rotation)
- **Automatic backup** : Every file modification or deletion is saved in `.obsigate-backup/` with timestamp
- **Secret redaction** : Automatic masking of JWTs, API keys, tokens, and connection strings in previews
- **Non-root user** : The Docker container runs under user `obsigate` (UID 1000)
- **Read-only volumes** : Vaults are mounted as `:ro` by default in docker-compose
- **Secrets in `.env`** : Passwords and tokens are never in `docker-compose.yml`
- **Atomic writes** : Data files (users.json, shares.json, webhooks.json) use tmp+replace

---

## 🏗️ Tech Stack

- **Backend** : Python 3.11 + FastAPI 0.110 + Uvicorn
- **Auth** : python-jose (JWT HS256) + argon2-cffi (Argon2id)
- **Security** : Rate limiting (IP + account lockout), audit logging, secret redaction, atomic writes
- **File Watcher** : watchdog 4.x (native inotify + fallback polling)
- **Frontend** : Vanilla JS + HTML + CSS (zero framework, zero build)
- **Markdown Rendering** : mistune 3.x
- **PDF Export** : WeasyPrint 60+
- **Compression** : GZip middleware (FastAPI), Cache-Control immutable for assets
- **Docker Image** : python:3.11-slim (multi-stage), `.dockerignore` for minimal context
- **User Storage** : Local JSON (`data/users.json`) — no database
- **Architecture** : SPA + REST API + SSE

---

## 🏠 Architecture

```
┌─────────────────┐     ┌─────────────────────────────────────────┐
│   Browser       │◄───►│  FastAPI (backend/main.py)             │
│   (SPA)         │ REST │                                         │
│                 │      │  ┌──────────────┐  ┌──────────────┐  │
│  app.js         │      │  │ indexer.py  │  │  search.py    │  │
│  style.css      │      │  │ (scan+cache)│  │ (in-memory)  │  │
│  index.html     │      │  └───────┬──────┘  └──────┬───────┘  │
└─────────────────┘      │        │              │              │
                         │        └──────┬───────┘              │
                         │             │                        │
                         │    ┌────────┴─────────┐            │
                         │    │ In-Memory Index │            │
                         │    │ (files, tags,   │            │
                         │    │  content, lookup)│            │
                         │    └──────────────────┘            │
                         └─────────────────────────────────────────┘
         ┌───────────────────────────────┐
         │  Filesystem (mounted vaults)  │
         │  /vaults/Recipes  (ro)        │
         │  /vaults/IT         (ro)      │
         └───────────────────────────────┘
```

**Data Flow:**
1. On startup, `indexer.py` scans all vaults in parallel (thread pool)
2. Content, tags (YAML + inline), and metadata are cached in memory
3. An O(1) lookup table is built for wikilink resolution
4. `watcher.py` starts file monitoring (native watchdog or polling)
5. Detected changes trigger incremental index updates
6. Changes are notified to the frontend via Server-Sent Events (SSE)
7. Search queries use the in-memory index (zero disk I/O)
8. The SPA frontend communicates via REST + SSE and manages state client-side

---

## 📝 Development

### 🧪 Tests and CI — Mandatory Rules (Before Each Push)

> ⚠️ **Absolute rule — human contributors and AI agents**: never `git push`
> without having **locally** executed the entire sequence below with **100 %
> success**. Push is not meant to discover failures: CI is a safety net, not a
> diagnostic tool.

#### Local Checks Before Commit

| Step | Command | Equivalent CI Job |
|---|---|---|
| Backend lint | `ruff check backend/` | `lint` |
| Frontend import validation | `node tests/frontend/validate-imports.mjs` | `lint` |
| Frontend unit tests | `node tests/frontend/unit.test.mjs` | `lint` |
| Backend tests | `pytest tests/ -q` | `test` |
| **E2E Playwright** | `npm run test:e2e` (~5 min) | `e2e` |

#### Local E2E Tests (`npm run test:e2e`)

The `scripts/run-e2e-local.sh` script faithfully reproduces the `e2e` CI job
conditions (`.gitea/workflows/ci.yml`):

- backend started natively (venv `.venv-e2e`, Python 3.11 via `uv`) ;
- authentication disabled (`OBSIGATE_AUTH_ENABLED=false`) ;
- fixtures `test_vault` (TestVault) and `test_dir` (TestDir) ;
- port `2029` (baseURL from `playwright.config.ts`) ;
- Playwright project `chromium-desktop` only (same as CI).

Prerequisites: `uv`, Node.js ≥ 20, Playwright browsers
(`npx playwright install chromium`). The venv is created automatically on
first run. Docker is not required.

```bash
npm run test:e2e                                  # full suite
bash scripts/run-e2e-local.sh --headed            # visible browser
bash scripts/run-e2e-local.sh -g "reset panes"    # filter on a test
```

The suite must end with **all tests passing** (60 currently), with no failure
or reliance on retries. In case of failure: fix and re-run locally until 100 %,
then only commit.

#### Test Writing Rules (Lesson Learned)

- **Check every selector in the actual DOM** before using it in a test. Real
  bug encountered: a test was targeting `#cp-input` while the command palette
  input only has the `.cp-input` class (no id) — the test could not pass in any
  environment.
- Any new or modified test must be **run locally** (at least with
  `-g "test name"`) before push.
- No workarounds that mask flakiness (arbitrary `waitForTimeout`,
  silent fallbacks): fix the root cause.

#### Commits and Push

- Commit message: imperative subject ≤ 50 chars; body (72 col.) only if it
  adds useful information.
- One commit = one logical change (do not mix fix and refactor).
- Push only when all 5 local steps are green. CI pipeline:
  `lint → test → security → build → e2e`.

### Project Structure
```
ObsiGate/
├── backend/              # FastAPI API
│   ├── main.py          # Endpoints, Pydantic models, markdown rendering
│   ├── indexer.py       # Vault scanning, in-memory index, lookup table
│   ├── search.py        # Full-text search engine with scoring
│   ├── watcher.py       # File monitoring (watchdog + debounce)
│   ├── auth/            # Authentication module
│   │   ├── password.py  # Argon2id hashing
│   │   ├── jwt_handler.py # JWT tokens + revocation
│   │   ├── user_store.py  # users.json CRUD (atomic)
│   │   ├── middleware.py  # FastAPI auth dependencies
│   │   └── router.py    # /api/auth/* endpoints
│   ├── create_admin.py  # User management CLI
│   └── requirements.txt
├── frontend/            # Web interface (Vanilla JS, zero framework)
│   ├── index.html       # SPA page + login screen
│   ├── app.js           # SPA logic, AuthManager, AdminPanel
│   └── style.css        # Styles (CSS variables, themes, responsive)
├── data/                # Persistent data (created on startup)
│   ├── users.json       # Users (Argon2id hashed)
│   └── secret.key       # JWT secret key (512 bits)
├── Dockerfile           # Multi-stage, healthcheck, non-root
├── docker-compose.yml   # Deployment with healthcheck and auth env vars
├── build.sh             # Automated build & deployment (docker compose build + up)
└── docs/CONTRIBUTING.md # Contribution guide
```

### Contributing

See [CONTRIBUTING.md](docs/CONTRIBUTING.md) for code standards and
[docs/DELIVERY_WORKFLOW.md](docs/DELIVERY_WORKFLOW.md) for the mandatory delivery process.

---

## 📄 License

This project is licensed under the **MIT License** - see the [LICENSE](LICENSE) file for details.

---

## 🤝 Support

- **Issues** : [git.dracodev.net/Projets/ObsiGate/issues](https://git.dracodev.net/Projets/ObsiGate/issues)
- **Documentation** : [git.dracodev.net/Projets/ObsiGate/wiki](https://git.dracodev.net/Projets/ObsiGate/wiki)
- **Author** : Bruno Beloeil

---

## 📝 Changelog

See [CHANGELOG.md](./CHANGELOG.md) for the complete version history (v1.0.0 → v1.7.0).

---

*Project: ObsiGate | Version: 1.7.0 | Last updated: May 2026*