Add audit logging, rate limiting, secret redactor, and backlinks
Implement several security and feature improvements across the backend and frontend: - New IP-based rate limiter for authentication endpoints - New audit logging system for sensitive operations - New secret redactor to mask sensitive patterns in rendered content - Configurable token TTL and IGNORED_DIRS via environment variables - Add backlink index and API endpoint - Add preview tab support with single/double-click behavior in tree - Add file backup before write/delete operations
This commit is contained in:
+321
@@ -0,0 +1,321 @@
|
||||
# Code Context
|
||||
|
||||
## Files Retrieved
|
||||
|
||||
1. `C:/dev/git/python/ObsiGate/backend/main.py` (lines 1-2504) - Core API endpoints, markdown rendering, SSE
|
||||
2. `C:/dev/git/python/ObsiGate/backend/auth/router.py` (full file, 263 lines) - Auth endpoints (login, logout, refresh, admin CRUD)
|
||||
3. `C:/dev/git/python/ObsiGate/backend/auth/jwt_handler.py` (full file, 153 lines) - JWT token creation, validation, revocation
|
||||
4. `C:/dev/git/python/ObsiGate/backend/indexer.py` (full file, ~728 lines) - File indexing, vault config, file lookup
|
||||
5. `C:/dev/git/python/ObsiGate/backend/search.py` (full file, ~700 lines) - Full-text search, TF-IDF, suggestions
|
||||
6. `C:/dev/git/python/ObsiGate/frontend/app.js` (lines 1-8046) - Frontend SPA (TOC, tabs, tree, search)
|
||||
7. `C:/dev/git/python/ObsiGate/frontend/style.css` (lines 5379-5476) - Tab bar styles
|
||||
|
||||
---
|
||||
|
||||
## 1. Backend: `main.py`
|
||||
|
||||
### PUT endpoint for saving files
|
||||
|
||||
**Location:** Line **717** (`@app.put("/api/file/{vault_name}/save", response_model=FileSaveResponse)`)
|
||||
- Function: `api_file_save` (line **718**)
|
||||
- Body expects `{"content": "..."}`
|
||||
- Ends at line ~750 with return `{"status": "ok", "vault": vault_name, "path": path, "size": len(content)}`
|
||||
|
||||
### DELETE endpoint for files
|
||||
|
||||
**Location:** Line **753** (`@app.delete("/api/file/{vault_name}", response_model=FileDeleteResponse)`)
|
||||
- Function: `api_file_delete` (line **754**)
|
||||
- Path provided as query parameter: `path: str = Query(...)`
|
||||
- Also calls `remove_single_file()` and broadcasts SSE `file_deleted` event
|
||||
- Ends at line ~797
|
||||
|
||||
### `_heading_slugify` function
|
||||
|
||||
**Location:** Lines **476–503** (inside the Markdown rendering helpers section)
|
||||
```python
|
||||
def _heading_slugify(text: str) -> str:
|
||||
```
|
||||
- Matches the JavaScript `slugify()` exactly:
|
||||
1. Lowercase
|
||||
2. NFD normalize + strip combining marks
|
||||
3. Keep only Unicode letters, numbers, spaces, hyphens
|
||||
4. Spaces → hyphens, collapse multiple hyphens
|
||||
5. Strip leading/trailing hyphens, fallback to `"heading"`
|
||||
|
||||
### `_add_heading_ids` function
|
||||
|
||||
**Location:** Lines **506–527**
|
||||
- Post-processes HTML to inject `id=""` attributes on `<h1>`–`<h6>` tags
|
||||
- Handles duplicate slugs with `-2`, `-3` suffix
|
||||
|
||||
### Health endpoint
|
||||
|
||||
**Location:** Lines **562–571** (`@app.get("/api/health", response_model=HealthResponse)`)
|
||||
- Returns `{ status, version, vaults, total_files }`
|
||||
- No authentication required
|
||||
|
||||
### Markdown rendering pipeline (wikilinks)
|
||||
|
||||
- `_convert_wikilinks()`: lines **528–549** — converts `[[target]]` / `[[target|display]]` to clickable HTML anchors
|
||||
- `_render_markdown()`: lines **552–577** — master renderer: preprocesses images → converts wikilinks → renders with mistune → adds heading IDs
|
||||
- Wikilinks render as `<a class="wikilink" data-vault="..." data-path="...">` when resolved, `<span class="wikilink-missing">` otherwise
|
||||
|
||||
---
|
||||
|
||||
## 2. Backend: `auth/router.py`
|
||||
|
||||
### Login endpoint
|
||||
|
||||
**Location:** Line **97** (`@router.post("/login")`)
|
||||
- Function: `login` (line **98**)
|
||||
- Accepts `LoginRequest` with `username`, `password`, `remember_me`
|
||||
- Rate limiting via lockout:
|
||||
- `is_locked()` check at line **108** → returns 429 after too many failures
|
||||
- `record_login_failure()` at line **112** → increments failure counter
|
||||
- Lockout message: `"Compte temporairement verrouillé (15min)"` (line **109**)
|
||||
- Success: calls `create_access_token()` + `create_refresh_token()`, sets cookies
|
||||
|
||||
### Rate limiting
|
||||
|
||||
**Location:** Lines **108–117** (inside `login` endpoint)
|
||||
- **There is NO decorator-based or middleware rate limiting.** Rate limiting is manual, login-only:
|
||||
- Checks `is_locked()` (line **108**) — if true, raises HTTP 429
|
||||
- Calls `record_login_failure()` (line **112**) on bad password
|
||||
- Shows remaining attempts when <= 2 (line **114–115**)
|
||||
- Implementation lives in `backend/auth/user_store.py`:
|
||||
- `record_login_failure()` at line **142**
|
||||
- `is_locked()` at line **167**
|
||||
- No rate limiting on other endpoints (no slowapi, no middleware, no global limiter)
|
||||
|
||||
---
|
||||
|
||||
## 3. Backend: `auth/jwt_handler.py`
|
||||
|
||||
### JWT TTL / expiration settings
|
||||
|
||||
**Location:** Lines **22–23**
|
||||
```python
|
||||
ACCESS_TOKEN_EXPIRE_SECONDS = 3600 # 1 hour
|
||||
REFRESH_TOKEN_EXPIRE_SECONDS = 604800 # 7 days
|
||||
```
|
||||
- Algorithm: `HS256` (line **20**)
|
||||
- Secret key auto-generated to `data/secret.key` on first run (line **29**)
|
||||
- Refresh cookie max_age in router.py: 30 days if `remember_me`, else 7 days (line **131**)
|
||||
|
||||
### `create_access_token` function
|
||||
|
||||
**Location:** Lines **48–60**
|
||||
```python
|
||||
def create_access_token(user: dict) -> str:
|
||||
```
|
||||
- Payload: `{ sub, role, vaults, jti, iat, exp, type: "access" }`
|
||||
- Encoded with `jwt.encode()` using HS256 and the secret key from `get_secret_key()`
|
||||
|
||||
### `create_refresh_token` function
|
||||
|
||||
**Location:** Lines **63–73**
|
||||
- Returns `(token_string, jti)` tuple
|
||||
- Payload: `{ sub, jti, iat, exp, type: "refresh" }`
|
||||
- Uses `REFRESH_TOKEN_EXPIRE_SECONDS`
|
||||
|
||||
---
|
||||
|
||||
## 4. Backend: `indexer.py`
|
||||
|
||||
### IGNORED_DIRS or similar
|
||||
|
||||
**There is NO `IGNORED_DIRS` constant.** The indexer indexes **everything** including hidden files (starting with `.`). This is stated explicitly in the docstring at line **206**:
|
||||
> "All files and directories are indexed, including hidden files (starting with '.')."
|
||||
|
||||
Hidden-file filtering is handled at the **UI/browse level** via vault settings (`hideHiddenFiles`) in `main.py` and `vault_settings.py`.
|
||||
|
||||
### `vault_config` handling
|
||||
|
||||
**Location:** Lines **15–16** (global)
|
||||
```python
|
||||
vault_config: Dict[str, Dict[str, Any]] = {}
|
||||
```
|
||||
- Type: `{name: {path, attachmentsPath, scanAttachmentsOnStartup, type}}`
|
||||
- Populated by `load_vault_config()` at lines **50–104**
|
||||
- Reads `VAULT_N_NAME`/`VAULT_N_PATH` and `DIR_N_NAME`/`DIR_N_PATH` env vars
|
||||
- Also has `vault_config.update(load_vault_config())` in `build_index()` at line **312**
|
||||
|
||||
### Key data structures
|
||||
|
||||
- `index`: dict of vaults → `{files, tags, path, paths, config}` (line **11**)
|
||||
- `_file_lookup`: `{filename_lower: [{vault, path}, ...]}` — O(1) wikilink resolution (line **22**)
|
||||
- `path_index`: `{vault_name: [{path, name, type}, ...]}` — tree filtering (line **25**)
|
||||
- `_index_lock`: `threading.Lock()` (line **18**)
|
||||
- `_index_generation`: int counter for staleness detection (line **24**)
|
||||
|
||||
---
|
||||
|
||||
## 5. Backend: `search.py`
|
||||
|
||||
### Wikilink / backlink functions
|
||||
|
||||
**There are NO wikilink or backlink functions in `search.py`.** The file handles:
|
||||
- Full-text search with TF-IDF via `InvertedIndex` class (line **218**)
|
||||
- `advanced_search()` (line **426**) — supports operators: `tag:`, `vault:`, `title:`, `path:`, `ext:`
|
||||
- Title suggestions: `suggest_titles()` (line **594**)
|
||||
- Tag suggestions: `suggest_tags()` (line **620**)
|
||||
|
||||
**Wikilink resolution** lives in `backend/indexer.py` via `find_file_in_index()` (line **653**) using `_file_lookup`.
|
||||
**Wikilink rendering** lives in `backend/main.py` via `_convert_wikilinks()` (line **528**).
|
||||
**Backlinks do not exist** anywhere in the codebase — no function computes "what links to this file."
|
||||
|
||||
---
|
||||
|
||||
## 6. Frontend: `app.js`
|
||||
|
||||
### Tree item click handler (sidebar file opening)
|
||||
|
||||
**Primary location:** Lines **2349–2360** (inside `_renderDirectoryInContainer` during tree rendering)
|
||||
```javascript
|
||||
fileItem.addEventListener("click", () => {
|
||||
scrollTreeItemIntoView(fileItem, false);
|
||||
openFile(vaultName, item.path);
|
||||
closeMobileSidebar();
|
||||
});
|
||||
```
|
||||
|
||||
**Second location (search results):** Lines **2637–2642** — same pattern in a different tree-rendering path.
|
||||
**Third location (tree search filter results):** Lines **2790–2795** — filter results click handler.
|
||||
|
||||
### `openFile` function
|
||||
|
||||
**Location:** Lines **3085–3106** (original `openFile`)
|
||||
- Sets `currentVault`, `currentPath`, fetches `/api/file/{vault}?path={path}`
|
||||
- Calls `renderFile(data)` which builds breadcrumb, tags, action buttons, then renders HTML
|
||||
|
||||
**Overridden at line 7604:**
|
||||
```javascript
|
||||
openFile = function(vault, path) {
|
||||
TabManager.open(vault, path);
|
||||
};
|
||||
```
|
||||
This wraps the original to use tab-based navigation. `TabManager.open()` creates/focuses a tab.
|
||||
|
||||
### Tab management functions (TabManager)
|
||||
|
||||
**Location:** Lines **7234–7598** (`const TabManager = { ... }`)
|
||||
- `init()` — line **7243** — grabs DOM refs
|
||||
- `open(vault, path, options)` — line **7247** — opens a file in a new/focused tab
|
||||
- `activate(tabId)` — line **7274** — switches to a tab, saves/restores state
|
||||
- `close(tabId)` — line **7330** — closes a tab, switches to adjacent
|
||||
- `closeAll()` — line **7348** — closes all, shows dashboard
|
||||
- `closeRight(tabId)` — line **7358** — closes tabs to the right
|
||||
- `closeOthers(tabId)` — line **7374** — closes all except current
|
||||
- `moveTab(fromIdx, toIdx)` — line **7387** — drag-and-drop reorder
|
||||
- `_renderTabs()` — line **7438** — DOM rendering of tab bar with icons, names, close buttons, drag & drop
|
||||
- `_showTabContextMenu(x, y, tabId)` — line **7558** — right-click menu (Close, Close Others, Close Right, Close All)
|
||||
|
||||
### TOC `slugify` function
|
||||
|
||||
**Location:** Lines **766–776** (inside `OutlineManager`)
|
||||
```javascript
|
||||
slugify(text) {
|
||||
return text
|
||||
.toLowerCase()
|
||||
.normalize("NFD")
|
||||
.replace(/[\u0300-\u036f]/g, "")
|
||||
.replace(/[^\p{L}\p{N}\s-]/gu, "")
|
||||
.replace(/\s+/g, "-")
|
||||
.replace(/-+/g, "-")
|
||||
.trim() || "heading";
|
||||
}
|
||||
```
|
||||
|
||||
### Backlinks UI
|
||||
|
||||
**There is NO backlinks UI or functionality anywhere in the frontend.**
|
||||
- No `backlink` string found in `app.js`, `style.css`, or `index.html`
|
||||
- No "Links to this page" panel, no backlink section in the editor, no backlink search in the sidebar
|
||||
|
||||
---
|
||||
|
||||
## 7. Frontend: `style.css`
|
||||
|
||||
### Tab-related styles
|
||||
|
||||
**Location:** Lines **5379–5480** (`.tab-bar` through `.tab-drop-indicator`)
|
||||
- `.tab-bar` (line **5379**): flex container, 36px min-height, border-bottom
|
||||
- `.tab-bar[hidden]` (line **5389**): `display: none`
|
||||
- `.tab-list` (line **5393**): horizontal flex with overflow-x auto
|
||||
- `.tab-item` (line **5406**): padding 6px 12px, 0.8rem, border-right, transitions
|
||||
- `.tab-item:hover` (line **5424**): bg hover, color change
|
||||
- `.tab-item.active` (line **5429**): bg primary, bottom accent border
|
||||
- `.tab-item .tab-icon` (line **5436**): 14×14, flex-shrink
|
||||
- `.tab-item .tab-name` (line **5443**): overflow ellipsis, max-width 150px
|
||||
- `.tab-item .tab-close` (line **5449**): 16×16, hidden by default (opacity: 0)
|
||||
- `.tab-item:hover .tab-close, .tab-item.active .tab-close` (lines **5461–5462**): opacity 0.6
|
||||
- `.tab-item .tab-close:hover` (line **5466**): opacity 1
|
||||
- `.tab-item.dragging` (line **5471**): opacity 0.5
|
||||
- `.tab-drop-indicator` (line **5476**): 2px accent bar for drag-drop
|
||||
|
||||
### Sidebar tab styles (sidebar-tab, not content-tab)
|
||||
|
||||
**Location:** Lines **744–802**
|
||||
- `.sidebar-tabs` (line **745**)
|
||||
- `.sidebar-tab` (line **754**): uppercase, accent border on active
|
||||
- `.sidebar-tab-panel` (line **793**): display none, scrollable
|
||||
|
||||
---
|
||||
|
||||
## Architecture Summary
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ Frontend (app.js ~8000 lines) │
|
||||
│ ┌──────────┐ ┌──────────────┐ ┌─────────────────┐ │
|
||||
│ │ Sidebar │ │ Content Area │ │ Right Sidebar │ │
|
||||
│ │ Tree │ │ TabManager │ │ TOC/Outline │ │
|
||||
│ │ Tags │ │ renderFile() │ │ (slugify) │ │
|
||||
│ │ Filter │ │ Breadcrumbs │ │ ReadingProgress │ │
|
||||
│ └──────────┘ └──────────────┘ └─────────────────┘ │
|
||||
│ ← openFile() → TabManager.open() → api() → backend│
|
||||
└─────────────────────────────────────────────────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────────────────────────────────────────┐
|
||||
│ Backend (FastAPI) │
|
||||
│ main.py: │
|
||||
│ PUT /api/file/{vault}/save (line 717) │
|
||||
│ DELETE /api/file/{vault} (line 753) │
|
||||
│ GET /api/file/{vault} (rendered HTML) │
|
||||
│ GET /api/health (line 562) │
|
||||
│ _heading_slugify() (line 476) │
|
||||
│ _convert_wikilinks() (line 528) │
|
||||
│ │
|
||||
│ auth/router.py: │
|
||||
│ POST /api/auth/login (line 97) │
|
||||
│ Rate limiting: lockout only (lines 108-117) │
|
||||
│ │
|
||||
│ auth/jwt_handler.py: │
|
||||
│ ACCESS_TOKEN_EXPIRE_SECONDS = 3600 (line 22) │
|
||||
│ REFRESH_TOKEN_EXPIRE_SECONDS = 604800 (line 23) │
|
||||
│ create_access_token() (line 48) │
|
||||
│ │
|
||||
│ indexer.py: │
|
||||
│ vault_config {} (line 15-16) │
|
||||
│ load_vault_config() (line 50) │
|
||||
│ ⚠ No IGNORED_DIRS — indexes everything │
|
||||
│ │
|
||||
│ search.py: │
|
||||
│ ⚠ No wikilink/backlink functions │
|
||||
│ Wikilinks resolved via indexer.find_file_in_index│
|
||||
└─────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Start Here
|
||||
|
||||
For any feature work, start with **`C:/dev/git/python/ObsiGate/backend/main.py`** — it contains the API surface, markdown rendering (wikilinks, heading IDs, slugify), and all the endpoint definitions that tie the frontend to the backend index/search/auth subsystems.
|
||||
|
||||
### Key Findings / Gaps
|
||||
- **No rate limiting** except manual lockout on login
|
||||
- **No backlinks** — neither computed in backend nor displayed in frontend
|
||||
- **No IGNORED_DIRS** — the indexer indexes everything; hidden-file hiding is at the UI layer
|
||||
- **No wikilink/backlink in search.py** — wikilink resolution is in `indexer.py`, rendering in `main.py`
|
||||
- **TabManager** is a self-contained singleton at the end of `app.js` (line 7234) wrapping the original `openFile`
|
||||
Reference in New Issue
Block a user