Coalesce index generation increments and add rebuild cooldown
This commit is contained in:
+169
-262
@@ -1,312 +1,219 @@
|
||||
# Code Context — ObsiGate Roadmap Implementation
|
||||
# Code Context — ObsiGate Bug Investigation
|
||||
|
||||
## Files Retrieved
|
||||
|
||||
### Backend
|
||||
1. `backend/main.py` (2599 lines total) — main FastAPI app with all endpoints, Pydantic models, SSE manager
|
||||
2. `backend/vault_settings.py` (138 lines) — per-vault settings persistence (hideHiddenFiles, etc.)
|
||||
|
||||
### Frontend
|
||||
3. `frontend/app.js` (~8187 lines) — vanilla JS SPA, all UI logic
|
||||
4. `frontend/index.html` (1083 lines) — page structure with modals and dashboard
|
||||
1. `frontend/app.js` (lines 5585–5665) — `showWelcome()` rebuilds dashboard HTML with only bookmarks + recent sections
|
||||
2. `frontend/index.html` (lines 360–406) — Initial dashboard DOM has all 4 sections: stats, bookmarks, conflicts, recent
|
||||
3. `frontend/app.js` (lines 7640–7672) — `TabManager.activate()` hides dashboard; `_showDashboard()` restores it
|
||||
4. `frontend/app.js` (lines 7497–7597) — `TabManager.openPreview()` / `openPersistent()` definitions
|
||||
5. `frontend/app.js` (lines 7778–7830) — `TabManager._renderTabs()` — **missing `preview` CSS class**
|
||||
6. `frontend/style.css` (lines 5450–5457) — CSS rules for `.tab-item.preview` (italic)
|
||||
7. `frontend/app.js` (lines 2340–2376) — Tree click handlers in `_renderDirectoryInContainer`
|
||||
8. `frontend/app.js` (lines 3144–3310) — `renderFile()` overwrites `content-area.innerHTML`, destroying dashboard DOM
|
||||
9. `frontend/app.js` (lines 3347–3380) — `DashboardStatsWidget` relies on `dashboard-stats-grid` element
|
||||
10. `frontend/app.js` (lines 3381–3434) — `DashboardConflictsWidget` relies on `dashboard-conflicts-section` element
|
||||
11. `backend/search.py` (lines 239–425) — `InvertedIndex` class, `is_stale()`, `rebuild()`, `get_inverted_index()`
|
||||
12. `backend/indexer.py` (lines 28, 330–601, 634–667, 770–839) — `_index_generation` counter and all increment sites
|
||||
13. `backend/watcher.py` (lines 191–226) — Debounce: dispatches batched events after `debounce_seconds` (2s default)
|
||||
14. `backend/main.py` (lines 376–420) — `_on_vault_change()` calls `update_single_file` per event → each increments `_index_generation`
|
||||
15. `backend/search.py` (lines 670–700, 844–875) — `advanced_search()` and `suggest_titles()` call `get_inverted_index()`
|
||||
|
||||
---
|
||||
|
||||
## 1. Pydantic Models Section — `backend/main.py`
|
||||
## Issue 1: Dashboard Layout — Sections Disappearing
|
||||
|
||||
**Location: lines 56–284** — All Pydantic response/request models defined in a single block after imports and before SSE Manager.
|
||||
### Root Cause: `showWelcome()` Rebuild Kills Stats & Conflicts Sections
|
||||
|
||||
### Key models with Field descriptions (lines 60–69):
|
||||
```python
|
||||
class VaultInfo(BaseModel):
|
||||
name: str = Field(description="Display name of the vault")
|
||||
file_count: int = Field(description="Number of indexed files")
|
||||
tag_count: int = Field(description="Number of unique tags")
|
||||
type: str = Field(default="VAULT", description="Type of the vault mapping (VAULT or DIR)")
|
||||
**`showWelcome()`** (`frontend/app.js`, lines 5585–5665):
|
||||
- It checks if `dashboard-home` or `dashboard-bookmarks-section` don't exist.
|
||||
- If either is missing, it replaces `content-area.innerHTML` with **only two sections**: bookmarks and recent.
|
||||
- **Stats and Conflicts sections are NOT included** in this rebuilt HTML.
|
||||
|
||||
class BrowseItem(BaseModel): # line 72
|
||||
name: str
|
||||
path: str
|
||||
type: str = Field(description="'file' or 'directory'")
|
||||
```
|
||||
|
||||
### Models WITHOUT Field descriptions (need updating):
|
||||
- `FileContentResponse` — **lines 89–100**
|
||||
- `FileRawResponse` — **lines 103–107**
|
||||
- `FileSaveResponse` — **lines 110–115**
|
||||
- `FileDeleteResponse` — **lines 118–122**
|
||||
- `SearchResultItem` — **lines 125–133**
|
||||
- `SearchResponse` — **lines 136–143** (has Field on `total`, `offset`, `limit`)
|
||||
- `TagsResponse` — **lines 146–149**
|
||||
- `TreeSearchResult` — **lines 152–158**
|
||||
- `TreeSearchResponse` — **lines 161–165**
|
||||
- `AdvancedSearchResultItem` — **lines 168–176**
|
||||
- `SearchFacets` — **lines 179–182**
|
||||
- `AdvancedSearchResponse` — **lines 185–193** (has Field on `query_time_ms`)
|
||||
- `TitleSuggestion` — **lines 196–200**
|
||||
- `SuggestResponse` — **lines 203–206**
|
||||
- `TagSuggestion` — **lines 209–212**
|
||||
- `TagSuggestResponse` — **lines 215–218**
|
||||
- `GraphNode` — **lines 221–228**
|
||||
- `GraphEdge` — **lines 231–236**
|
||||
- `GraphResponse` — **lines 239–244**
|
||||
- `ReloadResponse` — **lines 247–250**
|
||||
- `HealthResponse` — **lines 253–257**
|
||||
- `DirectoryCreateRequest` — **lines 260–262** (has Field)
|
||||
- `DirectoryCreateResponse` — **lines 265–269**
|
||||
- `DirectoryRenameRequest` — **lines 272–274** (has Field)
|
||||
- `DirectoryRenameResponse` — **lines 277–281**
|
||||
- `DirectoryDeleteResponse` — **lines 284–288**
|
||||
- `FileCreateRequest` — **lines 291–293** (has Field)
|
||||
- `FileCreateResponse` — **lines 296–299**
|
||||
- `FileRenameRequest` — **lines 302–304** (has Field)
|
||||
- `FileRenameResponse` — **lines 307–311**
|
||||
|
||||
---
|
||||
|
||||
## 2. Dashboard/Stats Endpoint — `backend/main.py`
|
||||
|
||||
### `/api/diagnostics` — **lines 2500–2547**
|
||||
```python
|
||||
@app.get("/api/diagnostics") # line 2500
|
||||
async def api_diagnostics(current_user=Depends(require_admin)):
|
||||
```
|
||||
Returns index stats, inverted index stats, config, and search executor info. Requires admin auth.
|
||||
|
||||
### `/api/health` — **lines 1048–1059**
|
||||
```python
|
||||
@app.get("/api/health", response_model=HealthResponse) # line 1048
|
||||
async def api_health():
|
||||
```
|
||||
Public health check (no auth). Returns status, version, vaults count, total_files.
|
||||
|
||||
---
|
||||
|
||||
## 3. SSE File Event Broadcasting — `backend/main.py`
|
||||
|
||||
**SSE Manager class: lines 349–381** — `SSEManager` with `connect()`, `disconnect()`, and `broadcast()` methods.
|
||||
|
||||
**All `sse_manager.broadcast()` calls:**
|
||||
|
||||
| Line | Event Type | Context |
|
||||
|------|-----------|---------|
|
||||
| 413 | `index_updated` | File watcher callback (`_on_vault_change`) — partial index changes |
|
||||
| 506 | `index_<event_type>` | Background indexing progress |
|
||||
| 1252 | `file_deleted` | DELETE file endpoint |
|
||||
| 1330 | `directory_created` | POST create directory |
|
||||
| 1401 | `directory_renamed` | PATCH rename directory |
|
||||
| 1462 | `directory_deleted` | DELETE directory |
|
||||
| 1532 | `file_created` | POST create file |
|
||||
| 1607 | `file_renamed` | PATCH rename file |
|
||||
| 1949 | `index_reloaded` | Force reindex endpoint |
|
||||
| 2114 | `vault_reloaded` | (likely during vault reload) |
|
||||
| 2196 | `vault_added` | POST /api/vaults/add |
|
||||
| 2215 | `vault_removed` | DELETE /api/vaults/remove |
|
||||
|
||||
**SSE endpoint: lines 2127–2169** — `GET /api/events` returns `StreamingResponse` with `text/event-stream`.
|
||||
|
||||
**Frontend SSE client: `frontend/app.js` lines 5773–6015**
|
||||
- `IndexUpdateManager` (IIFE module)
|
||||
- Listens for events: `connected`, `index_updated`, `index_reloaded`, `vault_added`, `vault_removed`, `index_start`, `index_progress`, `index_complete`
|
||||
- Auto-reconnects with exponential backoff (1s → 30s max)
|
||||
- The `_onIndexUpdated()` handler (line 5912) refreshes sidebar tree and tags when affected vault matches context
|
||||
|
||||
**⚠️ Note:** The frontend SSE client does NOT currently listen for `file_created`, `file_deleted`, `file_modified`, `file_renamed`, `directory_created`, `directory_renamed`, `directory_deleted` events — these are broadcast by the backend but not consumed by the frontend. The frontend only reacts to `index_updated` (which already includes all changes).
|
||||
|
||||
---
|
||||
|
||||
## 4. Configurations Endpoint — `backend/main.py`
|
||||
|
||||
### Config storage — **lines 2414–2458**
|
||||
```python
|
||||
_CONFIG_PATH = _BASE_DIR / "data" / "config.json" # line 2414
|
||||
|
||||
_DEFAULT_CONFIG = { # line 2416
|
||||
"search_workers": 2,
|
||||
"debounce_ms": 300,
|
||||
"results_per_page": 50,
|
||||
"min_query_length": 2,
|
||||
"search_timeout_ms": 30000,
|
||||
"max_content_size": 100000,
|
||||
"snippet_context_chars": 120,
|
||||
"max_snippet_highlights": 5,
|
||||
"title_boost": 3.0,
|
||||
"path_boost": 1.5,
|
||||
"watcher_enabled": True,
|
||||
"watcher_use_polling": False,
|
||||
"watcher_polling_interval": 5.0,
|
||||
"watcher_debounce": 2.0,
|
||||
"tag_boost": 2.0,
|
||||
"prefix_max_expansions": 50,
|
||||
"recent_files_limit": 20,
|
||||
```js
|
||||
// line 5595 — only rebuilds if home or bookmarksSection is missing
|
||||
if (area && (!home || !bookmarksSection)) {
|
||||
area.innerHTML = `
|
||||
<div id="dashboard-home" ...>
|
||||
<div id="dashboard-bookmarks-section" class="dashboard-section">...</div>
|
||||
<div id="dashboard-recent-section" class="dashboard-section">...</div>
|
||||
</div>`;
|
||||
// NOTE: No #dashboard-stats-section or #dashboard-conflicts-section!
|
||||
}
|
||||
```
|
||||
|
||||
### `GET /api/config` — **line 2464**: Returns merged config (requires auth)
|
||||
### `POST /api/config` — **line 2470**: Updates config (requires admin), validates types against `_DEFAULT_CONFIG`
|
||||
**The original dashboard** (`frontend/index.html`, lines 360–406) has 4 sections:
|
||||
1. `#dashboard-stats-section`
|
||||
2. `#dashboard-bookmarks-section`
|
||||
3. `#dashboard-conflicts-section`
|
||||
4. `#dashboard-recent-section`
|
||||
|
||||
---
|
||||
**When does the dashboard get destroyed?**
|
||||
- `renderFile()` at line 3302: `area.innerHTML = "";` — this wipes `content-area` including `#dashboard-home` (which is a child of `content-area`).
|
||||
- After the file renders, `activate()` at line 7642 hides the dashboard: `dashboard.style.display = "none"` — but at this point the dashboard DOM elements have been **removed from the DOM**.
|
||||
- Later, when `_showDashboard()` (line 7764) is called:
|
||||
```js
|
||||
_showDashboard() {
|
||||
const area = document.getElementById("content-area");
|
||||
area.innerHTML = ""; // clear file content
|
||||
const dashboard = document.getElementById("dashboard-home");
|
||||
if (dashboard) {
|
||||
dashboard.style.display = ""; // show it
|
||||
area.appendChild(dashboard); // move back into DOM
|
||||
}
|
||||
}
|
||||
```
|
||||
This works **only if** `dashboard-home` still exists in the document. But `renderFile()` called `area.innerHTML = ""` which destroyed it.
|
||||
|
||||
## 5. Dashboard-Home Element and Rendering — `frontend/app.js`
|
||||
- Next time `showWelcome()` is called (e.g., after `goHome()`), it detects `!home` → rebuilds with **only 2 sections**.
|
||||
|
||||
### Dashboard DOM structure → `frontend/index.html` lines 341–392
|
||||
```html
|
||||
<div id="dashboard-home" class="dashboard-home" role="region" aria-label="Tableau de bord">
|
||||
<div id="dashboard-bookmarks-section" class="dashboard-section">...</div>
|
||||
<div id="dashboard-recent-section" class="dashboard-section">...</div>
|
||||
</div>
|
||||
**Why "Recently opened" stays visible when stats/bookmarks disappear:**
|
||||
- The rebuilt HTML in `showWelcome()` **includes** `#dashboard-recent-section` and `#dashboard-bookmarks-section`, but **not** `#dashboard-stats-section` or `#dashboard-conflicts-section`.
|
||||
- `DashboardStatsWidget.load()` (line 3348) looks for `document.getElementById("dashboard-stats-grid")` — if not found, it silently returns.
|
||||
- `DashboardConflictsWidget.load()` (line 3383) looks for `document.getElementById("dashboard-conflicts-section")` — if not found, silently returns.
|
||||
- So stats and conflicts sections vanish, while bookmarks and recent survive the rebuild.
|
||||
|
||||
### How the Dashboard Visibility Toggle Works
|
||||
- `TabManager.activate()` (line 7642): `dashboard.style.display = "none"` — hides dashboard when a file tab is shown.
|
||||
- `TabManager._showDashboard()` (line 7764): `dashboard.style.display = ""` + appends to `content-area` — shows dashboard when all tabs close.
|
||||
|
||||
### CSS for Dashboard Sections
|
||||
`frontend/style.css` (lines 4739–4742):
|
||||
```css
|
||||
.dashboard-section {
|
||||
display: flex;
|
||||
flex-direction: column;
|
||||
}
|
||||
```
|
||||
|
||||
### Dashboard regeneration fallback → `app.js` lines 5417–5482 (`showWelcome()`)
|
||||
When `dashboard-home` or its children are missing, `showWelcome()` rebuilds the entire HTML structure inline.
|
||||
|
||||
### Dashboard Recent Widget → `app.js` lines 3344–3580 (`DashboardRecentWidget`)
|
||||
- `load(vaultFilter)` — line 3347
|
||||
- `render()` — line 3410
|
||||
- `_createCard(file, index)` — line 3436
|
||||
- `showLoading()` — line 3397
|
||||
- `showEmpty()` — line 3526
|
||||
|
||||
### Dashboard Bookmarks Widget → `app.js` lines 3583–3660 (`DashboardBookmarkWidget`)
|
||||
- `load(vaultFilter)` — line 3587
|
||||
- `render()` — line 3613
|
||||
- `_createCard(file, index)` — around line 3628
|
||||
|
||||
### Dashboard visibility toggling:
|
||||
- Show: `app.js` line 7590–7593 — `dashboard.style.display = ""`
|
||||
- Hide: `app.js` line 7462–7464 — `dashboard.style.display = "none"`
|
||||
All `.dashboard-section` elements are just flex columns — there's no show/hide logic in CSS beyond what JavaScript sets via inline `style.display`.
|
||||
|
||||
---
|
||||
|
||||
## 6. Configurations/Settings Modal — `frontend/app.js`
|
||||
## Issue 2: Click/Double-Click Handling
|
||||
|
||||
### Modal initialization → **lines 3906–3990** (`initConfigModal()`)
|
||||
Event binding for open/close, config fields, save buttons, reindex, reset, diary refresh, hidden files.
|
||||
|
||||
### Config modal opening → **line 3914**:
|
||||
```javascript
|
||||
openBtn.addEventListener("click", async () => {
|
||||
modal.classList.add("active");
|
||||
renderConfigFilters();
|
||||
loadConfigFields(); // loads frontend+backend config
|
||||
loadDiagnostics(); // loads /api/diagnostics
|
||||
loadAbout(); // loads /api/health
|
||||
await loadHiddenFilesSettings();
|
||||
### Tree Click Handlers — Correctly Wired
|
||||
In `_renderDirectoryInContainer` (`frontend/app.js`, lines 2354–2360):
|
||||
```js
|
||||
fileItem.addEventListener("click", () => {
|
||||
scrollTreeItemIntoView(fileItem, false);
|
||||
TabManager.openPreview(vaultName, item.path); // single click → preview
|
||||
closeMobileSidebar();
|
||||
});
|
||||
fileItem.addEventListener("dblclick", (e) => {
|
||||
e.preventDefault();
|
||||
TabManager.openPersistent(vaultName, item.path); // double click → persistent
|
||||
});
|
||||
```
|
||||
Same pattern at lines 3132–3133 (sidebar recent), 4909–4910 and 5090–5091 (search results). **Wiring is correct.**
|
||||
|
||||
### Config field loading → **lines 4043–4070** (`loadConfigFields()`)
|
||||
Loads frontend config from localStorage and backend config from `GET /api/config`.
|
||||
### openPreview() and openPersistent() — Properly Defined
|
||||
- `openPreview()` (line 7497): Creates tab with `preview: true`, closes existing preview, activates.
|
||||
- `openPersistent()` (line 7533): If a preview tab exists for this file, converts it to persistent (`previewTab.preview = false`). If already persistent, just activates. Otherwise creates new persistent tab via `this.open()`.
|
||||
|
||||
### Diagnostics rendering → **lines 4157–4207** (`loadDiagnostics()`, `renderDiagnostics()`)
|
||||
Fetches `GET /api/diagnostics` and renders in `#config-diagnostics`.
|
||||
**Both are correctly implemented.**
|
||||
|
||||
### About section → **lines 4211–4250+** (`loadAbout()`)
|
||||
Fetches `GET /api/health`.
|
||||
### _renderTabs() — BUG: Missing `preview` CSS Class
|
||||
`_renderTabs()` at line 7792:
|
||||
```js
|
||||
el.className = "tab-item" + (tab.id === this._activeTabId ? " active" : "");
|
||||
```
|
||||
**The `preview` class is NEVER added.** The CSS has rules for `.tab-item.preview`:
|
||||
```css
|
||||
/* style.css line 5450 */
|
||||
.tab-item.preview .tab-name { font-style: italic; opacity: 0.75; }
|
||||
.tab-item.preview { background: var(--bg-hover); }
|
||||
```
|
||||
But since the JS never adds `el.classList.add("preview")` when `tab.preview` is true, **preview tabs do not render in italic**.
|
||||
|
||||
### Config Modal HTML → `frontend/index.html` lines 395–564
|
||||
All the config sections: search params, recent history, backend params, tag filtering, watcher, hidden files, diagnostics, about.
|
||||
The fix should be:
|
||||
```js
|
||||
el.className = "tab-item" + (tab.id === this._activeTabId ? " active" : "") + (tab.preview ? " preview" : "");
|
||||
```
|
||||
|
||||
Additionally, the `click` handler on tabs at line 7824 calls `this.activate(tab.id)` — activating a preview tab should work fine since `activate()` already properly handles loading file content.
|
||||
|
||||
---
|
||||
|
||||
## 7. File Action Buttons — `frontend/app.js`
|
||||
## Issue 3: Search Performance — "Rebuilding Inverted Index..."
|
||||
|
||||
### Button creation → **lines 3213–3260**
|
||||
All 6 action buttons created in `renderFile()`:
|
||||
### InvertedIndex Architecture
|
||||
`backend/search.py` (lines 239–425):
|
||||
- `InvertedIndex` is a singleton stored in module variable `_inverted_index` (line 415).
|
||||
- `is_stale()` (line 270): returns `True` when `_indexer._index_generation != self._source_generation`.
|
||||
- `rebuild()` (line 278): logs `"Rebuilding inverted index..."`, iterates over the **entire** `index` dict (all vaults, all files), rebuilds all internal structures from scratch. Sets `self._source_generation = _indexer._index_generation` at line 345.
|
||||
- `get_inverted_index()` (line 418): **check-then-rebuild every call**:
|
||||
```python
|
||||
def get_inverted_index() -> InvertedIndex:
|
||||
if _inverted_index.is_stale():
|
||||
_inverted_index.rebuild()
|
||||
return _inverted_index
|
||||
```
|
||||
|
||||
| Button | Line | Icon | Action |
|
||||
|--------|------|------|--------|
|
||||
| Copy | 3213 | `copy` | Copies raw content to clipboard (fetches if needed) |
|
||||
| Source | 3231 | `code` | Toggles raw source view |
|
||||
| Download | 3233 | `download` | Triggers file download via `/api/file/{vault}/download` |
|
||||
| Edit | 3244 | `edit` | Calls `openEditor(vault, path)` |
|
||||
| Pop-out | 3250 | `external-link` | Opens in new window via `/popout/{vault}/{path}` |
|
||||
| TOC | 3256 | `list` | Toggles right sidebar TOC |
|
||||
### What triggers `_index_generation` increments?
|
||||
`backend/indexer.py` — the counter increments at **7 sites**, every time the index mutates:
|
||||
| Line | Trigger |
|
||||
|------|---------|
|
||||
| 336 | `rebuild_index()` — full index rebuild on startup or manual reindex |
|
||||
| 375 | Per-vault processing within `rebuild_index()` (once per vault) |
|
||||
| 468 | `reindex_vault()` — single vault reindex |
|
||||
| 601 | `_remove_file_from_structures()` — single file deletion |
|
||||
| 667 | `_add_file_to_structures()` — single file addition |
|
||||
| 794 | `remove_vault_from_index()` — vault removal |
|
||||
| 839 | `add_vault_to_index()` — vault addition |
|
||||
|
||||
### Button assembly → **line 3300**:
|
||||
```javascript
|
||||
area.appendChild(el("div", { class: "file-header" }, [...,
|
||||
el("div", { class: "file-actions" }, [
|
||||
copyBtn, sourceBtn, downloadBtn, editBtn, openNewWindowBtn, tocBtn
|
||||
])
|
||||
]));
|
||||
```
|
||||
### Frequent Rebuild Trigger: File Watcher Hot-Reload
|
||||
`backend/main.py` (lines 376–420) — `_on_vault_change()`:
|
||||
- Each file watcher event (create/modify/delete/move) calls `update_single_file()`, `remove_single_file()`, or `handle_file_move()`.
|
||||
- These call `_remove_file_from_structures()` + `_add_file_to_structures()`, **each incrementing `_index_generation`**.
|
||||
- The watcher debounces for 2 seconds (`backend/watcher.py` line 100), then dispatches **all batched events at once** (line 207–210).
|
||||
- Each event in the batch still increments `_index_generation` independently.
|
||||
|
||||
---
|
||||
**Result**: If you save 5 files simultaneously, `_index_generation` jumps by up to 5. The next search/autocomplete call triggers a full inverted index rebuild. On a vault with thousands of files, each rebuild is expensive.
|
||||
|
||||
## 8. Vault Settings — `backend/vault_settings.py`
|
||||
### Which API calls trigger the rebuild?
|
||||
`get_inverted_index()` is called from:
|
||||
- `advanced_search()` at `search.py` line 684 (the TF-IDF advanced search)
|
||||
- `suggest_titles()` at `search.py` line 862 (autocomplete)
|
||||
- `main.py` line 2518 (stats/diagnostic endpoint)
|
||||
|
||||
**Full file: 138 lines** — Per-vault UI display preferences stored in `/app/data/vault_settings.json`.
|
||||
The basic `search()` function (line 430+) does **NOT** use `get_inverted_index()` — it scans the raw `index` dict directly. But `suggest_titles()` is called on every keystroke in the search bar.
|
||||
|
||||
### Exports used by `main.py`:
|
||||
```python
|
||||
from backend.vault_settings import get_vault_setting, update_vault_setting, get_all_vault_settings, delete_vault_setting
|
||||
```
|
||||
(imported at line 46 of `main.py`)
|
||||
### Why frequent saves cause repeated rebuilds
|
||||
1. User saves file → watcher fires `modified` event → `_on_vault_change` → `update_single_file` → `_index_generation += 1` (or +2 for add+remove).
|
||||
2. User types in search bar → `suggest_titles()` → `get_inverted_index()` → `is_stale()` returns `True` → **full rebuild**.
|
||||
3. User saves again → generation increments → next keystroke triggers another full rebuild.
|
||||
|
||||
### Key functions:
|
||||
- `get_vault_setting(vault_name)` — line 82 — returns settings dict or None
|
||||
- `update_vault_setting(vault_name, settings)` — line 93 — partial update, auto-saves
|
||||
- `get_all_vault_settings()` — line 125 — returns all vault settings
|
||||
- `delete_vault_setting(vault_name)` — line 111 — removes vault settings
|
||||
- Storage format: `{"vault_name": {"hideHiddenFiles": true/false}, ...}`
|
||||
|
||||
### Current usage in `main.py`:
|
||||
- `get_vault_setting(vault_name)` used in browse (line 776) and graph (line 1981) endpoints for `hideHiddenFiles`
|
||||
|
||||
### Config Modal Hidden Files → `app.js` (search for `loadHiddenFilesSettings`)
|
||||
Front-facing CRUD for per-vault `hideHiddenFiles` setting in the Configurations modal.
|
||||
This is the core performance issue: **single-file index mutations invalidate the entire inverted index**, and it gets rebuilt from scratch on the very next query.
|
||||
|
||||
---
|
||||
|
||||
## Architecture Summary
|
||||
|
||||
### Data Flow for Dashboard
|
||||
```
|
||||
index.html (full dashboard DOM)
|
||||
→ renderFile() destroys dashboard-home via innerHTML
|
||||
→ activate() hides remaining dashboard ref
|
||||
→ _showDashboard() tries to re-append dashboard-home
|
||||
→ showWelcome() rebuilds from scratch (only 2/4 sections)
|
||||
→ Stats/Conflicts widgets silently no-op (target elements missing)
|
||||
```
|
||||
backend/main.py
|
||||
├── Pydantic models (lines 56-284) — request/response schemas
|
||||
├── SSEManager (lines 349-381) — broadcast to clients
|
||||
├── _on_vault_change (lines 390-417) — watcher callback → index update + SSE broadcast
|
||||
├── API endpoints:
|
||||
│ ├── /api/health (1048) — public health
|
||||
│ ├── /api/events (2127) — SSE stream
|
||||
│ ├── /api/config GET/POST (2464/2470) — app config CRUD
|
||||
│ ├── /api/diagnostics (2500) — index stats (admin only)
|
||||
│ ├── /api/file/* — CRUD with SSE broadcasts on create/delete/rename
|
||||
│ └── /api/directory/* — CRUD with SSE broadcasts
|
||||
├── _CONFIG_PATH, _DEFAULT_CONFIG (2414-2458)
|
||||
|
||||
backend/vault_settings.py
|
||||
├── Per-vault settings (hideHiddenFiles)
|
||||
├── JSON persistence in /app/data/vault_settings.json
|
||||
|
||||
frontend/index.html
|
||||
├── #dashboard-home (341-392) — bookmarks + recent sections
|
||||
├── #config-modal (395-564) — full config UI
|
||||
├── #editor-modal — CodeMirror editor
|
||||
├── #graph-modal — D3 graph view
|
||||
└── #help-modal — user guide
|
||||
|
||||
frontend/app.js
|
||||
├── AuthManager (~1532+)
|
||||
├── DashboardRecentWidget (3344-3580)
|
||||
├── DashboardBookmarkWidget (3583-3660)
|
||||
├── initConfigModal (3906-3990)
|
||||
├── loadConfigFields (4043-4070)
|
||||
├── loadDiagnostics / renderDiagnostics (4157-4207)
|
||||
├── showWelcome (5417-5482) — dashboard rebuild + render
|
||||
├── IndexUpdateManager / SSE client (5773-6015)
|
||||
├── renderFile (3075-3328) — file view with action buttons
|
||||
└── TabManager (7307+) — multi-tab support
|
||||
### Data Flow for Search Performance
|
||||
```
|
||||
File change → watcher debounce (2s) → _on_vault_change()
|
||||
→ update_single_file() → _remove + _add → _index_generation += 2
|
||||
→ OR remove_single_file() → _index_generation += 1
|
||||
User types → suggest_titles() → get_inverted_index()
|
||||
→ is_stale() checks generation counter
|
||||
→ if counter changed: full rebuild of InvertedIndex (O(N) over all files)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Start Here
|
||||
Open `frontend/app.js` at line 5585 (`showWelcome()`) — this is the dashboard rebuild function and the primary cause of Issue 1. The fix needs to:
|
||||
1. Include all 4 dashboard sections in `showWelcome()`'s rebuild HTML
|
||||
2. OR: store the dashboard DOM before `renderFile()` wipes it, and restore it in `_showDashboard()`
|
||||
|
||||
1. **For Pydantic Field descriptions**: Open `backend/main.py` at **line 89** (`FileContentResponse`) and add `Field(description=...)` to each field for models without descriptions through line 311.
|
||||
For Issue 2, open `frontend/app.js` at line 7792 (`_renderTabs`) — add the missing `preview` class.
|
||||
|
||||
2. **For dashboard stats widget**: Open `frontend/index.html` at **line 341** (`#dashboard-home`) and `frontend/app.js` at **line 5417** (`showWelcome()`). The dashboard currently has two sections (Bookmarks + Recently Opened). A new stats section would be added between those divs.
|
||||
|
||||
3. **For webhooks on file events**: The SSE broadcasts happen in `backend/main.py` at lines 1252, 1532, 1607 (file events) and 1330, 1401, 1462 (directory events). The frontend SSE client in `app.js` at line 5773 doesn't listen for individual file events — it only handles `index_updated`. Webhook firing should be added alongside the `sse_manager.broadcast()` calls.
|
||||
|
||||
4. **For Configurations modal**: Open `frontend/index.html` at **line 395** (`#config-modal`) and `frontend/app.js` at **line 3906** (`initConfigModal()`).
|
||||
For Issue 3, open `backend/search.py` at line 418 (`get_inverted_index`) and `backend/indexer.py` lines 600–667 — consider whether single-file incremental updates to the inverted index are feasible, or whether the generation counter should be debounced/coalesced for file watcher batches.
|
||||
|
||||
Reference in New Issue
Block a user