539 lines
14 KiB
Markdown
539 lines
14 KiB
Markdown
# Phase 1 Implementation Guide: Metadata-First Loading
|
|
|
|
## Overview
|
|
|
|
This guide provides step-by-step instructions to implement Phase 1 of the performance optimization strategy. This phase is the **quick win** that reduces startup time from 10-30 seconds to 2-5 seconds.
|
|
|
|
**Estimated Time**: 4-6 hours
|
|
**Difficulty**: Medium
|
|
**Risk Level**: Low (backward compatible)
|
|
|
|
---
|
|
|
|
## Step 1: Create Fast Metadata-Only Loader
|
|
|
|
### File: `server/index.mjs`
|
|
|
|
Add a new function to load only metadata without enrichment:
|
|
|
|
```javascript
|
|
// Add after loadVaultNotes() function (around line 175)
|
|
|
|
/**
|
|
* Fast metadata loader - no enrichment, no content
|
|
* Returns only: id, title, path, createdAt, updatedAt
|
|
* Used for initial UI load to minimize startup time
|
|
*/
|
|
const loadVaultMetadataOnly = async (vaultPath) => {
|
|
const notes = [];
|
|
|
|
const walk = async (currentDir) => {
|
|
if (!fs.existsSync(currentDir)) {
|
|
return;
|
|
}
|
|
|
|
const entries = fs.readdirSync(currentDir, { withFileTypes: true });
|
|
for (const entry of entries) {
|
|
const entryPath = path.join(currentDir, entry.name);
|
|
|
|
if (entry.isDirectory()) {
|
|
await walk(entryPath);
|
|
continue;
|
|
}
|
|
|
|
if (!isMarkdownFile(entry)) {
|
|
continue;
|
|
}
|
|
|
|
try {
|
|
// Read file WITHOUT enrichment (fast)
|
|
const content = fs.readFileSync(entryPath, 'utf-8');
|
|
const stats = fs.statSync(entryPath);
|
|
|
|
const relativePathWithExt = path.relative(vaultPath, entryPath).replace(/\\/g, '/');
|
|
const relativePath = relativePathWithExt.replace(/\.md$/i, '');
|
|
const id = slugifyPath(relativePath);
|
|
const fileNameWithExt = entry.name;
|
|
|
|
const fallbackTitle = path.basename(relativePath);
|
|
const title = extractTitle(content, fallbackTitle);
|
|
const finalId = id || slugifySegment(fallbackTitle) || fallbackTitle;
|
|
const createdDate = stats.birthtimeMs ? new Date(stats.birthtimeMs) : new Date(stats.ctimeMs);
|
|
const updatedDate = new Date(stats.mtimeMs);
|
|
|
|
notes.push({
|
|
id: finalId,
|
|
title,
|
|
// NO content field - saves memory
|
|
// NO tags - can be extracted on-demand
|
|
mtime: stats.mtimeMs,
|
|
fileName: fileNameWithExt,
|
|
filePath: relativePathWithExt,
|
|
originalPath: relativePath,
|
|
createdAt: createdDate.toISOString(),
|
|
updatedAt: updatedDate.toISOString()
|
|
// NO frontmatter - no enrichment
|
|
});
|
|
} catch (err) {
|
|
console.error(`Failed to read metadata for ${entryPath}:`, err);
|
|
}
|
|
}
|
|
};
|
|
|
|
await walk(vaultPath);
|
|
return notes;
|
|
};
|
|
```
|
|
|
|
---
|
|
|
|
## Step 2: Create Metadata Endpoint
|
|
|
|
### File: `server/index.mjs`
|
|
|
|
Add a new endpoint for fast metadata loading (around line 450, after `/api/files/list`):
|
|
|
|
```javascript
|
|
// NEW: Fast metadata endpoint (no content, no enrichment)
|
|
app.get('/api/vault/metadata', async (req, res) => {
|
|
try {
|
|
// Try Meilisearch first (already indexed)
|
|
const client = meiliClient();
|
|
const indexUid = vaultIndexName(vaultDir);
|
|
const index = await ensureIndexSettings(client, indexUid);
|
|
|
|
const result = await index.search('', {
|
|
limit: 10000,
|
|
attributesToRetrieve: ['id', 'title', 'path', 'createdAt', 'updatedAt']
|
|
});
|
|
|
|
const items = Array.isArray(result.hits) ? result.hits.map(hit => ({
|
|
id: hit.id,
|
|
title: hit.title,
|
|
filePath: hit.path,
|
|
createdAt: typeof hit.createdAt === 'number' ? new Date(hit.createdAt).toISOString() : hit.createdAt,
|
|
updatedAt: typeof hit.updatedAt === 'number' ? new Date(hit.updatedAt).toISOString() : hit.updatedAt,
|
|
})) : [];
|
|
|
|
res.json(items);
|
|
} catch (error) {
|
|
console.error('Failed to load metadata via Meilisearch, falling back to FS:', error);
|
|
try {
|
|
// Fallback: fast filesystem scan without enrichment
|
|
const notes = await loadVaultMetadataOnly(vaultDir);
|
|
res.json(notes.map(n => ({
|
|
id: n.id,
|
|
title: n.title,
|
|
filePath: n.filePath,
|
|
createdAt: n.createdAt,
|
|
updatedAt: n.updatedAt
|
|
})));
|
|
} catch (err2) {
|
|
console.error('FS fallback failed:', err2);
|
|
res.status(500).json({ error: 'Unable to load vault metadata.' });
|
|
}
|
|
}
|
|
});
|
|
```
|
|
|
|
---
|
|
|
|
## Step 3: Disable Enrichment During Startup
|
|
|
|
### File: `server/index.mjs`
|
|
|
|
Modify `loadVaultNotes()` to skip enrichment (around line 138-141):
|
|
|
|
**BEFORE:**
|
|
```javascript
|
|
try {
|
|
// Enrichir automatiquement le frontmatter lors du chargement
|
|
const absPath = entryPath;
|
|
const enrichResult = await enrichFrontmatterOnOpen(absPath);
|
|
const content = enrichResult.content;
|
|
|
|
const stats = fs.statSync(entryPath);
|
|
// ...
|
|
```
|
|
|
|
**AFTER:**
|
|
```javascript
|
|
try {
|
|
// Skip enrichment during initial load for performance
|
|
// Enrichment will happen on-demand when file is opened
|
|
const content = fs.readFileSync(entryPath, 'utf-8');
|
|
|
|
const stats = fs.statSync(entryPath);
|
|
// ...
|
|
```
|
|
|
|
**Why**: Enrichment is expensive and not needed for initial metadata load. It will still happen when the user opens a note via `/api/files` endpoint.
|
|
|
|
---
|
|
|
|
## Step 4: Update VaultService
|
|
|
|
### File: `src/app/services/vault.service.ts`
|
|
|
|
First, let's check the current implementation:
|
|
|
|
```bash
|
|
# Find VaultService
|
|
find src -name "*vault*service*" -type f
|
|
```
|
|
|
|
If it doesn't exist, we need to create it or update the existing service. Let's assume it exists and update it:
|
|
|
|
**Add metadata-first loading:**
|
|
|
|
```typescript
|
|
import { Injectable, signal, computed, inject } from '@angular/core';
|
|
import { HttpClient } from '@angular/common/http';
|
|
import { firstValueFrom } from 'rxjs';
|
|
|
|
@Injectable({ providedIn: 'root' })
|
|
export class VaultService {
|
|
private http = inject(HttpClient);
|
|
|
|
// Signals for vault state
|
|
private allNotesMetadata = signal<Note[]>([]);
|
|
private contentCache = new Map<string, string>();
|
|
private enrichmentCache = new Map<string, any>();
|
|
|
|
// Public computed signals
|
|
allNotes = computed(() => this.allNotesMetadata());
|
|
|
|
/**
|
|
* Initialize vault with metadata-first approach
|
|
* Step 1: Load metadata immediately (fast)
|
|
* Step 2: Load full content on-demand when note is selected
|
|
*/
|
|
async initializeVault(): Promise<void> {
|
|
try {
|
|
console.time('loadVaultMetadata');
|
|
|
|
// Load metadata only (fast)
|
|
const metadata = await firstValueFrom(
|
|
this.http.get<any[]>('/api/vault/metadata')
|
|
);
|
|
|
|
console.timeEnd('loadVaultMetadata');
|
|
console.log(`Loaded metadata for ${metadata.length} notes`);
|
|
|
|
// Convert metadata to Note objects with empty content
|
|
const notes = metadata.map(m => ({
|
|
id: m.id,
|
|
title: m.title,
|
|
filePath: m.filePath,
|
|
originalPath: m.filePath.replace(/\.md$/i, ''),
|
|
fileName: m.filePath.split('/').pop() || '',
|
|
content: '', // Empty initially
|
|
rawContent: '',
|
|
tags: [], // Will be extracted on-demand
|
|
frontmatter: {},
|
|
createdAt: m.createdAt,
|
|
updatedAt: m.updatedAt,
|
|
mtime: new Date(m.updatedAt).getTime()
|
|
}));
|
|
|
|
this.allNotesMetadata.set(notes);
|
|
|
|
} catch (error) {
|
|
console.error('Failed to initialize vault:', error);
|
|
throw error;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Ensure note content is loaded (lazy loading)
|
|
* Called when user selects a note
|
|
*/
|
|
async ensureNoteContent(noteId: string): Promise<Note | null> {
|
|
const notes = this.allNotesMetadata();
|
|
const note = notes.find(n => n.id === noteId);
|
|
|
|
if (!note) {
|
|
console.warn(`Note not found: ${noteId}`);
|
|
return null;
|
|
}
|
|
|
|
// If content already loaded, return
|
|
if (note.content) {
|
|
return note;
|
|
}
|
|
|
|
// Load content from server
|
|
try {
|
|
console.time(`loadNoteContent:${noteId}`);
|
|
|
|
const response = await firstValueFrom(
|
|
this.http.get<any>('/api/files', {
|
|
params: { path: note.filePath }
|
|
})
|
|
);
|
|
|
|
console.timeEnd(`loadNoteContent:${noteId}`);
|
|
|
|
// Update note with full content
|
|
note.content = response.content || response;
|
|
note.rawContent = response.rawContent || response;
|
|
|
|
// Extract tags if not already present
|
|
if (!note.tags || note.tags.length === 0) {
|
|
note.tags = this.extractTags(note.content);
|
|
}
|
|
|
|
// Extract frontmatter if available
|
|
if (response.frontmatter) {
|
|
note.frontmatter = response.frontmatter;
|
|
}
|
|
|
|
return note;
|
|
} catch (error) {
|
|
console.error(`Failed to load note content for ${noteId}:`, error);
|
|
return note;
|
|
}
|
|
}
|
|
|
|
/**
|
|
* Get note by ID (may have empty content if not yet loaded)
|
|
*/
|
|
getNoteById(noteId: string): Note | undefined {
|
|
return this.allNotesMetadata().find(n => n.id === noteId);
|
|
}
|
|
|
|
/**
|
|
* Extract tags from markdown content
|
|
*/
|
|
private extractTags(content: string): string[] {
|
|
const tagRegex = /(^|\s)#([A-Za-z0-9_\/-]+)/g;
|
|
const tags = new Set<string>();
|
|
let match;
|
|
while ((match = tagRegex.exec(content)) !== null) {
|
|
tags.add(match[2]);
|
|
}
|
|
return Array.from(tags);
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Step 5: Update AppComponent Initialization
|
|
|
|
### File: `src/app.component.ts`
|
|
|
|
Modify the `ngOnInit()` method to use the new metadata-first approach:
|
|
|
|
**BEFORE:**
|
|
```typescript
|
|
ngOnInit(): void {
|
|
this.themeService.initFromStorage();
|
|
this.logService.log('APP_START', { /* ... */ });
|
|
}
|
|
```
|
|
|
|
**AFTER:**
|
|
```typescript
|
|
ngOnInit(): void {
|
|
this.themeService.initFromStorage();
|
|
|
|
// Initialize vault with metadata-first approach
|
|
this.vaultService.initializeVault().then(() => {
|
|
console.log('Vault initialized');
|
|
this.logService.log('VAULT_INITIALIZED', {
|
|
notesCount: this.vaultService.allNotes().length
|
|
});
|
|
}).catch(error => {
|
|
console.error('Failed to initialize vault:', error);
|
|
this.logService.log('VAULT_INIT_FAILED', { error: String(error) }, 'error');
|
|
});
|
|
|
|
this.logService.log('APP_START', {
|
|
viewport: {
|
|
width: typeof window !== 'undefined' ? window.innerWidth : 0,
|
|
height: typeof window !== 'undefined' ? window.innerHeight : 0,
|
|
},
|
|
});
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Step 6: Update Note Selection to Load Content
|
|
|
|
### File: `src/app.component.ts`
|
|
|
|
Modify the `selectNote()` method to ensure content is loaded:
|
|
|
|
**BEFORE:**
|
|
```typescript
|
|
async selectNote(noteId: string): Promise<void> {
|
|
let note = this.vaultService.getNoteById(noteId);
|
|
if (!note) {
|
|
// ... fallback logic ...
|
|
}
|
|
this.selectedNoteId.set(noteId);
|
|
}
|
|
```
|
|
|
|
**AFTER:**
|
|
```typescript
|
|
async selectNote(noteId: string): Promise<void> {
|
|
let note = this.vaultService.getNoteById(noteId);
|
|
if (!note) {
|
|
// ... fallback logic ...
|
|
}
|
|
|
|
// Ensure content is loaded before rendering
|
|
if (!note.content) {
|
|
console.log(`Loading content for note: ${noteId}`);
|
|
note = await this.vaultService.ensureNoteContent(noteId);
|
|
if (!note) {
|
|
console.error(`Failed to load note: ${noteId}`);
|
|
return;
|
|
}
|
|
}
|
|
|
|
this.selectedNoteId.set(noteId);
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Step 7: Testing
|
|
|
|
### Test 1: Verify Metadata Endpoint
|
|
|
|
```bash
|
|
# Test the new endpoint
|
|
curl http://localhost:3000/api/vault/metadata | head -50
|
|
|
|
# Should return fast (< 1 second for 1000 files)
|
|
time curl http://localhost:3000/api/vault/metadata > /dev/null
|
|
```
|
|
|
|
### Test 2: Verify Startup Time
|
|
|
|
```bash
|
|
# Measure startup time in browser console
|
|
performance.mark('app-start');
|
|
// ... app loads ...
|
|
performance.mark('app-ready');
|
|
performance.measure('startup', 'app-start', 'app-ready');
|
|
console.log(performance.getEntriesByName('startup')[0].duration);
|
|
```
|
|
|
|
### Test 3: Verify Content Loading
|
|
|
|
1. Open browser DevTools → Network tab
|
|
2. Start the application
|
|
3. Verify that `/api/vault/metadata` is called first (small payload)
|
|
4. Click on a note
|
|
5. Verify that `/api/files?path=...` is called to load content
|
|
|
|
### Test 4: Performance Comparison
|
|
|
|
Create a test script to measure improvements:
|
|
|
|
```javascript
|
|
// test-performance.mjs
|
|
import fetch from 'node-fetch';
|
|
|
|
async function measureStartup() {
|
|
console.time('Metadata Load');
|
|
const response = await fetch('http://localhost:3000/api/vault/metadata');
|
|
const data = await response.json();
|
|
console.timeEnd('Metadata Load');
|
|
console.log(`Loaded ${data.length} notes`);
|
|
console.log(`Payload size: ${JSON.stringify(data).length} bytes`);
|
|
}
|
|
|
|
measureStartup();
|
|
```
|
|
|
|
---
|
|
|
|
## Step 8: Rollback Plan
|
|
|
|
If issues occur, you can quickly rollback:
|
|
|
|
1. **Revert server changes**: Comment out the new endpoint, restore enrichment in `loadVaultNotes()`
|
|
2. **Revert client changes**: Remove the `initializeVault()` call, use old approach
|
|
3. **Keep `/api/vault` endpoint**: It still works as before for backward compatibility
|
|
|
|
---
|
|
|
|
## Verification Checklist
|
|
|
|
- [ ] New endpoint `/api/vault/metadata` returns data in < 1 second
|
|
- [ ] Metadata payload is < 1MB for 1000 files
|
|
- [ ] App UI is interactive within 2-3 seconds
|
|
- [ ] Clicking on a note loads content smoothly
|
|
- [ ] No console errors or warnings
|
|
- [ ] All existing features still work
|
|
- [ ] Performance is improved by 50%+ compared to before
|
|
|
|
---
|
|
|
|
## Performance Metrics
|
|
|
|
### Before Phase 1
|
|
```
|
|
Startup time (1000 files): 15-25 seconds
|
|
Metadata endpoint: N/A
|
|
Network payload: 5-10MB
|
|
Memory usage: 200-300MB
|
|
```
|
|
|
|
### After Phase 1
|
|
```
|
|
Startup time (1000 files): 2-4 seconds ✓
|
|
Metadata endpoint: 0.5-1 second
|
|
Network payload: 0.5-1MB
|
|
Memory usage: 50-100MB ✓
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Issue: Metadata endpoint returns 500 error
|
|
|
|
**Solution**: Check server logs for Meilisearch errors. The fallback to `loadVaultMetadataOnly()` should work.
|
|
|
|
```bash
|
|
# Verify Meilisearch is running
|
|
curl http://localhost:7700/health
|
|
```
|
|
|
|
### Issue: Notes don't load when clicked
|
|
|
|
**Solution**: Ensure `ensureNoteContent()` is called in `selectNote()`. Check browser console for errors.
|
|
|
|
### Issue: Startup time hasn't improved
|
|
|
|
**Solution**:
|
|
1. Verify `/api/vault/metadata` is being called (check Network tab)
|
|
2. Check that enrichment was removed from `loadVaultNotes()`
|
|
3. Verify old `/api/vault` endpoint is not being called
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
After Phase 1 is complete and tested:
|
|
|
|
1. Monitor performance in production
|
|
2. Collect user feedback
|
|
3. Proceed with Phase 2 (Pagination) if needed
|
|
4. Consider Phase 3 (Server Caching) for high-traffic deployments
|
|
|
|
---
|
|
|
|
## References
|
|
|
|
- [Performance Optimization Strategy](./PERFORMANCE_OPTIMIZATION_STRATEGY.md)
|
|
- [Angular Change Detection](https://angular.io/guide/change-detection)
|
|
- [RxJS firstValueFrom](https://rxjs.dev/api/index/function/firstValueFrom)
|