ObsiViewer/docs/PERFORMENCE/phase1/IMPLEMENTATION_PHASE1_WINDSURF.md

Phase 1 Implementation Guide - Windsurf Edition

Status: Ready for Implementation

All new files have been created:

• ✅ server/vault-metadata-loader.mjs - Fast metadata loader
• ✅ server/performance-config.mjs - Performance configuration
• ✅ src/app/services/vault.service.ts - New VaultService with metadata-first
• ✅ scripts/test-performance.mjs - Performance testing script

Next Steps: Minimal Modifications to Existing Files

Step 1: Add Metadata Endpoint to server/index.mjs

Location: After line 478 (after /api/files/list endpoint)

Add these imports at the top of the file (around line 9):

import { loadVaultMetadataOnly } from './vault-metadata-loader.mjs';
import { MetadataCache, PerformanceLogger } from './performance-config.mjs';

Add this cache instance (around line 34, after const vaultEventClients = new Set();):

const metadataCache = new MetadataCache();

Add this endpoint (after line 478):

// Fast metadata endpoint - no content, no enrichment
// Used for initial UI load (Phase 1 optimization)
app.get('/api/vault/metadata', async (req, res) => {
  try {
    // Check cache first
    const cached = metadataCache.get();
    if (cached) {
      console.log('[/api/vault/metadata] Returning cached metadata');
      return res.json(cached);
    }

    // Try Meilisearch first (already indexed)
    const client = meiliClient();
    const indexUid = vaultIndexName(vaultDir);
    const index = await ensureIndexSettings(client, indexUid);
    
    const result = await index.search('', {
      limit: 10000,
      attributesToRetrieve: ['id', 'title', 'path', 'createdAt', 'updatedAt']
    });

    const items = Array.isArray(result.hits) ? result.hits.map(hit => ({
      id: hit.id,
      title: hit.title,
      filePath: hit.path,
      createdAt: typeof hit.createdAt === 'number' ? new Date(hit.createdAt).toISOString() : hit.createdAt,
      updatedAt: typeof hit.updatedAt === 'number' ? new Date(hit.updatedAt).toISOString() : hit.updatedAt,
    })) : [];

    metadataCache.set(items);
    console.log(`[/api/vault/metadata] Returned ${items.length} items from Meilisearch`);
    res.json(items);
  } catch (error) {
    console.error('Failed to load metadata via Meilisearch, falling back to FS:', error);
    try {
      // Fallback: fast filesystem scan without enrichment
      const notes = await loadVaultMetadataOnly(vaultDir);
      const metadata = notes.map(n => ({
        id: n.id,
        title: n.title,
        filePath: n.filePath,
        createdAt: n.createdAt,
        updatedAt: n.updatedAt
      }));
      
      metadataCache.set(metadata);
      console.log(`[/api/vault/metadata] Returned ${metadata.length} items from filesystem`);
      res.json(metadata);
    } catch (err2) {
      console.error('FS fallback failed:', err2);
      res.status(500).json({ error: 'Unable to load vault metadata.' });
    }
  }
});

Step 2: Disable Enrichment During Startup in server/index.mjs

Location: Lines 138-141 in loadVaultNotes() function

BEFORE:

try {
  // Enrichir automatiquement le frontmatter lors du chargement
  const absPath = entryPath;
  const enrichResult = await enrichFrontmatterOnOpen(absPath);
  const content = enrichResult.content;
  
  const stats = fs.statSync(entryPath);

AFTER:

try {
  // Skip enrichment during initial load for performance (Phase 1)
  // Enrichment will happen on-demand when file is opened via /api/files
  const content = fs.readFileSync(entryPath, 'utf-8');
  
  const stats = fs.statSync(entryPath);

Step 3: Invalidate Cache on File Changes in server/index.mjs

Location: Around line 260-275 (in the watcher 'add' event)

MODIFY the watcher.on('add') handler:

vaultWatcher.on('add', async (filePath) => {
  if (filePath.toLowerCase().endsWith('.md')) {
    // Invalidate metadata cache
    metadataCache.invalidate();
    
    // Enrichir le frontmatter pour les nouveaux fichiers
    try {
      const enrichResult = await enrichFrontmatterOnOpen(filePath);
      if (enrichResult.modified) {
        console.log('[Watcher] Enriched frontmatter for new file:', path.basename(filePath));
      }
    } catch (enrichError) {
      console.warn('[Watcher] Failed to enrich frontmatter for new file:', enrichError);
    }
    
    // Puis indexer dans Meilisearch
    upsertFile(filePath).catch(err => console.error('[Meili] Upsert on add failed:', err));
  }
});

MODIFY the watcher.on('change') handler:

vaultWatcher.on('change', (filePath) => {
  if (filePath.toLowerCase().endsWith('.md')) {
    // Invalidate metadata cache
    metadataCache.invalidate();
    
    upsertFile(filePath).catch(err => console.error('[Meili] Upsert on change failed:', err));
  }
});

MODIFY the watcher.on('unlink') handler:

vaultWatcher.on('unlink', (filePath) => {
  if (filePath.toLowerCase().endsWith('.md')) {
    // Invalidate metadata cache
    metadataCache.invalidate();
    
    const relativePath = path.relative(vaultDir, filePath).replace(/\\/g, '/');
    deleteFile(relativePath).catch(err => console.error('[Meili] Delete failed:', err));
  }
});

Step 4: Update src/app.component.ts

Location: In the ngOnInit() method (around line 330-360)

Find the ngOnInit() method and add vault initialization:

ngOnInit(): void {
  // Initialize theme from storage
  this.themeService.initFromStorage();
  
  // Initialize vault with metadata-first approach (Phase 1)
  this.vaultService.initializeVault().then(() => {
    console.log(`[AppComponent] Vault initialized with ${this.vaultService.getNotesCount()} notes`);
    this.logService.log('VAULT_INITIALIZED', {
      notesCount: this.vaultService.getNotesCount()
    });
  }).catch(error => {
    console.error('[AppComponent] Failed to initialize vault:', error);
    this.logService.log('VAULT_INIT_FAILED', { 
      error: error instanceof Error ? error.message : String(error) 
    }, 'error');
  });
  
  // Log app start
  this.logService.log('APP_START', {
    viewport: {
      width: typeof window !== 'undefined' ? window.innerWidth : 0,
      height: typeof window !== 'undefined' ? window.innerHeight : 0,
    },
  });
}

Location: In the selectNote() method (around line 380-400)

Find the selectNote() method and ensure content is loaded:

async selectNote(noteId: string): Promise<void> {
  let note = this.vaultService.getNoteById(noteId);
  if (!note) {
    // Try to lazy-load using Meilisearch/slug mapping
    console.error(`Note not found: ${noteId}`);
    return;
  }
  
  // Ensure content is loaded before rendering (Phase 1)
  if (!note.content) {
    console.log(`[AppComponent] Loading content for note: ${noteId}`);
    note = await this.vaultService.ensureNoteContent(noteId);
    if (!note) {
      console.error(`[AppComponent] Failed to load note: ${noteId}`);
      return;
    }
  }
  
  this.selectedNoteId.set(noteId);
  this.logService.log('NOTE_SELECTED', { noteId });
}

Testing

Test 1: Verify Metadata Endpoint

# Test the new endpoint
curl http://localhost:3000/api/vault/metadata | head -50

# Measure response time
time curl http://localhost:3000/api/vault/metadata > /dev/null

Test 2: Run Performance Comparison

# Run the performance test script
node scripts/test-performance.mjs

# With custom base URL
BASE_URL=http://localhost:4000 node scripts/test-performance.mjs

Test 3: Browser Console Measurements

// Measure startup time
performance.mark('app-start');
// ... app loads ...
performance.mark('app-ready');
performance.measure('startup', 'app-start', 'app-ready');
console.log(performance.getEntriesByName('startup')[0].duration);

// Check network requests
const requests = performance.getEntriesByType('resource');
const metadataReq = requests.find(r => r.name.includes('/api/vault/metadata'));
const vaultReq = requests.find(r => r.name.includes('/api/vault'));

if (metadataReq) {
  console.log(`Metadata endpoint: ${metadataReq.duration.toFixed(0)}ms, ${(metadataReq.transferSize / 1024).toFixed(0)}KB`);
}
if (vaultReq) {
  console.log(`Full vault endpoint: ${vaultReq.duration.toFixed(0)}ms, ${(vaultReq.transferSize / 1024 / 1024).toFixed(2)}MB`);
}

Expected Results

Before Phase 1

Startup time (1000 files):    15-25 seconds
Network payload:              5-10MB
Memory usage:                 200-300MB
Time to interactive:          20-35 seconds

After Phase 1

Startup time (1000 files):    2-4 seconds ✅ (75% faster)
Network payload:              0.5-1MB ✅ (90% reduction)
Memory usage:                 50-100MB ✅ (75% reduction)
Time to interactive:          3-5 seconds ✅ (80% faster)

Validation Checklist

• /api/vault/metadata endpoint returns data in < 1 second
• Metadata payload is < 1MB for 1000 files
• App UI is interactive within 2-3 seconds
• Clicking on a note loads content smoothly (< 500ms)
• No console errors or warnings
• All existing features still work
• Performance test script shows improvements
• Tests pass: npm run test
• E2E tests pass: npm run test:e2e

Troubleshooting

Issue: 404 on /api/vault/metadata

• Ensure imports are added at top of server/index.mjs
• Ensure endpoint is added before catch-all handlers
• Check server logs for errors

Issue: Notes don't load when clicked

• Verify ensureNoteContent() is called in selectNote()
• Check browser console for errors
• Verify /api/files endpoint is working

Issue: Startup time hasn't improved

• Verify /api/vault/metadata is being called (check Network tab)
• Check that enrichment was removed from loadVaultNotes()
• Verify old /api/vault endpoint is not being called

Next Steps

1. ✅ Create new files (DONE)
2. ⏳ Add imports and endpoint to server/index.mjs
3. ⏳ Disable enrichment in loadVaultNotes()
4. ⏳ Add cache invalidation to watcher
5. ⏳ Update app.component.ts initialization
6. ⏳ Test and validate
7. ⏳ Deploy to production