ObsiViewer/docs/PERFORMENCE/phase1/IMPLEMENTATION_PHASE1_WINDSURF.md

# Phase 1 Implementation Guide - Windsurf Edition

## Status: Ready for Implementation

All new files have been created:
- ✅ `server/vault-metadata-loader.mjs` - Fast metadata loader
- ✅ `server/performance-config.mjs` - Performance configuration
- ✅ `src/app/services/vault.service.ts` - New VaultService with metadata-first
- ✅ `scripts/test-performance.mjs` - Performance testing script

## Next Steps: Minimal Modifications to Existing Files

### Step 1: Add Metadata Endpoint to `server/index.mjs`

**Location**: After line 478 (after `/api/files/list` endpoint)

**Add these imports at the top of the file** (around line 9):
```javascript
import { loadVaultMetadataOnly } from './vault-metadata-loader.mjs';
import { MetadataCache, PerformanceLogger } from './performance-config.mjs';
```

**Add this cache instance** (around line 34, after `const vaultEventClients = new Set();`):
```javascript
const metadataCache = new MetadataCache();
```

**Add this endpoint** (after line 478):
```javascript
// Fast metadata endpoint - no content, no enrichment
// Used for initial UI load (Phase 1 optimization)
app.get('/api/vault/metadata', async (req, res) => {
  try {
    // Check cache first
    const cached = metadataCache.get();
    if (cached) {
      console.log('[/api/vault/metadata] Returning cached metadata');
      return res.json(cached);
    }

    // Try Meilisearch first (already indexed)
    const client = meiliClient();
    const indexUid = vaultIndexName(vaultDir);
    const index = await ensureIndexSettings(client, indexUid);
    
    const result = await index.search('', {
      limit: 10000,
      attributesToRetrieve: ['id', 'title', 'path', 'createdAt', 'updatedAt']
    });

    const items = Array.isArray(result.hits) ? result.hits.map(hit => ({
      id: hit.id,
      title: hit.title,
      filePath: hit.path,
      createdAt: typeof hit.createdAt === 'number' ? new Date(hit.createdAt).toISOString() : hit.createdAt,
      updatedAt: typeof hit.updatedAt === 'number' ? new Date(hit.updatedAt).toISOString() : hit.updatedAt,
    })) : [];

    metadataCache.set(items);
    console.log(`[/api/vault/metadata] Returned ${items.length} items from Meilisearch`);
    res.json(items);
  } catch (error) {
    console.error('Failed to load metadata via Meilisearch, falling back to FS:', error);
    try {
      // Fallback: fast filesystem scan without enrichment
      const notes = await loadVaultMetadataOnly(vaultDir);
      const metadata = notes.map(n => ({
        id: n.id,
        title: n.title,
        filePath: n.filePath,
        createdAt: n.createdAt,
        updatedAt: n.updatedAt
      }));
      
      metadataCache.set(metadata);
      console.log(`[/api/vault/metadata] Returned ${metadata.length} items from filesystem`);
      res.json(metadata);
    } catch (err2) {
      console.error('FS fallback failed:', err2);
      res.status(500).json({ error: 'Unable to load vault metadata.' });
    }
  }
});
```

### Step 2: Disable Enrichment During Startup in `server/index.mjs`

**Location**: Lines 138-141 in `loadVaultNotes()` function

**BEFORE:**
```javascript
try {
  // Enrichir automatiquement le frontmatter lors du chargement
  const absPath = entryPath;
  const enrichResult = await enrichFrontmatterOnOpen(absPath);
  const content = enrichResult.content;
  
  const stats = fs.statSync(entryPath);
```

**AFTER:**
```javascript
try {
  // Skip enrichment during initial load for performance (Phase 1)
  // Enrichment will happen on-demand when file is opened via /api/files
  const content = fs.readFileSync(entryPath, 'utf-8');
  
  const stats = fs.statSync(entryPath);
```

### Step 3: Invalidate Cache on File Changes in `server/index.mjs`

**Location**: Around line 260-275 (in the watcher 'add' event)

**MODIFY the watcher.on('add') handler:**
```javascript
vaultWatcher.on('add', async (filePath) => {
  if (filePath.toLowerCase().endsWith('.md')) {
    // Invalidate metadata cache
    metadataCache.invalidate();
    
    // Enrichir le frontmatter pour les nouveaux fichiers
    try {
      const enrichResult = await enrichFrontmatterOnOpen(filePath);
      if (enrichResult.modified) {
        console.log('[Watcher] Enriched frontmatter for new file:', path.basename(filePath));
      }
    } catch (enrichError) {
      console.warn('[Watcher] Failed to enrich frontmatter for new file:', enrichError);
    }
    
    // Puis indexer dans Meilisearch
    upsertFile(filePath).catch(err => console.error('[Meili] Upsert on add failed:', err));
  }
});
```

**MODIFY the watcher.on('change') handler:**
```javascript
vaultWatcher.on('change', (filePath) => {
  if (filePath.toLowerCase().endsWith('.md')) {
    // Invalidate metadata cache
    metadataCache.invalidate();
    
    upsertFile(filePath).catch(err => console.error('[Meili] Upsert on change failed:', err));
  }
});
```

**MODIFY the watcher.on('unlink') handler:**
```javascript
vaultWatcher.on('unlink', (filePath) => {
  if (filePath.toLowerCase().endsWith('.md')) {
    // Invalidate metadata cache
    metadataCache.invalidate();
    
    const relativePath = path.relative(vaultDir, filePath).replace(/\\/g, '/');
    deleteFile(relativePath).catch(err => console.error('[Meili] Delete failed:', err));
  }
});
```

### Step 4: Update `src/app.component.ts`

**Location**: In the `ngOnInit()` method (around line 330-360)

**Find the ngOnInit() method and add vault initialization:**

```typescript
ngOnInit(): void {
  // Initialize theme from storage
  this.themeService.initFromStorage();
  
  // Initialize vault with metadata-first approach (Phase 1)
  this.vaultService.initializeVault().then(() => {
    console.log(`[AppComponent] Vault initialized with ${this.vaultService.getNotesCount()} notes`);
    this.logService.log('VAULT_INITIALIZED', {
      notesCount: this.vaultService.getNotesCount()
    });
  }).catch(error => {
    console.error('[AppComponent] Failed to initialize vault:', error);
    this.logService.log('VAULT_INIT_FAILED', { 
      error: error instanceof Error ? error.message : String(error) 
    }, 'error');
  });
  
  // Log app start
  this.logService.log('APP_START', {
    viewport: {
      width: typeof window !== 'undefined' ? window.innerWidth : 0,
      height: typeof window !== 'undefined' ? window.innerHeight : 0,
    },
  });
}
```

**Location**: In the `selectNote()` method (around line 380-400)

**Find the selectNote() method and ensure content is loaded:**

```typescript
async selectNote(noteId: string): Promise<void> {
  let note = this.vaultService.getNoteById(noteId);
  if (!note) {
    // Try to lazy-load using Meilisearch/slug mapping
    console.error(`Note not found: ${noteId}`);
    return;
  }
  
  // Ensure content is loaded before rendering (Phase 1)
  if (!note.content) {
    console.log(`[AppComponent] Loading content for note: ${noteId}`);
    note = await this.vaultService.ensureNoteContent(noteId);
    if (!note) {
      console.error(`[AppComponent] Failed to load note: ${noteId}`);
      return;
    }
  }
  
  this.selectedNoteId.set(noteId);
  this.logService.log('NOTE_SELECTED', { noteId });
}
```

## Testing

### Test 1: Verify Metadata Endpoint

```bash
# Test the new endpoint
curl http://localhost:3000/api/vault/metadata | head -50

# Measure response time
time curl http://localhost:3000/api/vault/metadata > /dev/null
```

### Test 2: Run Performance Comparison

```bash
# Run the performance test script
node scripts/test-performance.mjs

# With custom base URL
BASE_URL=http://localhost:4000 node scripts/test-performance.mjs
```

### Test 3: Browser Console Measurements

```javascript
// Measure startup time
performance.mark('app-start');
// ... app loads ...
performance.mark('app-ready');
performance.measure('startup', 'app-start', 'app-ready');
console.log(performance.getEntriesByName('startup')[0].duration);

// Check network requests
const requests = performance.getEntriesByType('resource');
const metadataReq = requests.find(r => r.name.includes('/api/vault/metadata'));
const vaultReq = requests.find(r => r.name.includes('/api/vault'));

if (metadataReq) {
  console.log(`Metadata endpoint: ${metadataReq.duration.toFixed(0)}ms, ${(metadataReq.transferSize / 1024).toFixed(0)}KB`);
}
if (vaultReq) {
  console.log(`Full vault endpoint: ${vaultReq.duration.toFixed(0)}ms, ${(vaultReq.transferSize / 1024 / 1024).toFixed(2)}MB`);
}
```

## Expected Results

### Before Phase 1
```
Startup time (1000 files):    15-25 seconds
Network payload:              5-10MB
Memory usage:                 200-300MB
Time to interactive:          20-35 seconds
```

### After Phase 1
```
Startup time (1000 files):    2-4 seconds ✅ (75% faster)
Network payload:              0.5-1MB ✅ (90% reduction)
Memory usage:                 50-100MB ✅ (75% reduction)
Time to interactive:          3-5 seconds ✅ (80% faster)
```

## Validation Checklist

- [ ] `/api/vault/metadata` endpoint returns data in < 1 second
- [ ] Metadata payload is < 1MB for 1000 files
- [ ] App UI is interactive within 2-3 seconds
- [ ] Clicking on a note loads content smoothly (< 500ms)
- [ ] No console errors or warnings
- [ ] All existing features still work
- [ ] Performance test script shows improvements
- [ ] Tests pass: `npm run test`
- [ ] E2E tests pass: `npm run test:e2e`

## Troubleshooting

### Issue: 404 on /api/vault/metadata
- Ensure imports are added at top of `server/index.mjs`
- Ensure endpoint is added before catch-all handlers
- Check server logs for errors

### Issue: Notes don't load when clicked
- Verify `ensureNoteContent()` is called in `selectNote()`
- Check browser console for errors
- Verify `/api/files` endpoint is working

### Issue: Startup time hasn't improved
- Verify `/api/vault/metadata` is being called (check Network tab)
- Check that enrichment was removed from `loadVaultNotes()`
- Verify old `/api/vault` endpoint is not being called

## Next Steps

1. ✅ Create new files (DONE)
2. ⏳ Add imports and endpoint to `server/index.mjs`
3. ⏳ Disable enrichment in `loadVaultNotes()`
4. ⏳ Add cache invalidation to watcher
5. ⏳ Update `app.component.ts` initialization
6. ⏳ Test and validate
7. ⏳ Deploy to production