ObsiViewer/docs/PERFORMENCE/phase1/IMPLEMENTATION_PHASE1.md

Phase 1 Implementation Guide: Metadata-First Loading

Overview

This guide provides step-by-step instructions to implement Phase 1 of the performance optimization strategy. This phase is the quick win that reduces startup time from 10-30 seconds to 2-5 seconds.

Estimated Time: 4-6 hours Difficulty: Medium Risk Level: Low (backward compatible)

---

Step 1: Create Fast Metadata-Only Loader

File: server/index.mjs

Add a new function to load only metadata without enrichment:

// Add after loadVaultNotes() function (around line 175)

/**
 * Fast metadata loader - no enrichment, no content
 * Returns only: id, title, path, createdAt, updatedAt
 * Used for initial UI load to minimize startup time
 */
const loadVaultMetadataOnly = async (vaultPath) => {
  const notes = [];

  const walk = async (currentDir) => {
    if (!fs.existsSync(currentDir)) {
      return;
    }

    const entries = fs.readdirSync(currentDir, { withFileTypes: true });
    for (const entry of entries) {
      const entryPath = path.join(currentDir, entry.name);

      if (entry.isDirectory()) {
        await walk(entryPath);
        continue;
      }

      if (!isMarkdownFile(entry)) {
        continue;
      }

      try {
        // Read file WITHOUT enrichment (fast)
        const content = fs.readFileSync(entryPath, 'utf-8');
        const stats = fs.statSync(entryPath);
        
        const relativePathWithExt = path.relative(vaultPath, entryPath).replace(/\\/g, '/');
        const relativePath = relativePathWithExt.replace(/\.md$/i, '');
        const id = slugifyPath(relativePath);
        const fileNameWithExt = entry.name;

        const fallbackTitle = path.basename(relativePath);
        const title = extractTitle(content, fallbackTitle);
        const finalId = id || slugifySegment(fallbackTitle) || fallbackTitle;
        const createdDate = stats.birthtimeMs ? new Date(stats.birthtimeMs) : new Date(stats.ctimeMs);
        const updatedDate = new Date(stats.mtimeMs);

        notes.push({
          id: finalId,
          title,
          // NO content field - saves memory
          // NO tags - can be extracted on-demand
          mtime: stats.mtimeMs,
          fileName: fileNameWithExt,
          filePath: relativePathWithExt,
          originalPath: relativePath,
          createdAt: createdDate.toISOString(),
          updatedAt: updatedDate.toISOString()
          // NO frontmatter - no enrichment
        });
      } catch (err) {
        console.error(`Failed to read metadata for ${entryPath}:`, err);
      }
    }
  };

  await walk(vaultPath);
  return notes;
};

---

Step 2: Create Metadata Endpoint

File: server/index.mjs

Add a new endpoint for fast metadata loading (around line 450, after /api/files/list):

// NEW: Fast metadata endpoint (no content, no enrichment)
app.get('/api/vault/metadata', async (req, res) => {
  try {
    // Try Meilisearch first (already indexed)
    const client = meiliClient();
    const indexUid = vaultIndexName(vaultDir);
    const index = await ensureIndexSettings(client, indexUid);
    
    const result = await index.search('', {
      limit: 10000,
      attributesToRetrieve: ['id', 'title', 'path', 'createdAt', 'updatedAt']
    });

    const items = Array.isArray(result.hits) ? result.hits.map(hit => ({
      id: hit.id,
      title: hit.title,
      filePath: hit.path,
      createdAt: typeof hit.createdAt === 'number' ? new Date(hit.createdAt).toISOString() : hit.createdAt,
      updatedAt: typeof hit.updatedAt === 'number' ? new Date(hit.updatedAt).toISOString() : hit.updatedAt,
    })) : [];

    res.json(items);
  } catch (error) {
    console.error('Failed to load metadata via Meilisearch, falling back to FS:', error);
    try {
      // Fallback: fast filesystem scan without enrichment
      const notes = await loadVaultMetadataOnly(vaultDir);
      res.json(notes.map(n => ({
        id: n.id,
        title: n.title,
        filePath: n.filePath,
        createdAt: n.createdAt,
        updatedAt: n.updatedAt
      })));
    } catch (err2) {
      console.error('FS fallback failed:', err2);
      res.status(500).json({ error: 'Unable to load vault metadata.' });
    }
  }
});

---

Step 3: Disable Enrichment During Startup

File: server/index.mjs

Modify loadVaultNotes() to skip enrichment (around line 138-141):

BEFORE:

try {
  // Enrichir automatiquement le frontmatter lors du chargement
  const absPath = entryPath;
  const enrichResult = await enrichFrontmatterOnOpen(absPath);
  const content = enrichResult.content;
  
  const stats = fs.statSync(entryPath);
  // ...

AFTER:

try {
  // Skip enrichment during initial load for performance
  // Enrichment will happen on-demand when file is opened
  const content = fs.readFileSync(entryPath, 'utf-8');
  
  const stats = fs.statSync(entryPath);
  // ...

Why: Enrichment is expensive and not needed for initial metadata load. It will still happen when the user opens a note via /api/files endpoint.

---

Step 4: Update VaultService

File: src/app/services/vault.service.ts

First, let's check the current implementation:

# Find VaultService
find src -name "*vault*service*" -type f

If it doesn't exist, we need to create it or update the existing service. Let's assume it exists and update it:

Add metadata-first loading:

import { Injectable, signal, computed, inject } from '@angular/core';
import { HttpClient } from '@angular/common/http';
import { firstValueFrom } from 'rxjs';

@Injectable({ providedIn: 'root' })
export class VaultService {
  private http = inject(HttpClient);
  
  // Signals for vault state
  private allNotesMetadata = signal<Note[]>([]);
  private contentCache = new Map<string, string>();
  private enrichmentCache = new Map<string, any>();
  
  // Public computed signals
  allNotes = computed(() => this.allNotesMetadata());
  
  /**
   * Initialize vault with metadata-first approach
   * Step 1: Load metadata immediately (fast)
   * Step 2: Load full content on-demand when note is selected
   */
  async initializeVault(): Promise<void> {
    try {
      console.time('loadVaultMetadata');
      
      // Load metadata only (fast)
      const metadata = await firstValueFrom(
        this.http.get<any[]>('/api/vault/metadata')
      );
      
      console.timeEnd('loadVaultMetadata');
      console.log(`Loaded metadata for ${metadata.length} notes`);
      
      // Convert metadata to Note objects with empty content
      const notes = metadata.map(m => ({
        id: m.id,
        title: m.title,
        filePath: m.filePath,
        originalPath: m.filePath.replace(/\.md$/i, ''),
        fileName: m.filePath.split('/').pop() || '',
        content: '', // Empty initially
        rawContent: '',
        tags: [], // Will be extracted on-demand
        frontmatter: {},
        createdAt: m.createdAt,
        updatedAt: m.updatedAt,
        mtime: new Date(m.updatedAt).getTime()
      }));
      
      this.allNotesMetadata.set(notes);
      
    } catch (error) {
      console.error('Failed to initialize vault:', error);
      throw error;
    }
  }
  
  /**
   * Ensure note content is loaded (lazy loading)
   * Called when user selects a note
   */
  async ensureNoteContent(noteId: string): Promise<Note | null> {
    const notes = this.allNotesMetadata();
    const note = notes.find(n => n.id === noteId);
    
    if (!note) {
      console.warn(`Note not found: ${noteId}`);
      return null;
    }
    
    // If content already loaded, return
    if (note.content) {
      return note;
    }
    
    // Load content from server
    try {
      console.time(`loadNoteContent:${noteId}`);
      
      const response = await firstValueFrom(
        this.http.get<any>('/api/files', {
          params: { path: note.filePath }
        })
      );
      
      console.timeEnd(`loadNoteContent:${noteId}`);
      
      // Update note with full content
      note.content = response.content || response;
      note.rawContent = response.rawContent || response;
      
      // Extract tags if not already present
      if (!note.tags || note.tags.length === 0) {
        note.tags = this.extractTags(note.content);
      }
      
      // Extract frontmatter if available
      if (response.frontmatter) {
        note.frontmatter = response.frontmatter;
      }
      
      return note;
    } catch (error) {
      console.error(`Failed to load note content for ${noteId}:`, error);
      return note;
    }
  }
  
  /**
   * Get note by ID (may have empty content if not yet loaded)
   */
  getNoteById(noteId: string): Note | undefined {
    return this.allNotesMetadata().find(n => n.id === noteId);
  }
  
  /**
   * Extract tags from markdown content
   */
  private extractTags(content: string): string[] {
    const tagRegex = /(^|\s)#([A-Za-z0-9_\/-]+)/g;
    const tags = new Set<string>();
    let match;
    while ((match = tagRegex.exec(content)) !== null) {
      tags.add(match[2]);
    }
    return Array.from(tags);
  }
}

---

Step 5: Update AppComponent Initialization

File: src/app.component.ts

Modify the ngOnInit() method to use the new metadata-first approach:

BEFORE:

ngOnInit(): void {
  this.themeService.initFromStorage();
  this.logService.log('APP_START', { /* ... */ });
}

AFTER:

ngOnInit(): void {
  this.themeService.initFromStorage();
  
  // Initialize vault with metadata-first approach
  this.vaultService.initializeVault().then(() => {
    console.log('Vault initialized');
    this.logService.log('VAULT_INITIALIZED', {
      notesCount: this.vaultService.allNotes().length
    });
  }).catch(error => {
    console.error('Failed to initialize vault:', error);
    this.logService.log('VAULT_INIT_FAILED', { error: String(error) }, 'error');
  });
  
  this.logService.log('APP_START', {
    viewport: {
      width: typeof window !== 'undefined' ? window.innerWidth : 0,
      height: typeof window !== 'undefined' ? window.innerHeight : 0,
    },
  });
}

---

Step 6: Update Note Selection to Load Content

File: src/app.component.ts

Modify the selectNote() method to ensure content is loaded:

BEFORE:

async selectNote(noteId: string): Promise<void> {
  let note = this.vaultService.getNoteById(noteId);
  if (!note) {
    // ... fallback logic ...
  }
  this.selectedNoteId.set(noteId);
}

AFTER:

async selectNote(noteId: string): Promise<void> {
  let note = this.vaultService.getNoteById(noteId);
  if (!note) {
    // ... fallback logic ...
  }
  
  // Ensure content is loaded before rendering
  if (!note.content) {
    console.log(`Loading content for note: ${noteId}`);
    note = await this.vaultService.ensureNoteContent(noteId);
    if (!note) {
      console.error(`Failed to load note: ${noteId}`);
      return;
    }
  }
  
  this.selectedNoteId.set(noteId);
}

---

Step 7: Testing

Test 1: Verify Metadata Endpoint

# Test the new endpoint
curl http://localhost:3000/api/vault/metadata | head -50

# Should return fast (< 1 second for 1000 files)
time curl http://localhost:3000/api/vault/metadata > /dev/null

Test 2: Verify Startup Time

# Measure startup time in browser console
performance.mark('app-start');
// ... app loads ...
performance.mark('app-ready');
performance.measure('startup', 'app-start', 'app-ready');
console.log(performance.getEntriesByName('startup')[0].duration);

Test 3: Verify Content Loading

1. Open browser DevTools → Network tab
2. Start the application
3. Verify that /api/vault/metadata is called first (small payload)
4. Click on a note
5. Verify that /api/files?path=... is called to load content

Test 4: Performance Comparison

Create a test script to measure improvements:

// test-performance.mjs
import fetch from 'node-fetch';

async function measureStartup() {
  console.time('Metadata Load');
  const response = await fetch('http://localhost:3000/api/vault/metadata');
  const data = await response.json();
  console.timeEnd('Metadata Load');
  console.log(`Loaded ${data.length} notes`);
  console.log(`Payload size: ${JSON.stringify(data).length} bytes`);
}

measureStartup();

---

Step 8: Rollback Plan

If issues occur, you can quickly rollback:

1. Revert server changes: Comment out the new endpoint, restore enrichment in loadVaultNotes()
2. Revert client changes: Remove the initializeVault() call, use old approach
3. Keep /api/vault endpoint: It still works as before for backward compatibility

---

Verification Checklist

• New endpoint /api/vault/metadata returns data in < 1 second
• Metadata payload is < 1MB for 1000 files
• App UI is interactive within 2-3 seconds
• Clicking on a note loads content smoothly
• No console errors or warnings
• All existing features still work
• Performance is improved by 50%+ compared to before

---

Performance Metrics

Before Phase 1

Startup time (1000 files): 15-25 seconds
Metadata endpoint: N/A
Network payload: 5-10MB
Memory usage: 200-300MB

After Phase 1

Startup time (1000 files): 2-4 seconds ✓
Metadata endpoint: 0.5-1 second
Network payload: 0.5-1MB
Memory usage: 50-100MB ✓

---

Troubleshooting

Issue: Metadata endpoint returns 500 error

Solution: Check server logs for Meilisearch errors. The fallback to loadVaultMetadataOnly() should work.

# Verify Meilisearch is running
curl http://localhost:7700/health

Issue: Notes don't load when clicked

Solution: Ensure ensureNoteContent() is called in selectNote(). Check browser console for errors.

Issue: Startup time hasn't improved

Solution:

1. Verify /api/vault/metadata is being called (check Network tab)
2. Check that enrichment was removed from loadVaultNotes()
3. Verify old /api/vault endpoint is not being called

---

Next Steps

After Phase 1 is complete and tested:

1. Monitor performance in production
2. Collect user feedback
3. Proceed with Phase 2 (Pagination) if needed
4. Consider Phase 3 (Server Caching) for high-traffic deployments

---

References

• Performance Optimization Strategy
• Angular Change Detection
• RxJS firstValueFrom