ObsiViewer/docs/SEARCH/MEILISEARCH_SETUP.md

# 🔍 Meilisearch Setup Guide

Complete guide for testing and using the Meilisearch integration in ObsiViewer.

## Quick Start (5 minutes)

### 1. Start Meilisearch

```bash
npm run meili:up
```

This launches Meilisearch in Docker on port 7700.

### 2. Index Your Vault

```bash
npm run meili:reindex
```

This indexes all `.md` files from your vault. Progress will be logged to console.

### 3. Test the API

```bash
# Simple search
curl "http://localhost:4000/api/search?q=obsidian"

# With operators
curl "http://localhost:4000/api/search?q=tag:work path:Projects"

# All notes
curl "http://localhost:4000/api/search?q=*&limit=5"
```

### 4. Enable in Angular (Optional)

Edit `src/core/logging/environment.ts`:

```typescript
export const environment = {
  USE_MEILI: true,  // ← Change this to true
  // ...
};
```

Then restart your Angular dev server: `npm run dev`

## Testing Checklist

### ✅ Backend Tests

- [ ] **Docker started**: `docker ps` shows `obsiviewer-meilisearch` running
- [ ] **Indexing works**: `npm run meili:reindex` completes without errors
- [ ] **Basic search**: `curl "http://localhost:4000/api/search?q=*"` returns hits
- [ ] **Tag filter**: `curl "http://localhost:4000/api/search?q=tag:yourTag"` works
- [ ] **Path filter**: `curl "http://localhost:4000/api/search?q=path:YourFolder"` works
- [ ] **File filter**: `curl "http://localhost:4000/api/search?q=file:readme"` works
- [ ] **Highlights**: Response includes `_formatted` with `<mark>` tags
- [ ] **Facets**: Response includes `facetDistribution` with tags, parentDirs
- [ ] **Performance**: `npm run bench:search` shows P95 < 150ms

### ✅ Incremental Updates

Test that Chokidar automatically updates Meilisearch:

1. Start server: `node server/index.mjs`
2. Create a new `.md` file in vault with tag `#test-meilisearch`
3. Search: `curl "http://localhost:4000/api/search?q=tag:test-meilisearch"`
4. Verify the new file appears in results
5. Edit the file, search again - changes should be reflected
6. Delete the file, search again - should no longer appear

### ✅ Angular Integration

With `USE_MEILI: true`:

1. Open app in browser: `http://localhost:4200`
2. Use search bar
3. Open browser DevTools Network tab
4. Verify requests go to `/api/search` instead of local search
5. Results should show server-side highlights

## Query Examples

### Basic Searches

```bash
# Find all notes
curl "http://localhost:4000/api/search?q=*"

# Simple text search
curl "http://localhost:4000/api/search?q=obsidian"

# Phrase search
curl "http://localhost:4000/api/search?q=search%20engine"
```

### Operators

```bash
# Single tag
curl "http://localhost:4000/api/search?q=tag:dev"

# Multiple tags (both must match)
curl "http://localhost:4000/api/search?q=tag:dev tag:angular"

# Path filter
curl "http://localhost:4000/api/search?q=path:Projects/Angular"

# File name search
curl "http://localhost:4000/api/search?q=file:readme"

# Combined: tag + path + text
curl "http://localhost:4000/api/search?q=tag:dev path:Projects typescript"
```

### Typo Tolerance

```bash
# Intentional typos (should still match)
curl "http://localhost:4000/api/search?q=obsever"  # → "observer"
curl "http://localhost:4000/api/search?q=searh"    # → "search"
```

### Pagination & Sorting

```bash
# Limit results
curl "http://localhost:4000/api/search?q=*&limit=10"

# Offset (page 2)
curl "http://localhost:4000/api/search?q=*&limit=10&offset=10"

# Sort by date (newest first)
curl "http://localhost:4000/api/search?q=*&sort=updatedAt:desc"

# Sort by title
curl "http://localhost:4000/api/search?q=*&sort=title:asc"
```

## Performance Benchmarking

Run the benchmark suite:

```bash
npm run bench:search
```

This tests 5 different queries with 20 concurrent connections for 10 seconds each.

**Expected Results** (on modern dev machine with 1000 notes):
- P95: < 150ms ✅
- Average: 50-100ms
- Throughput: 200+ req/sec

If P95 exceeds 150ms, check:
1. Is Meilisearch running? `docker ps`
2. Is the index populated? Check `/health` endpoint
3. System resources (CPU/RAM)
4. Adjust `typoTolerance` or `searchableAttributes` in `meilisearch.client.mjs`

## Troubleshooting

### Meilisearch won't start

```bash
# Check if port 7700 is already in use
netstat -an | findstr 7700  # Windows
lsof -i :7700               # macOS/Linux

# View Meilisearch logs
docker logs obsiviewer-meilisearch

# Restart Meilisearch
npm run meili:down
npm run meili:up
```

### Search returns empty results

```bash
# Check if index exists and has documents
curl http://localhost:7700/indexes/notes_vault/stats \
  -H "Authorization: Bearer dev_meili_master_key_change_me"

# Reindex if needed
npm run meili:reindex
```

### "Connection refused" errors

Ensure Meilisearch is running:

```bash
docker ps | grep meilisearch
# Should show: obsiviewer-meilisearch ... Up ... 0.0.0.0:7700->7700/tcp
```

### Changes not reflected immediately

Wait 1-2 seconds after file changes for Chokidar to process. Check server logs:

```
[Meili] Upserted: Projects/MyNote.md
```

### Performance issues

1. **Reduce typo tolerance**: Edit `server/meilisearch.client.mjs`
   ```javascript
   typoTolerance: {
     enabled: true,
     minWordSizeForTypos: {
       oneTypo: 5,    // Was 3
       twoTypos: 9    // Was 6
     }
   }
   ```

2. **Limit searchable attributes**: Remove `headings` or `properties.*` if not needed

3. **Increase batch size**: Edit `server/meilisearch-indexer.mjs`
   ```javascript
   const batchSize = 1000;  // Was 750
   ```

## Environment Variables

### Development (local)

`.env` or shell:
```bash
VAULT_PATH=./vault
MEILI_HOST=http://127.0.0.1:7700
MEILI_API_KEY=dev_meili_master_key_change_me
```

### Docker Compose

`docker-compose/.env`:
```env
MEILI_MASTER_KEY=dev_meili_master_key_change_me
MEILI_ENV=development
```

### Production

**Important**: Change the master key in production!

```bash
MEILI_MASTER_KEY=$(openssl rand -base64 32)
MEILI_ENV=production
```

## API Reference

### GET /api/search

**Query Parameters**:
- `q` (string, required): Search query with optional operators
- `limit` (number, default: 20): Max results to return
- `offset` (number, default: 0): Pagination offset
- `sort` (string): Sort field and direction, e.g., `updatedAt:desc`
- `highlight` (boolean, default: true): Include `<mark>` highlights

**Response**:
```json
{
  "hits": [
    {
      "id": "Projects/MyNote.md",
      "title": "My Note",
      "path": "Projects/MyNote.md",
      "file": "MyNote.md",
      "tags": ["dev", "typescript"],
      "excerpt": "First 500 chars...",
      "_formatted": {
        "title": "My <mark>Note</mark>",
        "content": "...search <mark>term</mark>..."
      }
    }
  ],
  "estimatedTotalHits": 42,
  "facetDistribution": {
    "tags": { "dev": 15, "typescript": 8 },
    "parentDirs": { "Projects": 42 }
  },
  "processingTimeMs": 12,
  "query": "note"
}
```

### POST /api/reindex

Triggers a full reindex of all markdown files in the vault.

**Response**:
```json
{
  "ok": true,
  "indexed": true,
  "count": 1247,
  "elapsedMs": 3421
}
```

## Next Steps

### P1 Features (Future)

- [ ] Multi-language support (configure `locale` in Meilisearch)
- [ ] Synonyms configuration
- [ ] Stop words for common terms
- [ ] Advanced operators: `line:`, `section:`, `[property]:value`
- [ ] Faceted search UI (tag/folder selectors)
- [ ] Search result pagination in Angular
- [ ] Search analytics and logging
- [ ] Incremental indexing optimization (delta updates)

### Performance Optimization

- [ ] Redis cache layer for frequent queries
- [ ] CDN for static assets
- [ ] Index sharding for very large vaults (10k+ notes)
- [ ] Lazy loading of highlights (fetch on demand)

## Support

If you encounter issues:

1. Check server logs: `node server/index.mjs` (console output)
2. Check Meilisearch logs: `docker logs obsiviewer-meilisearch`
3. Verify environment variables are set correctly
4. Ensure vault path is correct and contains `.md` files
5. Test with curl before testing in Angular