diff --git a/docs/API_MCP_GUIDE.md b/docs/API_MCP_GUIDE.md new file mode 100644 index 0000000..1b3710b --- /dev/null +++ b/docs/API_MCP_GUIDE.md @@ -0,0 +1,303 @@ +# NewTube — Guide complet API REST + MCP + +> Référence développeur et LLM : tous les endpoints REST, le nouveau serveur MCP (dont transcript), exemples `curl` et JSON-RPC, auth, rate-limits, erreurs, variables d'environnement, tests. +> +> Fichiers sources : `server/index.mjs` (~3800 lignes), `server/transcript.mjs`, `server/providers/registry.mjs`, `mcp/server.mjs`, `api/README.md`. + +--- + +## Table des matières + +1. [Vue d'ensemble](#1-vue-densemble) +2. [Démarrage rapide](#2-démarrage-rapide) +3. [Conventions API](#3-conventions-api) +4. [Authentification](#4-authentification) +5. [Référence REST — lecture publique](#5-référence-rest--lecture-publique) +6. [Référence REST — transcript (détaillé)](#6-référence-rest--transcript-détaillé) +7. [Référence REST — espace utilisateur (JWT)](#7-référence-rest--espace-utilisateur-jwt) +8. [Référence REST — OAuth / import](#8-référence-rest--oauth--import) +9. [Référence REST — IA / proxies / santé](#9-référence-rest--ia--proxies--santé) +10. [Erreurs & rate-limits](#10-erreurs--rate-limits) +11. [Serveur MCP — installation](#11-serveur-mcp--installation) +12. [Serveur MCP — outils](#12-serveur-mcp--outils) +13. [MCP — transcript pas à pas](#13-mcp--transcript-pas-à-pas) +14. [Variables d'environnement](#14-variables-denvironnement) +15. [Tests](#15-tests) +16. [Dépannage](#16-dépannage) + +--- + +## 1. Vue d'ensemble + +``` +┌────────────┐ stdio JSON-RPC ┌──────────────┐ HTTP fetch ┌─────────────┐ +│ Claude / │ ──────────────────► │ mcp/server │ ───────────────► │ Express API │ +│ VS Code / │ tools/call │ .mjs (7 │ NEWTUBE_API_URL │ :4000 /api │ +│ opencode │ ◄────────────────── │ outils) │ ◄─────────────── │ + SQLite │ +└────────────┘ tools/list └──────────────┘ └─────────────┘ + ▲ ▲ + sans dépendance front Angular (dist/) + Node 20+ fetch natif yt-dlp + InnerTube +``` + +- **API REST** : Express, SQLite (`better-sqlite3`), JWT + cookies `sid`/`refreshToken`, montée sous `/api` **et** `/proxy/api` (même router `r` — le front prod appelle `/proxy/api`). +- **MCP** : nouveau `mcp/server.mjs`, **zéro-dépendance**, protocole `2024-11-05`, transport stdio. C'est un client HTTP fin vers l'API : toute la logique métier (InnerTube, yt-dlp, cache, quotas) reste côté API. + +IDs providers : +| Court | Long | Transcript | +|---|---|---| +| `yt` | `youtube` | ✅ (InnerTube + yt-dlp) | +| `dm` | `dailymotion` | ✅ | +| `pt` | `peertube` | ✅ (exige `instance`) | +| `tw` | `twitch` | ⚠️ pistes quasi inexistantes → `provider_unsupported` | +| `od` | `odysee` | ⚠️ idem | +| `ru` | `rumble` | ⚠️ idem | + +## 2. Démarrage rapide + +```bash +cp .env.example .env.local # renseigner JWT_SECRET, clés… +npm install +npm run api # API http://localhost:4000 (+ front dist/) +# autre terminal : +npm run mcp # MCP stdio (nécessite l'API) +# ou avec .env.local : +npm run mcp:dev +``` + +Santé : + +```bash +curl http://localhost:4000/healthz | jq +curl http://localhost:4000/api/search?q=lofi&providers=yt&limit=5 | jq +``` + +## 3. Conventions API + +- Base : `http://localhost:4000/api` (dev) ou `/api` / `/proxy/api` (prod, même chose). +- JSON partout (même les 429 : `{ error, retryAfterSec }` ou `{ available:false, error }`). +- Auth : `Authorization: Bearer ` **ou** cookies `sid`+`refreshToken` (routes `authMiddlewareCookieAware` : downloads, likes, subscriptions, playlists). +- Pagination : `page` (défaut 1), `pageSize`/`limit` (max 50). +- OpenAPI partiel : `GET /api/openapi.json` (playlists uniquement, à étendre). + +## 4. Authentification + +| Méthode | Route | Body | Réponse | +|---|---|---|---| +| POST | `/api/auth/register` | `{ username, email?, password }` | `{ user, accessToken }` + cookies | +| POST | `/api/auth/login` | `{ username|email, password, rememberMe? }` | `{ user, accessToken }` + cookies | +| POST | `/api/auth/refresh` | cookies | `{ accessToken }` | +| POST | `/api/auth/logout` | — | `{ ok:true }` + clear cookies | +| GET | `/api/user/me` | JWT | `{ id, username, email, … }` | +| GET/DELETE | `/api/auth/sessions`, `/api/auth/sessions/:id` | JWT | sessions actives | + +```bash +TOKEN=$(curl -s -X POST http://localhost:4000/api/auth/login \ + -H 'Content-Type: application/json' \ + -d '{"username":"demo","password":"demo123"}' | jq -r .accessToken) +curl -H "Authorization: Bearer $TOKEN" http://localhost:4000/api/user/me +``` + +Durées : `ACCESS_TTL_MIN` (15), `REFRESH_TTL_DAYS` (2), `REMEMBER_TTL_DAYS` (défaut 30). `JWT_SECRET` **obligatoire en prod**. + +## 5. Référence REST — lecture publique + +### 5.1 `GET /api/search` — recherche unifiée +``` +?q=lofi&providers=yt,dm&page=1&pageSize=24&sort=relevance|date|views +``` +Réponse `{ q, providers, groups:{ yt:[…], dm:[…] }, errors:{}, page, pageSize, sort }`. Item : `{ id,title,thumbnail,uploaderName,url,type,duration,isShort }`. Fan-out parallèle, `Promise.allSettled` (un provider KO → `groups[x]=[]` + `errors[x]`). `400` si `q<2`. + +### 5.2 `GET /api/search/suggest` — typeahead +``` +?q=lof&providers=yt,dm&limit=10 +``` +→ `{ q, groups:{ yt:[…], dm:[…], web:[…] } }`. Cache 5 min, rate-limit 60/min. Bonus sans clé : `web` (Bing/Google) + vrais titres Odysee (Lighthouse) si `od` demandé. + +### 5.3 `GET /api/details/:provider/:videoId` — métadonnées +Providers longs (`youtube|dailymotion|twitch|peertube|odysee|rumble`). Query : `instance` (PeerTube obligatoire), `slug` (Odysee), `sourceUrl` (prioritaire), `related=0` (désactive connexes). +Réponse `{ videoId,title,thumbnail,uploaderName,uploaderAvatar,views,duration,uploadedDate,description,url,related[] }` (`related` = watch-next InnerTube, YouTube uniquement, best-effort). + +```bash +curl "http://localhost:4000/api/details/youtube/dQw4w9WgXcQ" | jq '{title,uploaderName,views,related: (.related|length)}' +``` + +### 5.4 `GET /api/trending?provider=yt&limit=24` +Tendances YT sans clé (scrape). Phase 1 : `yt` uniquement (`400` sinon), fallback `{ items:[] }` jamais 500. + +### 5.5 Rumble dédié (`/api/rumble/…`) +`GET /browse`, `GET /search?q=…`, `GET /video/:videoId`, `GET /video/:videoId/preplay`. Rate-limit 10/min. + +## 6. Référence REST — transcript détaillé + +### 6.1 `GET /api/transcript/:provider/:videoId` + +``` +GET /api/transcript/youtube/dQw4w9WgXcQ?lang=fr +GET /api/transcript/peertube/abc123?lang=fr&instance=peertube.example.com +GET /api/transcript/odysee/mavideo?lang=en&slug=@chaine:mavideo +``` + +Query : `lang` (défaut `fr`, max 12 chars, filtrée par langues autorisées), `instance`, `slug`, `sourceUrl`, `langs` (override explicite `?langs=fr,en`, sinon préférences utilisateur si JWT, sinon `fr,en`). + +**Succès 200 :** +```json +{ "lang": "fr", "available": true, "languages": ["fr","en"], + "lines": [{ "t": 0.0, "dur": 2.5, "text": "Bonjour à tous" }] } +``` + +**Absence définitive 200 :** +```json +{ "lang": null, "available": false, "languages": ["en"], "lines": [], "reason": "no_subtitles" } +{ "lang": null, "available": false, "languages": [], "lines": [], "reason": "provider_unsupported" } +``` + +**Transitoire 502 (à réessayer) :** +```json +{ "available": false, "languages": ["fr","en"], "lines": [], + "error": "transcript_temporarily_unavailable", "retryable": true, "rateLimited": true } +``` + +**Autres :** `400 { available:false, error:"invalid_provider_or_video" }`, `429 { available:false, error:"rate_limited" }` (10/min). + +Pipeline YouTube : découverte InnerTube (`YT_TRANSCRIPT_SOURCE=innertube-first|ytdlp-only|innertube-only`, 0 piste → `no_subtitles` sans yt-dlp) → candidats ordonnés (`orderedTracks` + `firstPerLanguage`, max 6-8) → fetch timedtext (retry 429 : 2 s puis 8 s) → traduction serveur `&tlang=` en dernier recours → fallback `yt-dlp --write-sub --sub-langs` (90 s max). Parsers : `json3` → `vtt` → XML (`srv1/2/3`, `ttml`, ``), dédup rolling-window auto-captions, `TRANSCRIPT_MAX_LINES` (5000). Cache LRU 200 entrées / 24 h (`TRANSCRIPT_CACHE_TTL`). Anti-ban datacenter : `YT_COOKIES_FILE`, `YT_PO_TOKEN`, `YT_EGRESS_PROXY`. + +Exemples : + +```bash +# JSON brut +curl "http://localhost:4000/api/transcript/youtube/VIDEO_ID?lang=fr" | jq '{lang,available,n:(.lines|length)}' +# Texte horodaté pour LLM (jq) +curl -s "http://localhost:4000/api/transcript/youtube/VIDEO_ID?lang=fr" \ + | jq -r '.lines[] | "[\(.t|floor)] \(.text)"' | head -50 +``` + +### 6.2 Historique transcripts (JWT) +`POST /api/user/history/transcripts`, `GET /api/user/history/transcripts`, `GET /api/user/history/transcripts/:provider/:videoId`, `DELETE …/:id`, `DELETE …`. + +## 7. Référence REST — espace utilisateur (JWT) + +- **Préférences** : `GET/PATCH /api/user/preferences` (`defaultProviders`, `downloadLanguages` — pilote aussi le transcript). +- **Playlists** : `GET/POST /api/playlists`, `GET/PUT/DELETE /api/playlists/:id`, `POST /api/playlists/:id/videos`, `DELETE /api/playlists/:id/videos/:videoId?provider=yt`, `PUT …/reorder`, `GET /api/playlists/public`, `GET /api/playlists/:id/view`. +- **Likes** : `GET/POST/DELETE /api/user/likes`, `GET /api/user/likes/status?provider=&videoId=`. +- **Historiques** : `/api/user/history/search` (GET/POST/batch/DELETE), `/api/user/history/watch` (GET/POST/PATCH/DELETE), takeout `POST /api/user/history/takeout`. +- **Abonnements** : `GET/POST /api/subscriptions`, `POST /api/subscriptions/batch`, `DELETE /api/subscriptions/:id`, groupes `GET/POST/PATCH/DELETE /api/subscription-groups`, `PUT …/members`, `PUT /api/subscriptions/:id/groups`, resolve `POST /api/channels/resolve`, `GET /api/channels/:provider/:externalId[/content]`. +- **Téléchargements** (JWT cookie-aware) : `GET /api/download/:provider/:videoId/formats` (cache 10 min), `POST /api/download/:provider/:videoId`, `GET /api/download/jobs`, `GET /api/download/jobs/:id[/file]`, `POST /api/download/jobs/:id/retry`, `DELETE /api/download/jobs/:id`. Quotas : `DOWNLOAD_MAX_CONCURRENT=2`, `DOWNLOAD_STORAGE_QUOTA_BYTES=5GiB`, `DOWNLOAD_QUOTA_WINDOW_MS=30j`, `DOWNLOAD_PROVIDERS`. +- **Télémétrie** : `POST /api/telemetry/events` (whitelist : `search_submit`, `provider_picker_open`, `provider_apply`, `at_autocomplete_use`, `quick_menu_open`, `suggest_shown`, `suggest_used`), `GET /api/telemetry/events|summary`. + +## 8. Référence REST — OAuth / import + +Statut `GET /api/oauth/status`, URL `GET /api/oauth/:provider/url` (JWT), callback `GET /api/oauth/:provider/callback`, `GET /api/oauth/connections`, `DELETE /api/oauth/:provider`, preview `GET /api/oauth/:provider/preview`, import `POST /api/oauth/:provider/import`, Google : `GET /api/oauth/google/yt-channels|diag|watchlater|yt-history`, `PUT /api/oauth/google/yt-channel`, `POST /api/oauth/google/watchlater`, takeout `POST /api/user/history/takeout`. Env : `GOOGLE_CLIENT_ID/SECRET/REDIRECT_URI`, `TWITCH_CLIENT_ID/SECRET`, `OAUTH_APP_BASE_URL`. + +## 9. Référence REST — IA / proxies / santé + +- `GET /api/ai/status` → `{ ready }` ; `POST /api/ai/summarize` `{ comments: string[] }` → `{ summary }` (clé `GEMINI_API_KEY` côté serveur, 10/min). +- `GET /api/twitch-token` (secret jamais exposé), `GET /assets/config.js` (whitelist front uniquement). +- Proxies CORS : `ALL /api/dm/*`, `/api/odysee/*`, `/api/twitch-api/*`, `/api/twitch-auth/*`. +- Santé : `GET /api/health` (`{status:ok}`), `GET /healthz|/api/healthz` (mode YT, yt-dlp `binOk`+version, cookies/PO-token/proxy, cache mémoire+SQLite, métriques quota jour, clés `...XXXX` bannies — aucun secret). + +## 10. Erreurs & rate-limits + +| Cas | Code | Corps | +|---|---|---| +| Requête courte | 400 | `{ error:"q is required…" }` | +| Provider/vidéo invalide | 400 | `{ available:false, error:"invalid_provider_or_video" }` | +| Non authentifié | 401 | `{ error:"Unauthorized" }` | +| Provider download désactivé | 403 | `{ error:"download_disabled_for_provider" }` | +| Trop de requêtes | 429 | `{ error:"rate_limited", retryAfterSec }` ou `{ available:false, error:"rate_limited" }` | +| YouTube 429 épuisé | 502 | `{ available:false, error:"transcript_temporarily_unavailable", retryable:true }` | +| Erreur interne | 500 | `{ error:"search_failed"|"details_failed", details }` | + +Seaux : login 5/min, suggest 60/min, transcript 10/min, downloads lecture 120 / écriture 30 / formats 30, AI 10/min, Rumble 10/min. + +## 11. Serveur MCP — installation + +Prérequis : Node ≥ 20, API démarrée. Aucune dépendance à installer (fetch natif). + +```bash +npm run mcp # stdio, NEWTUBE_API_URL=http://localhost:4000/api +npm run mcp:dev # charge .env.local +NEWTUBE_API_URL=https://mon-serveur/api NEWTUBE_TOKEN= npm run mcp +``` + +**Claude Desktop** (`%APPDATA%\Claude\claude_desktop_config.json`) : +```json +{ "mcpServers": { "newtube": { + "command": "node", + "args": ["C:/dev/git/web/NewTube/mcp/server.mjs"], + "env": { "NEWTUBE_API_URL": "http://localhost:4000/api" } +} } } +``` + +**VS Code** (`.vscode/mcp.json`) / **opencode** (`opencode.json`) : même `command`/`args`/`env`. Redémarrer le client, vérifier `tools/list` → 7 outils. + +## 12. Serveur MCP — outils + +| Outil | Args | Appel REST sous-jacent | +|---|---|---| +| `search_videos` | `q*`, `providers`, `page`, `pageSize` (≤50), `sort` | `GET /search` | +| `suggest_queries` | `q*`, `providers`, `limit` (≤20) | `GET /search/suggest` | +| `get_video_details` | `provider*`, `videoId*`, `instance`, `slug`, `sourceUrl`, `related=true` | `GET /details/:p/:id` | +| `get_trending` | `provider=yt`, `limit` (≤50) | `GET /trending` | +| `get_transcript` | `provider*`, `videoId*`, `lang=fr`, `instance`, `slug`, `sourceUrl` | `GET /transcript/:p/:id` (JSON brut) | +| `get_transcript_text` | + `format=text\|srt`, `withTimestamps=true`, `maxChars=12000` | idem + mise en forme LLM | +| `health_check` | — | `GET /healthz` | + +Réponse MCP : `{ content:[{ type:"text", text:"\n\n\n\n--- JSON ---\n" }], isError? }`. Les erreurs API (connexion refusée, 429, 502 retryable) remontent en `isError:true` avec message actionnable (`npm run api`, réessayer…). + +Exemple JSON-RPC (stdio, 1 ligne = 1 message) : +```json +{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}} +{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}} +{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"get_transcript_text","arguments":{"provider":"youtube","videoId":"dQw4w9WgXcQ","lang":"fr","maxChars":8000}}} +``` + +Test manuel : +```bash +printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' | node ./mcp/server.mjs +``` + +## 13. MCP — transcript pas à pas + +1. `get_transcript(provider=youtube, videoId=…, lang=fr)` → diagnostic : `available`, `lang` effectif, `languages`, `lines.length`. +2. Si `available:false` + `reason=no_subtitles|provider_unsupported` → stop (définitif). Si `error=transcript_temporarily_unavailable, retryable:true` → attendre 30-60 s et réessayer (429 YouTube). +3. Sinon `get_transcript_text(format=text, withTimestamps=true, maxChars=12000)` → résumé/analyse/Q&A côté LLM. PeerTube : ne pas oublier `instance`. Odysee : `slug` si besoin. +4. Pour sous-titrage vidéo : `format=srt`. + +Recette LLM type : `search_videos("quantique", providers=yt)` → `get_video_details` (durée/vues) → `get_transcript_text` → synthèse horodatée `[00:12:30] …`. + +## 14. Variables d'environnement + +| Var | Défaut | Rôle | +|---|---|---| +| `PORT` | 4000 | API | +| `JWT_SECRET` | `dev-secret…` | **à définir en prod** | +| `NEWTUBE_API_URL` (MCP) | `http://localhost:4000/api` | cible du MCP | +| `NEWTUBE_TOKEN` (MCP) | — | JWT pour futurs outils authentifiés | +| `MCP_TOOL_TIMEOUT_MS` | 60000 | timeout fetch MCP | +| `YT_SEARCH_MODE` | `innertube-first` | `innertube-first\|only\|scrape-first\|api-first\|…` | +| `YT_TRANSCRIPT_SOURCE` | `innertube-first` | `innertube-first\|ytdlp-only\|innertube-only` | +| `YT_COOKIES_FILE`, `YT_PO_TOKEN`, `YT_EGRESS_PROXY` | — | anti-ban 429 | +| `TRANSCRIPT_CACHE_TTL`, `TRANSCRIPT_RATE_LIMIT` | 24 h, 10/min | transcript | +| `SUGGEST_CACHE_TTL_MS`, `SUGGEST_RATE_LIMIT` | 5 min, 60/min | suggest | +| `DOWNLOAD_*` | 2 conc., 5 GiB/30 j | downloads | +| `YOUTUBE_API_KEY(S)`, `TWITCH_*`, `GEMINI_API_KEY`, `GOOGLE_*` | — | clés providers/IA/OAuth | + +## 15. Tests + +```bash +npm run test:transcript # parseurs json3/vtt + contrat transcript +npm run test:suggest # typeahead + contrat suggest +npm run test:search-e2e # serveur réel isolé +npm run test:mcp # protocole MCP (initialize/list/validation) — nouveau +npm run test:playlists && npm run test:downloads && npm run test:telemetry +``` + +## 16. Dépannage + +- **MCP `API injoignable`** → `npm run api` d'abord, vérifier `NEWTUBE_API_URL` (sans `/` final géré, préfixe `/api` inclus). +- **Transcript 502 retryable** → 429 YouTube : attendre, réduire la fréquence, configurer `YT_COOKIES_FILE` + `YT_EGRESS_PROXY`, `YT_TRANSCRIPT_SOURCE=ytdlp-only` en test. +- **`provider_unsupported` (tw/od/ru)** → normal, pas de pistes yt-dlp. +- **`peertube_instance_required`** → passer `instance` (query REST ou arg MCP). +- **429 `rate_limited`** → respecter `Retry-After`/`retryAfterSec`. +- **stdout du MCP pollué** → le serveur log sur **stderr** uniquement ; ne jamais `console.log` sur stdout (canal JSON-RPC). diff --git a/mcp/server.mjs b/mcp/server.mjs new file mode 100644 index 0000000..0407246 --- /dev/null +++ b/mcp/server.mjs @@ -0,0 +1,330 @@ +#!/usr/bin/env node +/** + * NewTube MCP Server (stdio, zero-dependency). + * + * Expose l'API REST NewTube comme outils MCP : + * - search_videos, suggest_queries, get_video_details, get_trending + * - get_transcript, get_transcript_text (JSON + texteLLM-friendly) + * - health_check + * + * Protocole : JSON-RPC 2.0 sur stdio (1 objet JSON par ligne), compatible + * MCP 2024-11-05 : initialize / notifications/initialized / tools/list / + * tools/call / ping. + * + * Prérequis : API NewTube démarrée (`npm run api`, défaut http://localhost:4000/api). + * + * Env : + * NEWTUBE_API_URL (défaut http://localhost:4000/api) + * NEWTUBE_TOKEN (JWT optionnel -> Authorization: Bearer, requis pour /download, /user/*) + * MCP_TOOL_TIMEOUT_MS (défaut 60000, transcript = appels yt-dlp longs) + * + * Config clients : + * Claude Desktop / VS Code / opencode : + * { "mcpServers": { "newtube": { "command": "node", "args": ["C:/dev/git/web/NewTube/mcp/server.mjs"], + * "env": { "NEWTUBE_API_URL": "http://localhost:4000/api" } } } } + */ + +const API_BASE = (process.env.NEWTUBE_API_URL || 'http://localhost:4000/api').replace(/\/+$/, ''); +const TOKEN = (process.env.NEWTUBE_TOKEN || '').trim(); +const TIMEOUT_MS = Number(process.env.MCP_TOOL_TIMEOUT_MS || 60000); +const SERVER_VERSION = '1.0.0'; + +const PROVIDERS = ['yt', 'dm', 'tw', 'pt', 'od', 'ru']; +const TRANSCRIPT_PROVIDERS = ['youtube', 'yt', 'dailymotion', 'dm', 'peertube', 'pt', 'twitch', 'tw', 'odysee', 'od', 'rumble', 'ru']; + +// ---------------------------------------------------------------- HTTP helper + +async function apiFetch(path, { method = 'GET', body, auth = false } = {}) { + const url = `${API_BASE}${path.startsWith('/') ? path : `/${path}`}`; + const headers = { Accept: 'application/json', 'Content-Type': 'application/json' }; + if (auth && TOKEN) headers.Authorization = `Bearer ${TOKEN}`; + const ctrl = new AbortController(); + const timer = setTimeout(() => ctrl.abort(), TIMEOUT_MS); + try { + const res = await fetch(url, { + method, + headers, + signal: ctrl.signal, + ...(body !== undefined ? { body: JSON.stringify(body) } : {}), + }); + const text = await res.text(); + let data; + try { data = text ? JSON.parse(text) : null; } + catch { data = { _raw: text.slice(0, 2000) }; } + if (!res.ok) { + const msg = data?.error || data?.message || `http_${res.status}`; + const details = data?.details ? ` — ${String(data.details).slice(0, 300)}` : ''; + throw new Error(`${msg}${details} (HTTP ${res.status} ${method} ${path})`); + } + return data; + } catch (e) { + if (e?.name === 'AbortError') throw new Error(`timeout_after_${TIMEOUT_MS}ms (${method} ${path}) — API démarrée ? npm run api`); + if (e?.cause?.code === 'ECONNREFUSED') throw new Error(`API injoignable à ${API_BASE} — lancez 'npm run api' d'abord`); + throw e; + } finally { + clearTimeout(timer); + } +} + +function fmtTs(sec) { + const s = Math.max(0, Number(sec || 0)); + const h = Math.floor(s / 3600), m = Math.floor((s % 3600) / 60), ss = Math.floor(s % 60); + return `${String(h).padStart(2, '0')}:${String(m).padStart(2, '0')}:${String(ss).padStart(2, '0')}`; +} + +function transcriptToText(data, { withTimestamps = true, maxChars = 12000 } = {}) { + const lines = Array.isArray(data?.lines) ? data.lines : []; + let out = lines.map((l) => (withTimestamps ? `[${fmtTs(l.t)}] ${l.text}` : String(l.text || ''))).join('\n'); + if (out.length > maxChars) out = out.slice(0, maxChars) + '\n…[tronqué]'; + return out; +} + +function transcriptToSrt(data) { + const lines = Array.isArray(data?.lines) ? data.lines : []; + const fmt = (s) => { + const ms = Math.round(Number(s || 0) * 1000); + const h = String(Math.floor(ms / 3600000)).padStart(2, '0'); + const m = String(Math.floor((ms % 3600000) / 60000)).padStart(2, '0'); + const sec = String(Math.floor((ms % 60000) / 1000)).padStart(2, '0'); + const milli = String(ms % 1000).padStart(3, '0'); + return `${h}:${m}:${sec},${milli}`; + }; + return lines.map((l, i) => `${i + 1}\n${fmt(l.t)} --> ${fmt(Number(l.t) + Number(l.dur || 2))}\n${l.text}\n`).join('\n'); +} + +// ---------------------------------------------------------------- Tools + +const TOOLS = [ + { + name: 'search_videos', + description: 'Recherche unifiée multi-providers (YouTube, Dailymotion, Twitch, PeerTube, Odysee, Rumble). q min 2 caractères.', + inputSchema: { + type: 'object', + properties: { + q: { type: 'string', description: 'Requête (min 2 caractères)' }, + providers: { type: 'string', description: 'CSV parmi yt,dm,tw,pt,od,ru. Défaut: tous.' }, + page: { type: 'integer', minimum: 1, default: 1 }, + pageSize: { type: 'integer', minimum: 1, maximum: 50, default: 10 }, + sort: { type: 'string', enum: ['relevance', 'date', 'views'], default: 'relevance' }, + }, + required: ['q'], + }, + }, + { + name: 'suggest_queries', + description: 'Typeahead de requêtes groupées par provider. Idéal avant search_videos.', + inputSchema: { + type: 'object', + properties: { + q: { type: 'string', description: 'Début de requête (min 2 caractères)' }, + providers: { type: 'string', description: 'CSV providers, défaut tous' }, + limit: { type: 'integer', minimum: 1, maximum: 20, default: 10 }, + }, + required: ['q'], + }, + }, + { + name: 'get_video_details', + description: "Métadonnées d'une vidéo + vidéos connexes YouTube (watch-next InnerTube).", + inputSchema: { + type: 'object', + properties: { + provider: { type: 'string', description: 'youtube|dailymotion|twitch|peertube|odysee|rumble (alias yt,dm,tw,pt,od,ru acceptés)' }, + videoId: { type: 'string' }, + instance: { type: 'string', description: 'Requis pour PeerTube (ex. peertube.example.com)' }, + slug: { type: 'string', description: 'Slug Odysee si différent de videoId' }, + sourceUrl: { type: 'string', description: "URL canonique (prioritaire sur la reconstruction)" }, + related: { type: 'boolean', default: true, description: 'Inclure related[] YouTube (related=0 pour désactiver)' }, + }, + required: ['provider', 'videoId'], + }, + }, + { + name: 'get_trending', + description: 'Tendances YouTube sans clé API (scrape InnerTube, phase 1 : provider yt uniquement).', + inputSchema: { + type: 'object', + properties: { provider: { type: 'string', default: 'yt' }, limit: { type: 'integer', minimum: 1, maximum: 50, default: 10 } }, + }, + }, + { + name: 'get_transcript', + description: "Transcript brut JSON d'une vidéo : { lang, available, languages, lines: [{t,dur,text}] }. Langues filtrées par préférences serveur (défaut fr,en). 200 available:false si absent, 502 retryable:true si YouTube rate-limite.", + inputSchema: { + type: 'object', + properties: { + provider: { type: 'string', description: TRANSCRIPT_PROVIDERS.join(' | ') }, + videoId: { type: 'string' }, + lang: { type: 'string', default: 'fr', description: "Langue préférée (ex. fr, en, fr-CA). Fallback auto + traduction serveur &tlang" }, + instance: { type: 'string', description: 'Instance PeerTube si provider peertube' }, + slug: { type: 'string', description: 'Slug Odysee' }, + sourceUrl: { type: 'string', description: 'URL canonique (prioritaire)' }, + }, + required: ['provider', 'videoId'], + }, + }, + { + name: 'get_transcript_text', + description: "Transcript mis en forme pour LLM : texte plein horodaté (ou SRT), tronqué proprement. Recommandé pour résumer/analyser une vidéo.", + inputSchema: { + type: 'object', + properties: { + provider: { type: 'string' }, + videoId: { type: 'string' }, + lang: { type: 'string', default: 'fr' }, + instance: { type: 'string' }, + slug: { type: 'string' }, + sourceUrl: { type: 'string' }, + format: { type: 'string', enum: ['text', 'srt'], default: 'text' }, + withTimestamps: { type: 'boolean', default: true }, + maxChars: { type: 'integer', minimum: 500, maximum: 60000, default: 12000 }, + }, + required: ['provider', 'videoId'], + }, + }, + { + name: 'health_check', + description: "Santé API : mode YouTube (YT_SEARCH_MODE), binaire yt-dlp, cache, quota jour, clés. Équivalent GET /healthz.", + inputSchema: { type: 'object', properties: {} }, + }, +]; + +async function callTool(name, args = {}) { + switch (name) { + case 'search_videos': { + const q = String(args.q || '').trim(); + if (q.length < 2) throw new Error('invalid_query: q min 2 caractères'); + const qs = new URLSearchParams({ + q, + ...(args.providers ? { providers: String(args.providers) } : {}), + page: String(args.page ?? 1), + pageSize: String(args.pageSize ?? 10), + sort: String(args.sort ?? 'relevance'), + }); + const data = await apiFetch(`/search?${qs}`); + const counts = Object.fromEntries(Object.entries(data.groups || {}).map(([k, v]) => [k, v.length])); + return { summary: `${data.groups ? Object.values(data.groups).flat().length : 0} résultats pour "${q}" ${JSON.stringify(counts)}`, data }; + } + case 'suggest_queries': { + const q = String(args.q || '').trim(); + if (q.length < 2) throw new Error('invalid_query: q min 2 caractères'); + const qs = new URLSearchParams({ q, ...(args.providers ? { providers: String(args.providers) } : {}), limit: String(args.limit ?? 10) }); + return { summary: `Suggestions pour "${q}"`, data: await apiFetch(`/search/suggest?${qs}`) }; + } + case 'get_video_details': { + const provider = String(args.provider || '').toLowerCase(); + const videoId = String(args.videoId || ''); + if (!provider || !videoId) throw new Error('provider et videoId requis'); + const qs = new URLSearchParams({ + ...(args.instance ? { instance: String(args.instance) } : {}), + ...(args.slug ? { slug: String(args.slug) } : {}), + ...(args.sourceUrl ? { sourceUrl: String(args.sourceUrl) } : {}), + ...(args.related === false ? { related: '0' } : {}), + }); + const qstr = qs.toString() ? `?${qs}` : ''; + const data = await apiFetch(`/details/${encodeURIComponent(provider)}/${encodeURIComponent(videoId)}${qstr}`); + return { summary: `${data.title || videoId} — ${data.uploaderName || '?'} (${data.duration || 0}s, ${data.views || 0} vues)`, data }; + } + case 'get_trending': { + const provider = String(args.provider || 'yt'); + const limit = Math.min(50, Math.max(1, Number(args.limit ?? 10))); + const data = await apiFetch(`/trending?provider=${encodeURIComponent(provider)}&limit=${limit}`); + return { summary: `${(data.items || []).length} tendances (${provider})`, data }; + } + case 'get_transcript': + case 'get_transcript_text': { + const provider = String(args.provider || '').toLowerCase(); + const videoId = String(args.videoId || ''); + if (!provider || !videoId) throw new Error('provider et videoId requis'); + const qs = new URLSearchParams({ + ...(args.lang ? { lang: String(args.lang) } : {}), + ...(args.instance ? { instance: String(args.instance) } : {}), + ...(args.slug ? { slug: String(args.slug) } : {}), + ...(args.sourceUrl ? { sourceUrl: String(args.sourceUrl) } : {}), + }); + const qstr = qs.toString() ? `?${qs}` : ''; + const data = await apiFetch(`/transcript/${encodeURIComponent(provider)}/${encodeURIComponent(videoId)}${qstr}`); + if (name === 'get_transcript') { + const n = (data.lines || []).length; + const summary = data.available + ? `Transcript ${data.lang} : ${n} lignes, langues=${(data.languages || []).join(',')}` + : `Transcript indisponible (${data.reason || data.error || 'no_subtitles'}), langues=${(data.languages || []).join(',') || '—'}`; + return { summary, data }; + } + // text variant + if (!data.available) { + return { summary: `Transcript indisponible (${data.reason || data.error})`, data, text: '' }; + } + const format = args.format === 'srt' ? 'srt' : 'text'; + const text = format === 'srt' + ? transcriptToSrt(data).slice(0, Number(args.maxChars ?? 12000)) + : transcriptToText(data, { withTimestamps: args.withTimestamps !== false, maxChars: Number(args.maxChars ?? 12000) }); + return { summary: `Transcript ${data.lang} (${(data.lines || []).length} lignes) en ${format}`, data, text }; + } + case 'health_check': { + const data = await apiFetch('/healthz'); + return { summary: `API ${data.status} — YT mode=${data.youtube?.mode}, yt-dlp=${data.youtube?.ytdlp?.version || 'n/a'}`, data }; + } + default: + throw new Error(`unknown_tool: ${name}`); + } +} + +// ---------------------------------------------------------------- JSON-RPC stdio + +const readline = await import('node:readline'); +const rl = readline.createInterface({ input: process.stdin, crlfDelay: Infinity }); +let initialized = false; + +function send(obj) { + process.stdout.write(JSON.stringify(obj) + '\n'); +} +const ok = (id, result) => send({ jsonrpc: '2.0', id, result }); +const err = (id, code, message, data) => send({ jsonrpc: '2.0', id, error: { code, message, ...(data !== undefined ? { data } : {}) } }); + +function mcpText(summary, data, extraText) { + const payload = extraText ? `${summary}\n\n${extraText}\n\n--- JSON ---\n${JSON.stringify(data).slice(0, 8000)}` + : `${summary}\n\n${JSON.stringify(data).slice(0, 8000)}`; + return { content: [{ type: 'text', text: payload }] }; +} + +rl.on('line', async (line) => { + if (!line.trim()) return; + let msg; + try { msg = JSON.parse(line); } + catch { return; /* ignore */ } + const { id, method, params } = msg; + try { + if (method === 'initialize') { + ok(id, { + protocolVersion: '2024-11-05', + capabilities: { tools: {} }, + serverInfo: { name: 'newtube', version: SERVER_VERSION }, + }); + } else if (method === 'notifications/initialized') { + initialized = true; + // notification : pas de réponse + } else if (method === 'ping') { + ok(id, {}); + } else if (method === 'tools/list') { + ok(id, { tools: TOOLS }); + } else if (method === 'tools/call') { + const toolName = params?.name; + const toolArgs = params?.arguments || {}; + if (!TOOLS.find((t) => t.name === toolName)) { err(id, -32602, `unknown_tool: ${toolName}`); return; } + try { + const { summary, data, text } = await callTool(toolName, toolArgs); + ok(id, mcpText(summary, data, text)); + } catch (e) { + ok(id, { content: [{ type: 'text', text: `Erreur ${toolName} : ${e.message}` }], isError: true }); + } + } else { + if (id !== undefined) err(id, -32601, `method_not_found: ${method}`); + } + } catch (e) { + if (id !== undefined) err(id, -32603, String(e?.message || e)); + } +}); + +// Log de démarrage sur stderr (ne jamais polluer stdout = canal JSON-RPC) +console.error(`[newtube-mcp] v${SERVER_VERSION} — API=${API_BASE} (stdio)`); diff --git a/mcp/server.test.mjs b/mcp/server.test.mjs new file mode 100644 index 0000000..f545ff8 --- /dev/null +++ b/mcp/server.test.mjs @@ -0,0 +1,62 @@ +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import { spawn } from 'node:child_process'; + +function rpc(proc, obj) { + return new Promise((resolve) => { + const onData = (buf) => { + const lines = String(buf).split('\n').filter(Boolean); + for (const l of lines) { + try { + const msg = JSON.parse(l); + if (msg.id === obj.id) { + proc.stdout.off('data', onData); + resolve(msg); + return; + } + } catch {} + } + }; + proc.stdout.on('data', onData); + proc.stdin.write(JSON.stringify(obj) + '\n'); + }); +} + +describe('NewTube MCP server (stdio)', () => { + it('initialize + tools/list expose 7 outils dont transcript', async () => { + const proc = spawn(process.execPath, ['./mcp/server.mjs'], { env: { ...process.env, NEWTUBE_API_URL: 'http://127.0.0.1:9/api' } }); + try { + const init = await rpc(proc, { jsonrpc: '2.0', id: 1, method: 'initialize', params: {} }); + assert.equal(init.result.serverInfo.name, 'newtube'); + assert.ok(init.result.capabilities.tools); + + const list = await rpc(proc, { jsonrpc: '2.0', id: 2, method: 'tools/list', params: {} }); + const names = list.result.tools.map((t) => t.name); + for (const expected of ['search_videos', 'get_video_details', 'get_trending', 'get_transcript', 'get_transcript_text', 'health_check']) { + assert.ok(names.includes(expected), `outil manquant: ${expected}`); + } + const tr = list.result.tools.find((t) => t.name === 'get_transcript'); + assert.ok(tr.inputSchema.properties.provider); + assert.ok(tr.inputSchema.properties.videoId); + + const ping = await rpc(proc, { jsonrpc: '2.0', id: 3, method: 'ping', params: {} }); + assert.deepEqual(ping.result, {}); + } finally { + proc.kill(); + } + }); + + it('validation locale sans API (q trop court -> isError)', async () => { + const proc = spawn(process.execPath, ['./mcp/server.mjs'], { env: { ...process.env, NEWTUBE_API_URL: 'http://127.0.0.1:9/api' } }); + try { + const call = await rpc(proc, { + jsonrpc: '2.0', id: 10, method: 'tools/call', + params: { name: 'search_videos', arguments: { q: 'x' } }, + }); + assert.equal(call.result.isError, true); + assert.match(call.result.content[0].text, /min 2/); + } finally { + proc.kill(); + } + }); +}); diff --git a/mcp/tools.test.mjs b/mcp/tools.test.mjs new file mode 100644 index 0000000..e74000c --- /dev/null +++ b/mcp/tools.test.mjs @@ -0,0 +1,101 @@ +// Tests des 7 outils MCP contre un stub HTTP (aucune API réelle, aucun réseau). +// Stub -> NEWTUBE_API_URL, puis pilotage stdio JSON-RPC (tools/call). +// Run: npm run test:mcp (inclus via --test mcp/*.test.mjs si appelé en glob, +// sinon exécution directe node --test mcp/tools.test.mjs) +import { describe, it, before, after } from 'node:test'; +import assert from 'node:assert/strict'; +import http from 'node:http'; +import { spawn } from 'node:child_process'; + +const LINES = [ + { t: 1, dur: 2, text: 'Bonjour' }, + { t: 65, dur: 2.5, text: 'deuxième ligne' }, + { t: 120, dur: 2, text: `remplissage ${'x'.repeat(900)}` }, +]; + +const stub = http.createServer((req, res) => { + const u = new URL(req.url, 'http://x'); + const json = (code, obj) => { res.writeHead(code, { 'content-type': 'application/json' }); res.end(JSON.stringify(obj)); }; + if (u.pathname === '/api/healthz') return json(200, { status: 'ok', youtube: { mode: 'test' } }); + if (u.pathname === '/api/search') return json(200, { q: 'q', providers: ['yt'], groups: { yt: [{ id: 'v1', title: 'T' }] }, page: 1, pageSize: 10 }); + if (u.pathname === '/api/search/suggest') return json(200, { q: 'q', groups: { yt: ['abc'] } }); + if (u.pathname === '/api/details/youtube/v1') return json(200, { videoId: 'v1', title: 'Demo', uploaderName: 'U', duration: 60, views: 10 }); + if (u.pathname === '/api/trending') return json(200, { provider: 'yt', items: [{ id: 't1' }] }); + if (u.pathname === '/api/transcript/youtube/v1') return json(200, { lang: 'fr', available: true, languages: ['fr', 'en'], lines: LINES }); + if (u.pathname === '/api/transcript/youtube/none') return json(200, { lang: null, available: false, languages: [], lines: [], reason: 'no_subtitles' }); + return json(404, { error: 'not_found' }); +}); +await new Promise((r) => stub.listen(0, '127.0.0.1', r)); +const STUB_URL = `http://127.0.0.1:${stub.address().port}/api`; + +function startMcp() { + return spawn(process.execPath, ['./mcp/server.mjs'], { env: { ...process.env, NEWTUBE_API_URL: STUB_URL } }); +} +function rpc(proc, id, method, params = {}) { + return new Promise((resolve, reject) => { + const timer = setTimeout(() => reject(new Error(`timeout ${method}`)), 15000); + const onData = (buf) => { + for (const l of String(buf).split('\n')) { + if (!l.trim()) continue; + try { + const m = JSON.parse(l); + if (m.id === id) { clearTimeout(timer); proc.stdout.off('data', onData); resolve(m); return; } + } catch {} + } + }; + proc.stdout.on('data', onData); + proc.stdin.write(JSON.stringify({ jsonrpc: '2.0', id, method, params }) + '\n'); + }); +} +async function call(proc, id, name, args) { + const m = await rpc(proc, id, 'tools/call', { name, arguments: args }); + assert.ok(!m.error, `RPC error: ${JSON.stringify(m.error)}`); + assert.equal(m.result.isError, undefined, `tool ${name} isError: ${m.result.content?.[0]?.text?.slice(0, 300)}`); + return m.result.content[0].text; +} + +let proc; +before(async () => { + proc = startMcp(); + await rpc(proc, 1, 'initialize'); +}); +after(() => { try { proc.kill(); } catch {} stub.close(); }); + +describe('MCP tools (stub API)', () => { + it('search_videos / suggest_queries / get_video_details / get_trending / health_check', async () => { + const t1 = await call(proc, 11, 'search_videos', { q: 'test' }); + assert.match(t1, /r.sultats|résultats|1 r/); + const t2 = await call(proc, 12, 'suggest_queries', { q: 'ab' }); + assert.match(t2, /Suggestions/); + const t3 = await call(proc, 13, 'get_video_details', { provider: 'youtube', videoId: 'v1' }); + assert.match(t3, /Demo/); + const t4 = await call(proc, 14, 'get_trending', { provider: 'yt', limit: 5 }); + assert.match(t4, /tendances/); + const t5 = await call(proc, 15, 'health_check', {}); + assert.match(t5, /test/); + }); + it('get_transcript JSON brut', async () => { + const t = await call(proc, 21, 'get_transcript', { provider: 'youtube', videoId: 'v1', lang: 'fr' }); + assert.match(t, /3 lignes/); + assert.match(t, /Bonjour/); + }); + it('get_transcript indisponible -> message clair, pas d erreur', async () => { + const t = await call(proc, 22, 'get_transcript', { provider: 'youtube', videoId: 'none' }); + assert.match(t, /indisponible/); + assert.match(t, /no_subtitles/); + }); + it('get_transcript_text horodaté + srt + troncature', async () => { + const txt = await call(proc, 23, 'get_transcript_text', { provider: 'youtube', videoId: 'v1', lang: 'fr' }); + assert.match(txt, /\[00:00:01\] Bonjour/); + assert.match(txt, /\[00:01:05\]/); + const srt = await call(proc, 24, 'get_transcript_text', { provider: 'youtube', videoId: 'v1', format: 'srt' }); + assert.match(srt, /-->/); + const cut = await call(proc, 25, 'get_transcript_text', { provider: 'youtube', videoId: 'v1', maxChars: 500 }); + assert.match(cut, /tronqu/); + }); + it('outil inconnu -> erreur JSON-RPC -32602', async () => { + const m = await rpc(proc, 99, 'tools/call', { name: 'nope', arguments: {} }); + assert.ok(m.error); + assert.equal(m.error.code, -32602); + }); +}); diff --git a/package.json b/package.json index 21802a3..ad7d0aa 100644 --- a/package.json +++ b/package.json @@ -9,6 +9,10 @@ "preview": "ng serve --configuration=production", "api": "node --env-file=.env.local ./server/index.mjs", "api:watch": "node --watch --env-file=.env.local ./server/index.mjs", + "mcp": "node ./mcp/server.mjs", + "mcp:dev": "node --env-file=.env.local ./mcp/server.mjs", + "test:mcp": "node --test ./mcp/server.test.mjs ./mcp/tools.test.mjs", + "test:api": "node --test ./server/tests/api_coverage.test.mjs", "test:playlists": "node ./server/tests/playlist_visibility.test.mjs", "test:preferences": "node ./server/tests/preferences.test.mjs", "test:search-e2e": "node ./server/tests/search.e2e.test.mjs", diff --git a/server/db.mjs b/server/db.mjs index 191d0f5..26f6a04 100644 --- a/server/db.mjs +++ b/server/db.mjs @@ -771,14 +771,20 @@ export function listLikedVideos({ userId, limit = 100, q }) { } // -------------------- Playlists -------------------- +// Metrics are observability-only: they must never break the core operation +// (e.g. a playlist DELETE must stay 204 even if the metrics insert fails). export function recordPlaylistMetric({ userId, playlistId, action, meta }) { - const id = cryptoRandomId(); - const created_at = nowIso(); - const meta_json = meta ? JSON.stringify(meta) : null; - db.prepare(`INSERT INTO playlist_metrics (id, user_id, playlist_id, action, meta_json, created_at) - VALUES (?, ?, ?, ?, ?, ?)`) - .run(id, userId, playlistId, action, meta_json, created_at); - return { id, created_at }; + try { + const id = cryptoRandomId(); + const created_at = nowIso(); + const meta_json = meta ? JSON.stringify(meta) : null; + db.prepare(`INSERT INTO playlist_metrics (id, user_id, playlist_id, action, meta_json, created_at) + VALUES (?, ?, ?, ?, ?, ?)`) + .run(id, userId, playlistId, action, meta_json, created_at); + return { id, created_at }; + } catch { + return null; + } } export function createPlaylist({ userId, title, description, thumbnail, isPrivate = true }) { @@ -864,8 +870,13 @@ export function deletePlaylist({ userId, id }) { const cur = db.prepare(`SELECT * FROM playlists WHERE id = ?`).get(id); if (!cur) return { removed: false }; if (cur.user_id !== userId) return 'forbidden'; + // Record BEFORE the row disappears: playlist_metrics.playlist_id references + // playlists(id), so inserting after the DELETE violates the FK constraint. + try { recordPlaylistMetric({ userId, playlistId: id, action: 'delete' }); } catch {} + // Explicit child cleanup (works even on old DBs whose FK lacks ON DELETE CASCADE). + try { db.prepare(`DELETE FROM playlist_items WHERE playlist_id = ?`).run(id); } catch {} + try { db.prepare(`DELETE FROM playlist_metrics WHERE playlist_id = ?`).run(id); } catch {} const info = db.prepare(`DELETE FROM playlists WHERE id = ?`).run(id); - recordPlaylistMetric({ userId, playlistId: id, action: 'delete' }); return { removed: (info.changes || 0) > 0 }; } diff --git a/server/index.mjs b/server/index.mjs index 3738ecd..c1874ea 100644 --- a/server/index.mjs +++ b/server/index.mjs @@ -3387,6 +3387,10 @@ async function transcriptViaYtDlp(url, langs, netOpts = {}) { noWarnings: true, noCheckCertificates: true, noPlaylist: true, + // Espace les requêtes sous-titres : les rafales déclenchent des 429 + // YouTube que les retries immédiats ne font qu'aggraver. + sleepRequests: 2, + sleepSubtitles: 5, ...netOpts, output: path.join(dir, '%(id)s'), }); @@ -3515,11 +3519,11 @@ r.get('/transcript/:provider/:videoId', transcriptLimiter, async (req, res) => { const allowedSet = new Set(allowed.map(transcriptPrimary)); const keepAllowed = (cands) => (cands || []).filter(c => allowedSet.has(transcriptPrimary(c.lang))); const baseCands = keepAllowed(normalized === 'youtube' - ? firstPerLanguage(orderedTracks(meta, lang), 10) + ? firstPerLanguage(orderedTracks(meta, lang), 6) : orderedTracks(meta, lang).slice(0, 5)); // Server-side auto-translation only towards an allowed target language. const candidates = normalized === 'youtube' - ? baseCands.concat(keepAllowed(translatedFallbacks(baseCands, lang))).slice(0, 12) + ? baseCands.concat(keepAllowed(translatedFallbacks(baseCands, lang))).slice(0, 8) : baseCands; for (const cand of candidates) { let attempt = 0; @@ -3534,11 +3538,12 @@ r.get('/transcript/:provider/:videoId', transcriptLimiter, async (req, res) => { } catch (e) { const msg = String(e?.message || e); console.warn('[transcript] track fetch failed:', cand.lang, msg); - // One retry after a short pause on transient 429s. - if (msg.includes(':429') && attempt === 0) { + // Backoff exponentiel + jitter sur 429 (buckets YouTube à la minute) : + // 1 retry @1.5s ne suffit pas quand l'IP est limitée. + if (msg.includes(':429') && attempt < 2) { sawRateLimit = true; attempt += 1; - await sleep(1500); + await sleep([2000, 8000][attempt - 1] + Math.floor(Math.random() * 1000)); continue; } if (msg.includes(':429')) sawRateLimit = true; @@ -3546,6 +3551,8 @@ r.get('/transcript/:provider/:videoId', transcriptLimiter, async (req, res) => { } } if (lines.length) break; + // Refroidissement entre candidats quand YouTube limite (évite d'aggraver le 429). + if (sawRateLimit) await sleep(700); } if (!lines || lines.length === 0) { // Direct timedtext fetches failed (429 / Sorry pages / impersonation): diff --git a/server/tests/api_coverage.test.mjs b/server/tests/api_coverage.test.mjs new file mode 100644 index 0000000..d410d1b --- /dev/null +++ b/server/tests/api_coverage.test.mjs @@ -0,0 +1,194 @@ +// Couverture HTTP des routes API non couvertes par les autres suites : +// health, search/suggest (400 + fallback), transcript (400), trending (400 + shape), +// ai/status, openapi, auth (register/login/me), preferences, playlists CRUD, +// likes, history search/watch, subscriptions, telemetry, download jobs. +// Offline-safe : aucune assertion sur le contenu provider externe, uniquement +// les formes stables (status, clés JSON). yt-dlp n'est jamais invoqué avec succès. +// Run: npm run test:api + +import { describe, it, before, after } from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import path from 'node:path'; +import os from 'node:os'; +import net from 'node:net'; +import { spawn } from 'node:child_process'; + +const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'newtube-api-cov-')); +const dbPath = path.join(tmpDir, 'cov.db'); +const PORT = await new Promise((resolve) => { + const s = net.createServer(); + s.listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); }); +}); +const base = `http://127.0.0.1:${PORT}`; +const server = spawn(process.execPath, ['./server/index.mjs'], { + cwd: path.resolve(import.meta.dirname, '..', '..'), + env: { ...process.env, PORT: String(PORT), NEWTUBE_DB_FILE: dbPath, JWT_SECRET: 'cov-test-secret', NODE_ENV: 'test' }, + stdio: ['ignore', 'pipe', 'pipe'], +}); +let logs = ''; +server.stdout.on('data', (d) => { logs += d; }); +server.stderr.on('data', (d) => { logs += d; }); + +async function waitUp(timeout = 25000) { + const t0 = Date.now(); + while (Date.now() - t0 < timeout) { + try { const r = await fetch(`${base}/api/health`); if (r.status < 500) return true; } catch {} + await new Promise((r) => setTimeout(r, 300)); + } + return false; +} +const up = await waitUp(); +assert.ok(up, `API ne démarre pas:\n${logs.slice(-3000)}`); + +const J = async (url, opts = {}) => { + const r = await fetch(url, opts); + const body = await r.json().catch(() => ({})); + return { status: r.status, body }; +}; +const auth = (t) => ({ Authorization: `Bearer ${t}` }); +const uniq = (p) => `${p}_${Date.now()}_${Math.floor(Math.random() * 1e6)}`; + +let token; +const user = uniq('covuser'); + +before(async () => { + const reg = await J(`${base}/api/auth/register`, { + method: 'POST', headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ username: user, password: 'Passw0rd!' }), + }); + assert.ok([201, 409].includes(reg.status), `register ${reg.status}`); + const login = await J(`${base}/api/auth/login`, { + method: 'POST', headers: { 'content-type': 'application/json' }, + body: JSON.stringify({ username: user, password: 'Passw0rd!' }), + }); + assert.equal(login.status, 200); + assert.ok(login.body.accessToken); + token = login.body.accessToken; +}); +after(() => { try { server.kill(); } catch {} }); + +describe('lecture publique / santé', () => { + it('GET /api/health -> ok', async () => { + const r = await J(`${base}/api/health`); + assert.equal(r.status, 200); assert.equal(r.body.status, 'ok'); + }); + it('GET /healthz expose youtube.mode sans secret', async () => { + const r = await J(`${base}/healthz`); + assert.equal(r.status, 200); + assert.equal(r.body.status, 'ok'); + assert.ok(r.body.youtube?.mode); + assert.ok(!JSON.stringify(r.body).includes('GEMINI')); + }); + it('GET /api/search?q=x -> 400', async () => { + const r = await J(`${base}/api/search?q=x`); + assert.equal(r.status, 400); + }); + it('GET /api/search providers inconnus -> fallback registre', async () => { + const r = await J(`${base}/api/search?q=test&providers=xx,yy&pageSize=2`); + assert.equal(r.status, 200); + assert.ok(r.body.providers.length >= 6); + assert.ok(r.body.groups && typeof r.body.groups === 'object'); + }); + it('GET /api/search/suggest?q=x -> 400', async () => { + const r = await J(`${base}/api/search/suggest?q=x`); + assert.equal(r.status, 400); + }); + it('GET /api/transcript provider inconnu -> 400 available:false', async () => { + const r = await J(`${base}/api/transcript/bogus/vid`); + assert.equal(r.status, 400); + assert.equal(r.body.available, false); + }); + it('GET /api/details provider inconnu -> 500 details_failed (sans yt-dlp)', async () => { + const r = await J(`${base}/api/details/bogus/vid`); + assert.equal(r.status, 500); + assert.equal(r.body.error, 'details_failed'); + }); + it('GET /api/trending provider non-yt -> 400', async () => { + const r = await J(`${base}/api/trending?provider=dm`); + assert.equal(r.status, 400); + }); + it('GET /api/trending?provider=yt -> shape {provider,items[]}', async () => { + const r = await J(`${base}/api/trending?provider=yt&limit=2`); + assert.equal(r.status, 200); + assert.equal(r.body.provider, 'yt'); + assert.ok(Array.isArray(r.body.items)); + }); + it('GET /api/ai/status -> {ready:boolean}', async () => { + const r = await J(`${base}/api/ai/status`); + assert.equal(r.status, 200); + assert.equal(typeof r.body.ready, 'boolean'); + }); + it('GET /api/openapi.json documente playlists', async () => { + const r = await J(`${base}/api/openapi.json`); + assert.equal(r.status, 200); + assert.ok(r.body.paths?.['/playlists']); + }); +}); + +describe('auth + espace utilisateur', () => { + it('register/login 400 sans password, 401 mauvais password', async () => { + const r1 = await J(`${base}/api/auth/login`, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ username: user }) }); + assert.equal(r1.status, 400); + const r2 = await J(`${base}/api/auth/login`, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ username: user, password: 'mauvais' }) }); + assert.equal(r2.status, 401); + }); + it('protégées sans token -> 401', async () => { + for (const u of ['/api/user/me', '/api/playlists', '/api/user/likes', '/api/subscriptions', '/api/download/jobs']) { + const r = await J(`${base}${u}`); + assert.equal(r.status, 401, u); + } + }); + it('GET /api/user/me + preferences round-trip', async () => { + const me = await J(`${base}/api/user/me`, { headers: auth(token) }); + assert.equal(me.status, 200); assert.ok(me.body.id); + const p = await J(`${base}/api/user/preferences`, { headers: auth(token) }); + assert.equal(p.status, 200); + const w = await J(`${base}/api/user/preferences`, { method: 'PATCH', headers: { ...auth(token), 'content-type': 'application/json' }, body: JSON.stringify({ defaultProviders: ['yt'] }) }); + assert.equal(w.status, 200); + }); + it('playlists CRUD complet', async () => { + const h = { ...auth(token), 'content-type': 'application/json' }; + const c = await J(`${base}/api/playlists`, { method: 'POST', headers: h, body: JSON.stringify({ title: 'Cov' }) }); + assert.equal(c.status, 201); const id = c.body.id; assert.ok(id); + const bad = await J(`${base}/api/playlists`, { method: 'POST', headers: h, body: JSON.stringify({}) }); + assert.equal(bad.status, 400); + const g = await J(`${base}/api/playlists/${id}`, { headers: auth(token) }); + assert.equal(g.status, 200); + const add = await J(`${base}/api/playlists/${id}/videos`, { method: 'POST', headers: h, body: JSON.stringify({ provider: 'youtube', videoId: 'v1', title: 'T', thumbnail: 'http://x/t.jpg' }) }); + assert.equal(add.status, 201); + const del = await J(`${base}/api/playlists/${id}/videos/v1?provider=youtube`, { method: 'DELETE', headers: auth(token) }); + assert.equal(del.status, 200); + const u = await J(`${base}/api/playlists/${id}`, { method: 'PUT', headers: h, body: JSON.stringify({ title: 'Cov2' }) }); + assert.equal(u.status, 200); assert.equal(u.body.title, 'Cov2'); + const rm = await fetch(`${base}/api/playlists/${id}`, { method: 'DELETE', headers: auth(token) }); + assert.equal(rm.status, 204); + }); + it('likes + history search/watch', async () => { + const h = { ...auth(token), 'content-type': 'application/json' }; + // title + thumbnail fournis -> pas d'appel yt-dlp d'enrichissement (offline-safe) + const like = await J(`${base}/api/user/likes`, { method: 'POST', headers: h, body: JSON.stringify({ provider: 'youtube', videoId: 'v1', title: 'T', thumbnail: 'http://x/t.jpg' }) }); + assert.ok([200, 201].includes(like.status)); + const st = await J(`${base}/api/user/likes/status?provider=youtube&videoId=v1`, { headers: auth(token) }); + assert.equal(st.status, 200); + const hs = await J(`${base}/api/user/history/search`, { method: 'POST', headers: h, body: JSON.stringify({ query: 'cov' }) }); + assert.ok([200, 201].includes(hs.status)); + const ls = await J(`${base}/api/user/history/search`, { headers: auth(token) }); + assert.equal(ls.status, 200); + const hw = await J(`${base}/api/user/history/watch`, { method: 'POST', headers: h, body: JSON.stringify({ provider: 'youtube', videoId: 'v1' }) }); + assert.ok([200, 201].includes(hw.status)); + const lw = await J(`${base}/api/user/history/watch`, { headers: auth(token) }); + assert.equal(lw.status, 200); + }); + it('subscriptions + telemetry + download jobs', async () => { + const h = { ...auth(token), 'content-type': 'application/json' }; + const sub = await J(`${base}/api/subscriptions`, { method: 'POST', headers: h, body: JSON.stringify({ provider: 'yt', externalId: 'ch1', title: 'C' }) }); + assert.ok([200, 201].includes(sub.status)); + const ls = await J(`${base}/api/subscriptions`, { headers: auth(token) }); + assert.equal(ls.status, 200); + const tel = await J(`${base}/api/telemetry/events`, { method: 'POST', headers: h, body: JSON.stringify({ event: 'search_submit' }) }); + assert.ok([200, 201].includes(tel.status)); + const dj = await J(`${base}/api/download/jobs`, { headers: auth(token) }); + assert.equal(dj.status, 200); + }); +});