feat(mcp): serveur MCP stdio 7 outils + guide API/MCP + couverture tests
CI / build-and-test (push) Successful in 14m3s

- mcp/server.mjs : search, suggest, details, trending, transcript,
  transcript_text, health_check (client fin vers API REST)
- docs/API_MCP_GUIDE.md : reference complete REST + MCP + transcript
- tests : server/tests/api_coverage.test.mjs (17 cas HTTP),
  mcp/tools.test.mjs (7 outils sur stub), scripts test:mcp/test:api
- fix(db): DELETE playlist 500 FK -> metrique avant delete + cleanup
  enfants + recordPlaylistMetric tolerant (observabilite)
This commit is contained in:
2026-09-27 18:39:26 -04:00
parent d0d1c07535
commit 1812b86dd4
8 changed files with 1025 additions and 13 deletions
+303
View File
@@ -0,0 +1,303 @@
# NewTube — Guide complet API REST + MCP
> Référence développeur et LLM : tous les endpoints REST, le nouveau serveur MCP (dont transcript), exemples `curl` et JSON-RPC, auth, rate-limits, erreurs, variables d'environnement, tests.
>
> Fichiers sources : `server/index.mjs` (~3800 lignes), `server/transcript.mjs`, `server/providers/registry.mjs`, `mcp/server.mjs`, `api/README.md`.
---
## Table des matières
1. [Vue d'ensemble](#1-vue-densemble)
2. [Démarrage rapide](#2-démarrage-rapide)
3. [Conventions API](#3-conventions-api)
4. [Authentification](#4-authentification)
5. [Référence REST — lecture publique](#5-référence-rest--lecture-publique)
6. [Référence REST — transcript (détaillé)](#6-référence-rest--transcript-détaillé)
7. [Référence REST — espace utilisateur (JWT)](#7-référence-rest--espace-utilisateur-jwt)
8. [Référence REST — OAuth / import](#8-référence-rest--oauth--import)
9. [Référence REST — IA / proxies / santé](#9-référence-rest--ia--proxies--santé)
10. [Erreurs & rate-limits](#10-erreurs--rate-limits)
11. [Serveur MCP — installation](#11-serveur-mcp--installation)
12. [Serveur MCP — outils](#12-serveur-mcp--outils)
13. [MCP — transcript pas à pas](#13-mcp--transcript-pas-à-pas)
14. [Variables d'environnement](#14-variables-denvironnement)
15. [Tests](#15-tests)
16. [Dépannage](#16-dépannage)
---
## 1. Vue d'ensemble
```
┌────────────┐ stdio JSON-RPC ┌──────────────┐ HTTP fetch ┌─────────────┐
│ Claude / │ ──────────────────► │ mcp/server │ ───────────────► │ Express API │
│ VS Code / │ tools/call │ .mjs (7 │ NEWTUBE_API_URL │ :4000 /api │
│ opencode │ ◄────────────────── │ outils) │ ◄─────────────── │ + SQLite │
└────────────┘ tools/list └──────────────┘ └─────────────┘
▲ ▲
sans dépendance front Angular (dist/)
Node 20+ fetch natif yt-dlp + InnerTube
```
- **API REST** : Express, SQLite (`better-sqlite3`), JWT + cookies `sid`/`refreshToken`, montée sous `/api` **et** `/proxy/api` (même router `r` — le front prod appelle `/proxy/api`).
- **MCP** : nouveau `mcp/server.mjs`, **zéro-dépendance**, protocole `2024-11-05`, transport stdio. C'est un client HTTP fin vers l'API : toute la logique métier (InnerTube, yt-dlp, cache, quotas) reste côté API.
IDs providers :
| Court | Long | Transcript |
|---|---|---|
| `yt` | `youtube` | ✅ (InnerTube + yt-dlp) |
| `dm` | `dailymotion` | ✅ |
| `pt` | `peertube` | ✅ (exige `instance`) |
| `tw` | `twitch` | ⚠️ pistes quasi inexistantes → `provider_unsupported` |
| `od` | `odysee` | ⚠️ idem |
| `ru` | `rumble` | ⚠️ idem |
## 2. Démarrage rapide
```bash
cp .env.example .env.local # renseigner JWT_SECRET, clés…
npm install
npm run api # API http://localhost:4000 (+ front dist/)
# autre terminal :
npm run mcp # MCP stdio (nécessite l'API)
# ou avec .env.local :
npm run mcp:dev
```
Santé :
```bash
curl http://localhost:4000/healthz | jq
curl http://localhost:4000/api/search?q=lofi&providers=yt&limit=5 | jq
```
## 3. Conventions API
- Base : `http://localhost:4000/api` (dev) ou `/api` / `/proxy/api` (prod, même chose).
- JSON partout (même les 429 : `{ error, retryAfterSec }` ou `{ available:false, error }`).
- Auth : `Authorization: Bearer <accessToken>` **ou** cookies `sid`+`refreshToken` (routes `authMiddlewareCookieAware` : downloads, likes, subscriptions, playlists).
- Pagination : `page` (défaut 1), `pageSize`/`limit` (max 50).
- OpenAPI partiel : `GET /api/openapi.json` (playlists uniquement, à étendre).
## 4. Authentification
| Méthode | Route | Body | Réponse |
|---|---|---|---|
| POST | `/api/auth/register` | `{ username, email?, password }` | `{ user, accessToken }` + cookies |
| POST | `/api/auth/login` | `{ username|email, password, rememberMe? }` | `{ user, accessToken }` + cookies |
| POST | `/api/auth/refresh` | cookies | `{ accessToken }` |
| POST | `/api/auth/logout` | — | `{ ok:true }` + clear cookies |
| GET | `/api/user/me` | JWT | `{ id, username, email, … }` |
| GET/DELETE | `/api/auth/sessions`, `/api/auth/sessions/:id` | JWT | sessions actives |
```bash
TOKEN=$(curl -s -X POST http://localhost:4000/api/auth/login \
-H 'Content-Type: application/json' \
-d '{"username":"demo","password":"demo123"}' | jq -r .accessToken)
curl -H "Authorization: Bearer $TOKEN" http://localhost:4000/api/user/me
```
Durées : `ACCESS_TTL_MIN` (15), `REFRESH_TTL_DAYS` (2), `REMEMBER_TTL_DAYS` (défaut 30). `JWT_SECRET` **obligatoire en prod**.
## 5. Référence REST — lecture publique
### 5.1 `GET /api/search` — recherche unifiée
```
?q=lofi&providers=yt,dm&page=1&pageSize=24&sort=relevance|date|views
```
Réponse `{ q, providers, groups:{ yt:[…], dm:[…] }, errors:{}, page, pageSize, sort }`. Item : `{ id,title,thumbnail,uploaderName,url,type,duration,isShort }`. Fan-out parallèle, `Promise.allSettled` (un provider KO → `groups[x]=[]` + `errors[x]`). `400` si `q<2`.
### 5.2 `GET /api/search/suggest` — typeahead
```
?q=lof&providers=yt,dm&limit=10
```
→ `{ q, groups:{ yt:[…], dm:[…], web:[…] } }`. Cache 5 min, rate-limit 60/min. Bonus sans clé : `web` (Bing/Google) + vrais titres Odysee (Lighthouse) si `od` demandé.
### 5.3 `GET /api/details/:provider/:videoId` — métadonnées
Providers longs (`youtube|dailymotion|twitch|peertube|odysee|rumble`). Query : `instance` (PeerTube obligatoire), `slug` (Odysee), `sourceUrl` (prioritaire), `related=0` (désactive connexes).
Réponse `{ videoId,title,thumbnail,uploaderName,uploaderAvatar,views,duration,uploadedDate,description,url,related[] }` (`related` = watch-next InnerTube, YouTube uniquement, best-effort).
```bash
curl "http://localhost:4000/api/details/youtube/dQw4w9WgXcQ" | jq '{title,uploaderName,views,related: (.related|length)}'
```
### 5.4 `GET /api/trending?provider=yt&limit=24`
Tendances YT sans clé (scrape). Phase 1 : `yt` uniquement (`400` sinon), fallback `{ items:[] }` jamais 500.
### 5.5 Rumble dédié (`/api/rumble/…`)
`GET /browse`, `GET /search?q=…`, `GET /video/:videoId`, `GET /video/:videoId/preplay`. Rate-limit 10/min.
## 6. Référence REST — transcript détaillé
### 6.1 `GET /api/transcript/:provider/:videoId`
```
GET /api/transcript/youtube/dQw4w9WgXcQ?lang=fr
GET /api/transcript/peertube/abc123?lang=fr&instance=peertube.example.com
GET /api/transcript/odysee/mavideo?lang=en&slug=@chaine:mavideo
```
Query : `lang` (défaut `fr`, max 12 chars, filtrée par langues autorisées), `instance`, `slug`, `sourceUrl`, `langs` (override explicite `?langs=fr,en`, sinon préférences utilisateur si JWT, sinon `fr,en`).
**Succès 200 :**
```json
{ "lang": "fr", "available": true, "languages": ["fr","en"],
"lines": [{ "t": 0.0, "dur": 2.5, "text": "Bonjour à tous" }] }
```
**Absence définitive 200 :**
```json
{ "lang": null, "available": false, "languages": ["en"], "lines": [], "reason": "no_subtitles" }
{ "lang": null, "available": false, "languages": [], "lines": [], "reason": "provider_unsupported" }
```
**Transitoire 502 (à réessayer) :**
```json
{ "available": false, "languages": ["fr","en"], "lines": [],
"error": "transcript_temporarily_unavailable", "retryable": true, "rateLimited": true }
```
**Autres :** `400 { available:false, error:"invalid_provider_or_video" }`, `429 { available:false, error:"rate_limited" }` (10/min).
Pipeline YouTube : découverte InnerTube (`YT_TRANSCRIPT_SOURCE=innertube-first|ytdlp-only|innertube-only`, 0 piste → `no_subtitles` sans yt-dlp) → candidats ordonnés (`orderedTracks` + `firstPerLanguage`, max 6-8) → fetch timedtext (retry 429 : 2 s puis 8 s) → traduction serveur `&tlang=` en dernier recours → fallback `yt-dlp --write-sub --sub-langs` (90 s max). Parsers : `json3` → `vtt` → XML (`srv1/2/3`, `ttml`, `<text>`), dédup rolling-window auto-captions, `TRANSCRIPT_MAX_LINES` (5000). Cache LRU 200 entrées / 24 h (`TRANSCRIPT_CACHE_TTL`). Anti-ban datacenter : `YT_COOKIES_FILE`, `YT_PO_TOKEN`, `YT_EGRESS_PROXY`.
Exemples :
```bash
# JSON brut
curl "http://localhost:4000/api/transcript/youtube/VIDEO_ID?lang=fr" | jq '{lang,available,n:(.lines|length)}'
# Texte horodaté pour LLM (jq)
curl -s "http://localhost:4000/api/transcript/youtube/VIDEO_ID?lang=fr" \
| jq -r '.lines[] | "[\(.t|floor)] \(.text)"' | head -50
```
### 6.2 Historique transcripts (JWT)
`POST /api/user/history/transcripts`, `GET /api/user/history/transcripts`, `GET /api/user/history/transcripts/:provider/:videoId`, `DELETE …/:id`, `DELETE …`.
## 7. Référence REST — espace utilisateur (JWT)
- **Préférences** : `GET/PATCH /api/user/preferences` (`defaultProviders`, `downloadLanguages` — pilote aussi le transcript).
- **Playlists** : `GET/POST /api/playlists`, `GET/PUT/DELETE /api/playlists/:id`, `POST /api/playlists/:id/videos`, `DELETE /api/playlists/:id/videos/:videoId?provider=yt`, `PUT …/reorder`, `GET /api/playlists/public`, `GET /api/playlists/:id/view`.
- **Likes** : `GET/POST/DELETE /api/user/likes`, `GET /api/user/likes/status?provider=&videoId=`.
- **Historiques** : `/api/user/history/search` (GET/POST/batch/DELETE), `/api/user/history/watch` (GET/POST/PATCH/DELETE), takeout `POST /api/user/history/takeout`.
- **Abonnements** : `GET/POST /api/subscriptions`, `POST /api/subscriptions/batch`, `DELETE /api/subscriptions/:id`, groupes `GET/POST/PATCH/DELETE /api/subscription-groups`, `PUT …/members`, `PUT /api/subscriptions/:id/groups`, resolve `POST /api/channels/resolve`, `GET /api/channels/:provider/:externalId[/content]`.
- **Téléchargements** (JWT cookie-aware) : `GET /api/download/:provider/:videoId/formats` (cache 10 min), `POST /api/download/:provider/:videoId`, `GET /api/download/jobs`, `GET /api/download/jobs/:id[/file]`, `POST /api/download/jobs/:id/retry`, `DELETE /api/download/jobs/:id`. Quotas : `DOWNLOAD_MAX_CONCURRENT=2`, `DOWNLOAD_STORAGE_QUOTA_BYTES=5GiB`, `DOWNLOAD_QUOTA_WINDOW_MS=30j`, `DOWNLOAD_PROVIDERS`.
- **Télémétrie** : `POST /api/telemetry/events` (whitelist : `search_submit`, `provider_picker_open`, `provider_apply`, `at_autocomplete_use`, `quick_menu_open`, `suggest_shown`, `suggest_used`), `GET /api/telemetry/events|summary`.
## 8. Référence REST — OAuth / import
Statut `GET /api/oauth/status`, URL `GET /api/oauth/:provider/url` (JWT), callback `GET /api/oauth/:provider/callback`, `GET /api/oauth/connections`, `DELETE /api/oauth/:provider`, preview `GET /api/oauth/:provider/preview`, import `POST /api/oauth/:provider/import`, Google : `GET /api/oauth/google/yt-channels|diag|watchlater|yt-history`, `PUT /api/oauth/google/yt-channel`, `POST /api/oauth/google/watchlater`, takeout `POST /api/user/history/takeout`. Env : `GOOGLE_CLIENT_ID/SECRET/REDIRECT_URI`, `TWITCH_CLIENT_ID/SECRET`, `OAUTH_APP_BASE_URL`.
## 9. Référence REST — IA / proxies / santé
- `GET /api/ai/status` → `{ ready }` ; `POST /api/ai/summarize` `{ comments: string[] }` → `{ summary }` (clé `GEMINI_API_KEY` côté serveur, 10/min).
- `GET /api/twitch-token` (secret jamais exposé), `GET /assets/config.js` (whitelist front uniquement).
- Proxies CORS : `ALL /api/dm/*`, `/api/odysee/*`, `/api/twitch-api/*`, `/api/twitch-auth/*`.
- Santé : `GET /api/health` (`{status:ok}`), `GET /healthz|/api/healthz` (mode YT, yt-dlp `binOk`+version, cookies/PO-token/proxy, cache mémoire+SQLite, métriques quota jour, clés `...XXXX` bannies — aucun secret).
## 10. Erreurs & rate-limits
| Cas | Code | Corps |
|---|---|---|
| Requête courte | 400 | `{ error:"q is required…" }` |
| Provider/vidéo invalide | 400 | `{ available:false, error:"invalid_provider_or_video" }` |
| Non authentifié | 401 | `{ error:"Unauthorized" }` |
| Provider download désactivé | 403 | `{ error:"download_disabled_for_provider" }` |
| Trop de requêtes | 429 | `{ error:"rate_limited", retryAfterSec }` ou `{ available:false, error:"rate_limited" }` |
| YouTube 429 épuisé | 502 | `{ available:false, error:"transcript_temporarily_unavailable", retryable:true }` |
| Erreur interne | 500 | `{ error:"search_failed"|"details_failed", details }` |
Seaux : login 5/min, suggest 60/min, transcript 10/min, downloads lecture 120 / écriture 30 / formats 30, AI 10/min, Rumble 10/min.
## 11. Serveur MCP — installation
Prérequis : Node ≥ 20, API démarrée. Aucune dépendance à installer (fetch natif).
```bash
npm run mcp # stdio, NEWTUBE_API_URL=http://localhost:4000/api
npm run mcp:dev # charge .env.local
NEWTUBE_API_URL=https://mon-serveur/api NEWTUBE_TOKEN=<jwt> npm run mcp
```
**Claude Desktop** (`%APPDATA%\Claude\claude_desktop_config.json`) :
```json
{ "mcpServers": { "newtube": {
"command": "node",
"args": ["C:/dev/git/web/NewTube/mcp/server.mjs"],
"env": { "NEWTUBE_API_URL": "http://localhost:4000/api" }
} } }
```
**VS Code** (`.vscode/mcp.json`) / **opencode** (`opencode.json`) : même `command`/`args`/`env`. Redémarrer le client, vérifier `tools/list` → 7 outils.
## 12. Serveur MCP — outils
| Outil | Args | Appel REST sous-jacent |
|---|---|---|
| `search_videos` | `q*`, `providers`, `page`, `pageSize` (≤50), `sort` | `GET /search` |
| `suggest_queries` | `q*`, `providers`, `limit` (≤20) | `GET /search/suggest` |
| `get_video_details` | `provider*`, `videoId*`, `instance`, `slug`, `sourceUrl`, `related=true` | `GET /details/:p/:id` |
| `get_trending` | `provider=yt`, `limit` (≤50) | `GET /trending` |
| `get_transcript` | `provider*`, `videoId*`, `lang=fr`, `instance`, `slug`, `sourceUrl` | `GET /transcript/:p/:id` (JSON brut) |
| `get_transcript_text` | + `format=text\|srt`, `withTimestamps=true`, `maxChars=12000` | idem + mise en forme LLM |
| `health_check` | — | `GET /healthz` |
Réponse MCP : `{ content:[{ type:"text", text:"<résumé>\n\n<extrait>\n\n--- JSON ---\n<tronqué 8ko>" }], isError? }`. Les erreurs API (connexion refusée, 429, 502 retryable) remontent en `isError:true` avec message actionnable (`npm run api`, réessayer…).
Exemple JSON-RPC (stdio, 1 ligne = 1 message) :
```json
{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}
{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}
{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"get_transcript_text","arguments":{"provider":"youtube","videoId":"dQw4w9WgXcQ","lang":"fr","maxChars":8000}}}
```
Test manuel :
```bash
printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' '{"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}' | node ./mcp/server.mjs
```
## 13. MCP — transcript pas à pas
1. `get_transcript(provider=youtube, videoId=…, lang=fr)` → diagnostic : `available`, `lang` effectif, `languages`, `lines.length`.
2. Si `available:false` + `reason=no_subtitles|provider_unsupported` → stop (définitif). Si `error=transcript_temporarily_unavailable, retryable:true` → attendre 30-60 s et réessayer (429 YouTube).
3. Sinon `get_transcript_text(format=text, withTimestamps=true, maxChars=12000)` → résumé/analyse/Q&A côté LLM. PeerTube : ne pas oublier `instance`. Odysee : `slug` si besoin.
4. Pour sous-titrage vidéo : `format=srt`.
Recette LLM type : `search_videos("quantique", providers=yt)` → `get_video_details` (durée/vues) → `get_transcript_text` → synthèse horodatée `[00:12:30] …`.
## 14. Variables d'environnement
| Var | Défaut | Rôle |
|---|---|---|
| `PORT` | 4000 | API |
| `JWT_SECRET` | `dev-secret…` | **à définir en prod** |
| `NEWTUBE_API_URL` (MCP) | `http://localhost:4000/api` | cible du MCP |
| `NEWTUBE_TOKEN` (MCP) | — | JWT pour futurs outils authentifiés |
| `MCP_TOOL_TIMEOUT_MS` | 60000 | timeout fetch MCP |
| `YT_SEARCH_MODE` | `innertube-first` | `innertube-first\|only\|scrape-first\|api-first\|…` |
| `YT_TRANSCRIPT_SOURCE` | `innertube-first` | `innertube-first\|ytdlp-only\|innertube-only` |
| `YT_COOKIES_FILE`, `YT_PO_TOKEN`, `YT_EGRESS_PROXY` | — | anti-ban 429 |
| `TRANSCRIPT_CACHE_TTL`, `TRANSCRIPT_RATE_LIMIT` | 24 h, 10/min | transcript |
| `SUGGEST_CACHE_TTL_MS`, `SUGGEST_RATE_LIMIT` | 5 min, 60/min | suggest |
| `DOWNLOAD_*` | 2 conc., 5 GiB/30 j | downloads |
| `YOUTUBE_API_KEY(S)`, `TWITCH_*`, `GEMINI_API_KEY`, `GOOGLE_*` | — | clés providers/IA/OAuth |
## 15. Tests
```bash
npm run test:transcript # parseurs json3/vtt + contrat transcript
npm run test:suggest # typeahead + contrat suggest
npm run test:search-e2e # serveur réel isolé
npm run test:mcp # protocole MCP (initialize/list/validation) — nouveau
npm run test:playlists && npm run test:downloads && npm run test:telemetry
```
## 16. Dépannage
- **MCP `API injoignable`** → `npm run api` d'abord, vérifier `NEWTUBE_API_URL` (sans `/` final géré, préfixe `/api` inclus).
- **Transcript 502 retryable** → 429 YouTube : attendre, réduire la fréquence, configurer `YT_COOKIES_FILE` + `YT_EGRESS_PROXY`, `YT_TRANSCRIPT_SOURCE=ytdlp-only` en test.
- **`provider_unsupported` (tw/od/ru)** → normal, pas de pistes yt-dlp.
- **`peertube_instance_required`** → passer `instance` (query REST ou arg MCP).
- **429 `rate_limited`** → respecter `Retry-After`/`retryAfterSec`.
- **stdout du MCP pollué** → le serveur log sur **stderr** uniquement ; ne jamais `console.log` sur stdout (canal JSON-RPC).
+330
View File
@@ -0,0 +1,330 @@
#!/usr/bin/env node
/**
* NewTube MCP Server (stdio, zero-dependency).
*
* Expose l'API REST NewTube comme outils MCP :
* - search_videos, suggest_queries, get_video_details, get_trending
* - get_transcript, get_transcript_text (JSON + texteLLM-friendly)
* - health_check
*
* Protocole : JSON-RPC 2.0 sur stdio (1 objet JSON par ligne), compatible
* MCP 2024-11-05 : initialize / notifications/initialized / tools/list /
* tools/call / ping.
*
* Prérequis : API NewTube démarrée (`npm run api`, défaut http://localhost:4000/api).
*
* Env :
* NEWTUBE_API_URL (défaut http://localhost:4000/api)
* NEWTUBE_TOKEN (JWT optionnel -> Authorization: Bearer, requis pour /download, /user/*)
* MCP_TOOL_TIMEOUT_MS (défaut 60000, transcript = appels yt-dlp longs)
*
* Config clients :
* Claude Desktop / VS Code / opencode :
* { "mcpServers": { "newtube": { "command": "node", "args": ["C:/dev/git/web/NewTube/mcp/server.mjs"],
* "env": { "NEWTUBE_API_URL": "http://localhost:4000/api" } } } }
*/
const API_BASE = (process.env.NEWTUBE_API_URL || 'http://localhost:4000/api').replace(/\/+$/, '');
const TOKEN = (process.env.NEWTUBE_TOKEN || '').trim();
const TIMEOUT_MS = Number(process.env.MCP_TOOL_TIMEOUT_MS || 60000);
const SERVER_VERSION = '1.0.0';
const PROVIDERS = ['yt', 'dm', 'tw', 'pt', 'od', 'ru'];
const TRANSCRIPT_PROVIDERS = ['youtube', 'yt', 'dailymotion', 'dm', 'peertube', 'pt', 'twitch', 'tw', 'odysee', 'od', 'rumble', 'ru'];
// ---------------------------------------------------------------- HTTP helper
async function apiFetch(path, { method = 'GET', body, auth = false } = {}) {
const url = `${API_BASE}${path.startsWith('/') ? path : `/${path}`}`;
const headers = { Accept: 'application/json', 'Content-Type': 'application/json' };
if (auth && TOKEN) headers.Authorization = `Bearer ${TOKEN}`;
const ctrl = new AbortController();
const timer = setTimeout(() => ctrl.abort(), TIMEOUT_MS);
try {
const res = await fetch(url, {
method,
headers,
signal: ctrl.signal,
...(body !== undefined ? { body: JSON.stringify(body) } : {}),
});
const text = await res.text();
let data;
try { data = text ? JSON.parse(text) : null; }
catch { data = { _raw: text.slice(0, 2000) }; }
if (!res.ok) {
const msg = data?.error || data?.message || `http_${res.status}`;
const details = data?.details ? ` — ${String(data.details).slice(0, 300)}` : '';
throw new Error(`${msg}${details} (HTTP ${res.status} ${method} ${path})`);
}
return data;
} catch (e) {
if (e?.name === 'AbortError') throw new Error(`timeout_after_${TIMEOUT_MS}ms (${method} ${path}) — API démarrée ? npm run api`);
if (e?.cause?.code === 'ECONNREFUSED') throw new Error(`API injoignable à ${API_BASE} — lancez 'npm run api' d'abord`);
throw e;
} finally {
clearTimeout(timer);
}
}
function fmtTs(sec) {
const s = Math.max(0, Number(sec || 0));
const h = Math.floor(s / 3600), m = Math.floor((s % 3600) / 60), ss = Math.floor(s % 60);
return `${String(h).padStart(2, '0')}:${String(m).padStart(2, '0')}:${String(ss).padStart(2, '0')}`;
}
function transcriptToText(data, { withTimestamps = true, maxChars = 12000 } = {}) {
const lines = Array.isArray(data?.lines) ? data.lines : [];
let out = lines.map((l) => (withTimestamps ? `[${fmtTs(l.t)}] ${l.text}` : String(l.text || ''))).join('\n');
if (out.length > maxChars) out = out.slice(0, maxChars) + '\n…[tronqué]';
return out;
}
function transcriptToSrt(data) {
const lines = Array.isArray(data?.lines) ? data.lines : [];
const fmt = (s) => {
const ms = Math.round(Number(s || 0) * 1000);
const h = String(Math.floor(ms / 3600000)).padStart(2, '0');
const m = String(Math.floor((ms % 3600000) / 60000)).padStart(2, '0');
const sec = String(Math.floor((ms % 60000) / 1000)).padStart(2, '0');
const milli = String(ms % 1000).padStart(3, '0');
return `${h}:${m}:${sec},${milli}`;
};
return lines.map((l, i) => `${i + 1}\n${fmt(l.t)} --> ${fmt(Number(l.t) + Number(l.dur || 2))}\n${l.text}\n`).join('\n');
}
// ---------------------------------------------------------------- Tools
const TOOLS = [
{
name: 'search_videos',
description: 'Recherche unifiée multi-providers (YouTube, Dailymotion, Twitch, PeerTube, Odysee, Rumble). q min 2 caractères.',
inputSchema: {
type: 'object',
properties: {
q: { type: 'string', description: 'Requête (min 2 caractères)' },
providers: { type: 'string', description: 'CSV parmi yt,dm,tw,pt,od,ru. Défaut: tous.' },
page: { type: 'integer', minimum: 1, default: 1 },
pageSize: { type: 'integer', minimum: 1, maximum: 50, default: 10 },
sort: { type: 'string', enum: ['relevance', 'date', 'views'], default: 'relevance' },
},
required: ['q'],
},
},
{
name: 'suggest_queries',
description: 'Typeahead de requêtes groupées par provider. Idéal avant search_videos.',
inputSchema: {
type: 'object',
properties: {
q: { type: 'string', description: 'Début de requête (min 2 caractères)' },
providers: { type: 'string', description: 'CSV providers, défaut tous' },
limit: { type: 'integer', minimum: 1, maximum: 20, default: 10 },
},
required: ['q'],
},
},
{
name: 'get_video_details',
description: "Métadonnées d'une vidéo + vidéos connexes YouTube (watch-next InnerTube).",
inputSchema: {
type: 'object',
properties: {
provider: { type: 'string', description: 'youtube|dailymotion|twitch|peertube|odysee|rumble (alias yt,dm,tw,pt,od,ru acceptés)' },
videoId: { type: 'string' },
instance: { type: 'string', description: 'Requis pour PeerTube (ex. peertube.example.com)' },
slug: { type: 'string', description: 'Slug Odysee si différent de videoId' },
sourceUrl: { type: 'string', description: "URL canonique (prioritaire sur la reconstruction)" },
related: { type: 'boolean', default: true, description: 'Inclure related[] YouTube (related=0 pour désactiver)' },
},
required: ['provider', 'videoId'],
},
},
{
name: 'get_trending',
description: 'Tendances YouTube sans clé API (scrape InnerTube, phase 1 : provider yt uniquement).',
inputSchema: {
type: 'object',
properties: { provider: { type: 'string', default: 'yt' }, limit: { type: 'integer', minimum: 1, maximum: 50, default: 10 } },
},
},
{
name: 'get_transcript',
description: "Transcript brut JSON d'une vidéo : { lang, available, languages, lines: [{t,dur,text}] }. Langues filtrées par préférences serveur (défaut fr,en). 200 available:false si absent, 502 retryable:true si YouTube rate-limite.",
inputSchema: {
type: 'object',
properties: {
provider: { type: 'string', description: TRANSCRIPT_PROVIDERS.join(' | ') },
videoId: { type: 'string' },
lang: { type: 'string', default: 'fr', description: "Langue préférée (ex. fr, en, fr-CA). Fallback auto + traduction serveur &tlang" },
instance: { type: 'string', description: 'Instance PeerTube si provider peertube' },
slug: { type: 'string', description: 'Slug Odysee' },
sourceUrl: { type: 'string', description: 'URL canonique (prioritaire)' },
},
required: ['provider', 'videoId'],
},
},
{
name: 'get_transcript_text',
description: "Transcript mis en forme pour LLM : texte plein horodaté (ou SRT), tronqué proprement. Recommandé pour résumer/analyser une vidéo.",
inputSchema: {
type: 'object',
properties: {
provider: { type: 'string' },
videoId: { type: 'string' },
lang: { type: 'string', default: 'fr' },
instance: { type: 'string' },
slug: { type: 'string' },
sourceUrl: { type: 'string' },
format: { type: 'string', enum: ['text', 'srt'], default: 'text' },
withTimestamps: { type: 'boolean', default: true },
maxChars: { type: 'integer', minimum: 500, maximum: 60000, default: 12000 },
},
required: ['provider', 'videoId'],
},
},
{
name: 'health_check',
description: "Santé API : mode YouTube (YT_SEARCH_MODE), binaire yt-dlp, cache, quota jour, clés. Équivalent GET /healthz.",
inputSchema: { type: 'object', properties: {} },
},
];
async function callTool(name, args = {}) {
switch (name) {
case 'search_videos': {
const q = String(args.q || '').trim();
if (q.length < 2) throw new Error('invalid_query: q min 2 caractères');
const qs = new URLSearchParams({
q,
...(args.providers ? { providers: String(args.providers) } : {}),
page: String(args.page ?? 1),
pageSize: String(args.pageSize ?? 10),
sort: String(args.sort ?? 'relevance'),
});
const data = await apiFetch(`/search?${qs}`);
const counts = Object.fromEntries(Object.entries(data.groups || {}).map(([k, v]) => [k, v.length]));
return { summary: `${data.groups ? Object.values(data.groups).flat().length : 0} résultats pour "${q}" ${JSON.stringify(counts)}`, data };
}
case 'suggest_queries': {
const q = String(args.q || '').trim();
if (q.length < 2) throw new Error('invalid_query: q min 2 caractères');
const qs = new URLSearchParams({ q, ...(args.providers ? { providers: String(args.providers) } : {}), limit: String(args.limit ?? 10) });
return { summary: `Suggestions pour "${q}"`, data: await apiFetch(`/search/suggest?${qs}`) };
}
case 'get_video_details': {
const provider = String(args.provider || '').toLowerCase();
const videoId = String(args.videoId || '');
if (!provider || !videoId) throw new Error('provider et videoId requis');
const qs = new URLSearchParams({
...(args.instance ? { instance: String(args.instance) } : {}),
...(args.slug ? { slug: String(args.slug) } : {}),
...(args.sourceUrl ? { sourceUrl: String(args.sourceUrl) } : {}),
...(args.related === false ? { related: '0' } : {}),
});
const qstr = qs.toString() ? `?${qs}` : '';
const data = await apiFetch(`/details/${encodeURIComponent(provider)}/${encodeURIComponent(videoId)}${qstr}`);
return { summary: `${data.title || videoId} — ${data.uploaderName || '?'} (${data.duration || 0}s, ${data.views || 0} vues)`, data };
}
case 'get_trending': {
const provider = String(args.provider || 'yt');
const limit = Math.min(50, Math.max(1, Number(args.limit ?? 10)));
const data = await apiFetch(`/trending?provider=${encodeURIComponent(provider)}&limit=${limit}`);
return { summary: `${(data.items || []).length} tendances (${provider})`, data };
}
case 'get_transcript':
case 'get_transcript_text': {
const provider = String(args.provider || '').toLowerCase();
const videoId = String(args.videoId || '');
if (!provider || !videoId) throw new Error('provider et videoId requis');
const qs = new URLSearchParams({
...(args.lang ? { lang: String(args.lang) } : {}),
...(args.instance ? { instance: String(args.instance) } : {}),
...(args.slug ? { slug: String(args.slug) } : {}),
...(args.sourceUrl ? { sourceUrl: String(args.sourceUrl) } : {}),
});
const qstr = qs.toString() ? `?${qs}` : '';
const data = await apiFetch(`/transcript/${encodeURIComponent(provider)}/${encodeURIComponent(videoId)}${qstr}`);
if (name === 'get_transcript') {
const n = (data.lines || []).length;
const summary = data.available
? `Transcript ${data.lang} : ${n} lignes, langues=${(data.languages || []).join(',')}`
: `Transcript indisponible (${data.reason || data.error || 'no_subtitles'}), langues=${(data.languages || []).join(',') || '—'}`;
return { summary, data };
}
// text variant
if (!data.available) {
return { summary: `Transcript indisponible (${data.reason || data.error})`, data, text: '' };
}
const format = args.format === 'srt' ? 'srt' : 'text';
const text = format === 'srt'
? transcriptToSrt(data).slice(0, Number(args.maxChars ?? 12000))
: transcriptToText(data, { withTimestamps: args.withTimestamps !== false, maxChars: Number(args.maxChars ?? 12000) });
return { summary: `Transcript ${data.lang} (${(data.lines || []).length} lignes) en ${format}`, data, text };
}
case 'health_check': {
const data = await apiFetch('/healthz');
return { summary: `API ${data.status} — YT mode=${data.youtube?.mode}, yt-dlp=${data.youtube?.ytdlp?.version || 'n/a'}`, data };
}
default:
throw new Error(`unknown_tool: ${name}`);
}
}
// ---------------------------------------------------------------- JSON-RPC stdio
const readline = await import('node:readline');
const rl = readline.createInterface({ input: process.stdin, crlfDelay: Infinity });
let initialized = false;
function send(obj) {
process.stdout.write(JSON.stringify(obj) + '\n');
}
const ok = (id, result) => send({ jsonrpc: '2.0', id, result });
const err = (id, code, message, data) => send({ jsonrpc: '2.0', id, error: { code, message, ...(data !== undefined ? { data } : {}) } });
function mcpText(summary, data, extraText) {
const payload = extraText ? `${summary}\n\n${extraText}\n\n--- JSON ---\n${JSON.stringify(data).slice(0, 8000)}`
: `${summary}\n\n${JSON.stringify(data).slice(0, 8000)}`;
return { content: [{ type: 'text', text: payload }] };
}
rl.on('line', async (line) => {
if (!line.trim()) return;
let msg;
try { msg = JSON.parse(line); }
catch { return; /* ignore */ }
const { id, method, params } = msg;
try {
if (method === 'initialize') {
ok(id, {
protocolVersion: '2024-11-05',
capabilities: { tools: {} },
serverInfo: { name: 'newtube', version: SERVER_VERSION },
});
} else if (method === 'notifications/initialized') {
initialized = true;
// notification : pas de réponse
} else if (method === 'ping') {
ok(id, {});
} else if (method === 'tools/list') {
ok(id, { tools: TOOLS });
} else if (method === 'tools/call') {
const toolName = params?.name;
const toolArgs = params?.arguments || {};
if (!TOOLS.find((t) => t.name === toolName)) { err(id, -32602, `unknown_tool: ${toolName}`); return; }
try {
const { summary, data, text } = await callTool(toolName, toolArgs);
ok(id, mcpText(summary, data, text));
} catch (e) {
ok(id, { content: [{ type: 'text', text: `Erreur ${toolName} : ${e.message}` }], isError: true });
}
} else {
if (id !== undefined) err(id, -32601, `method_not_found: ${method}`);
}
} catch (e) {
if (id !== undefined) err(id, -32603, String(e?.message || e));
}
});
// Log de démarrage sur stderr (ne jamais polluer stdout = canal JSON-RPC)
console.error(`[newtube-mcp] v${SERVER_VERSION} — API=${API_BASE} (stdio)`);
+62
View File
@@ -0,0 +1,62 @@
import { describe, it } from 'node:test';
import assert from 'node:assert/strict';
import { spawn } from 'node:child_process';
function rpc(proc, obj) {
return new Promise((resolve) => {
const onData = (buf) => {
const lines = String(buf).split('\n').filter(Boolean);
for (const l of lines) {
try {
const msg = JSON.parse(l);
if (msg.id === obj.id) {
proc.stdout.off('data', onData);
resolve(msg);
return;
}
} catch {}
}
};
proc.stdout.on('data', onData);
proc.stdin.write(JSON.stringify(obj) + '\n');
});
}
describe('NewTube MCP server (stdio)', () => {
it('initialize + tools/list expose 7 outils dont transcript', async () => {
const proc = spawn(process.execPath, ['./mcp/server.mjs'], { env: { ...process.env, NEWTUBE_API_URL: 'http://127.0.0.1:9/api' } });
try {
const init = await rpc(proc, { jsonrpc: '2.0', id: 1, method: 'initialize', params: {} });
assert.equal(init.result.serverInfo.name, 'newtube');
assert.ok(init.result.capabilities.tools);
const list = await rpc(proc, { jsonrpc: '2.0', id: 2, method: 'tools/list', params: {} });
const names = list.result.tools.map((t) => t.name);
for (const expected of ['search_videos', 'get_video_details', 'get_trending', 'get_transcript', 'get_transcript_text', 'health_check']) {
assert.ok(names.includes(expected), `outil manquant: ${expected}`);
}
const tr = list.result.tools.find((t) => t.name === 'get_transcript');
assert.ok(tr.inputSchema.properties.provider);
assert.ok(tr.inputSchema.properties.videoId);
const ping = await rpc(proc, { jsonrpc: '2.0', id: 3, method: 'ping', params: {} });
assert.deepEqual(ping.result, {});
} finally {
proc.kill();
}
});
it('validation locale sans API (q trop court -> isError)', async () => {
const proc = spawn(process.execPath, ['./mcp/server.mjs'], { env: { ...process.env, NEWTUBE_API_URL: 'http://127.0.0.1:9/api' } });
try {
const call = await rpc(proc, {
jsonrpc: '2.0', id: 10, method: 'tools/call',
params: { name: 'search_videos', arguments: { q: 'x' } },
});
assert.equal(call.result.isError, true);
assert.match(call.result.content[0].text, /min 2/);
} finally {
proc.kill();
}
});
});
+101
View File
@@ -0,0 +1,101 @@
// Tests des 7 outils MCP contre un stub HTTP (aucune API réelle, aucun réseau).
// Stub -> NEWTUBE_API_URL, puis pilotage stdio JSON-RPC (tools/call).
// Run: npm run test:mcp (inclus via --test mcp/*.test.mjs si appelé en glob,
// sinon exécution directe node --test mcp/tools.test.mjs)
import { describe, it, before, after } from 'node:test';
import assert from 'node:assert/strict';
import http from 'node:http';
import { spawn } from 'node:child_process';
const LINES = [
{ t: 1, dur: 2, text: 'Bonjour' },
{ t: 65, dur: 2.5, text: 'deuxième ligne' },
{ t: 120, dur: 2, text: `remplissage ${'x'.repeat(900)}` },
];
const stub = http.createServer((req, res) => {
const u = new URL(req.url, 'http://x');
const json = (code, obj) => { res.writeHead(code, { 'content-type': 'application/json' }); res.end(JSON.stringify(obj)); };
if (u.pathname === '/api/healthz') return json(200, { status: 'ok', youtube: { mode: 'test' } });
if (u.pathname === '/api/search') return json(200, { q: 'q', providers: ['yt'], groups: { yt: [{ id: 'v1', title: 'T' }] }, page: 1, pageSize: 10 });
if (u.pathname === '/api/search/suggest') return json(200, { q: 'q', groups: { yt: ['abc'] } });
if (u.pathname === '/api/details/youtube/v1') return json(200, { videoId: 'v1', title: 'Demo', uploaderName: 'U', duration: 60, views: 10 });
if (u.pathname === '/api/trending') return json(200, { provider: 'yt', items: [{ id: 't1' }] });
if (u.pathname === '/api/transcript/youtube/v1') return json(200, { lang: 'fr', available: true, languages: ['fr', 'en'], lines: LINES });
if (u.pathname === '/api/transcript/youtube/none') return json(200, { lang: null, available: false, languages: [], lines: [], reason: 'no_subtitles' });
return json(404, { error: 'not_found' });
});
await new Promise((r) => stub.listen(0, '127.0.0.1', r));
const STUB_URL = `http://127.0.0.1:${stub.address().port}/api`;
function startMcp() {
return spawn(process.execPath, ['./mcp/server.mjs'], { env: { ...process.env, NEWTUBE_API_URL: STUB_URL } });
}
function rpc(proc, id, method, params = {}) {
return new Promise((resolve, reject) => {
const timer = setTimeout(() => reject(new Error(`timeout ${method}`)), 15000);
const onData = (buf) => {
for (const l of String(buf).split('\n')) {
if (!l.trim()) continue;
try {
const m = JSON.parse(l);
if (m.id === id) { clearTimeout(timer); proc.stdout.off('data', onData); resolve(m); return; }
} catch {}
}
};
proc.stdout.on('data', onData);
proc.stdin.write(JSON.stringify({ jsonrpc: '2.0', id, method, params }) + '\n');
});
}
async function call(proc, id, name, args) {
const m = await rpc(proc, id, 'tools/call', { name, arguments: args });
assert.ok(!m.error, `RPC error: ${JSON.stringify(m.error)}`);
assert.equal(m.result.isError, undefined, `tool ${name} isError: ${m.result.content?.[0]?.text?.slice(0, 300)}`);
return m.result.content[0].text;
}
let proc;
before(async () => {
proc = startMcp();
await rpc(proc, 1, 'initialize');
});
after(() => { try { proc.kill(); } catch {} stub.close(); });
describe('MCP tools (stub API)', () => {
it('search_videos / suggest_queries / get_video_details / get_trending / health_check', async () => {
const t1 = await call(proc, 11, 'search_videos', { q: 'test' });
assert.match(t1, /r.sultats|résultats|1 r/);
const t2 = await call(proc, 12, 'suggest_queries', { q: 'ab' });
assert.match(t2, /Suggestions/);
const t3 = await call(proc, 13, 'get_video_details', { provider: 'youtube', videoId: 'v1' });
assert.match(t3, /Demo/);
const t4 = await call(proc, 14, 'get_trending', { provider: 'yt', limit: 5 });
assert.match(t4, /tendances/);
const t5 = await call(proc, 15, 'health_check', {});
assert.match(t5, /test/);
});
it('get_transcript JSON brut', async () => {
const t = await call(proc, 21, 'get_transcript', { provider: 'youtube', videoId: 'v1', lang: 'fr' });
assert.match(t, /3 lignes/);
assert.match(t, /Bonjour/);
});
it('get_transcript indisponible -> message clair, pas d erreur', async () => {
const t = await call(proc, 22, 'get_transcript', { provider: 'youtube', videoId: 'none' });
assert.match(t, /indisponible/);
assert.match(t, /no_subtitles/);
});
it('get_transcript_text horodaté + srt + troncature', async () => {
const txt = await call(proc, 23, 'get_transcript_text', { provider: 'youtube', videoId: 'v1', lang: 'fr' });
assert.match(txt, /\[00:00:01\] Bonjour/);
assert.match(txt, /\[00:01:05\]/);
const srt = await call(proc, 24, 'get_transcript_text', { provider: 'youtube', videoId: 'v1', format: 'srt' });
assert.match(srt, /-->/);
const cut = await call(proc, 25, 'get_transcript_text', { provider: 'youtube', videoId: 'v1', maxChars: 500 });
assert.match(cut, /tronqu/);
});
it('outil inconnu -> erreur JSON-RPC -32602', async () => {
const m = await rpc(proc, 99, 'tools/call', { name: 'nope', arguments: {} });
assert.ok(m.error);
assert.equal(m.error.code, -32602);
});
});
+4
View File
@@ -9,6 +9,10 @@
"preview": "ng serve --configuration=production",
"api": "node --env-file=.env.local ./server/index.mjs",
"api:watch": "node --watch --env-file=.env.local ./server/index.mjs",
"mcp": "node ./mcp/server.mjs",
"mcp:dev": "node --env-file=.env.local ./mcp/server.mjs",
"test:mcp": "node --test ./mcp/server.test.mjs ./mcp/tools.test.mjs",
"test:api": "node --test ./server/tests/api_coverage.test.mjs",
"test:playlists": "node ./server/tests/playlist_visibility.test.mjs",
"test:preferences": "node ./server/tests/preferences.test.mjs",
"test:search-e2e": "node ./server/tests/search.e2e.test.mjs",
+19 -8
View File
@@ -771,14 +771,20 @@ export function listLikedVideos({ userId, limit = 100, q }) {
}
// -------------------- Playlists --------------------
// Metrics are observability-only: they must never break the core operation
// (e.g. a playlist DELETE must stay 204 even if the metrics insert fails).
export function recordPlaylistMetric({ userId, playlistId, action, meta }) {
const id = cryptoRandomId();
const created_at = nowIso();
const meta_json = meta ? JSON.stringify(meta) : null;
db.prepare(`INSERT INTO playlist_metrics (id, user_id, playlist_id, action, meta_json, created_at)
VALUES (?, ?, ?, ?, ?, ?)`)
.run(id, userId, playlistId, action, meta_json, created_at);
return { id, created_at };
try {
const id = cryptoRandomId();
const created_at = nowIso();
const meta_json = meta ? JSON.stringify(meta) : null;
db.prepare(`INSERT INTO playlist_metrics (id, user_id, playlist_id, action, meta_json, created_at)
VALUES (?, ?, ?, ?, ?, ?)`)
.run(id, userId, playlistId, action, meta_json, created_at);
return { id, created_at };
} catch {
return null;
}
}
export function createPlaylist({ userId, title, description, thumbnail, isPrivate = true }) {
@@ -864,8 +870,13 @@ export function deletePlaylist({ userId, id }) {
const cur = db.prepare(`SELECT * FROM playlists WHERE id = ?`).get(id);
if (!cur) return { removed: false };
if (cur.user_id !== userId) return 'forbidden';
// Record BEFORE the row disappears: playlist_metrics.playlist_id references
// playlists(id), so inserting after the DELETE violates the FK constraint.
try { recordPlaylistMetric({ userId, playlistId: id, action: 'delete' }); } catch {}
// Explicit child cleanup (works even on old DBs whose FK lacks ON DELETE CASCADE).
try { db.prepare(`DELETE FROM playlist_items WHERE playlist_id = ?`).run(id); } catch {}
try { db.prepare(`DELETE FROM playlist_metrics WHERE playlist_id = ?`).run(id); } catch {}
const info = db.prepare(`DELETE FROM playlists WHERE id = ?`).run(id);
recordPlaylistMetric({ userId, playlistId: id, action: 'delete' });
return { removed: (info.changes || 0) > 0 };
}
+12 -5
View File
@@ -3387,6 +3387,10 @@ async function transcriptViaYtDlp(url, langs, netOpts = {}) {
noWarnings: true,
noCheckCertificates: true,
noPlaylist: true,
// Espace les requêtes sous-titres : les rafales déclenchent des 429
// YouTube que les retries immédiats ne font qu'aggraver.
sleepRequests: 2,
sleepSubtitles: 5,
...netOpts,
output: path.join(dir, '%(id)s'),
});
@@ -3515,11 +3519,11 @@ r.get('/transcript/:provider/:videoId', transcriptLimiter, async (req, res) => {
const allowedSet = new Set(allowed.map(transcriptPrimary));
const keepAllowed = (cands) => (cands || []).filter(c => allowedSet.has(transcriptPrimary(c.lang)));
const baseCands = keepAllowed(normalized === 'youtube'
? firstPerLanguage(orderedTracks(meta, lang), 10)
? firstPerLanguage(orderedTracks(meta, lang), 6)
: orderedTracks(meta, lang).slice(0, 5));
// Server-side auto-translation only towards an allowed target language.
const candidates = normalized === 'youtube'
? baseCands.concat(keepAllowed(translatedFallbacks(baseCands, lang))).slice(0, 12)
? baseCands.concat(keepAllowed(translatedFallbacks(baseCands, lang))).slice(0, 8)
: baseCands;
for (const cand of candidates) {
let attempt = 0;
@@ -3534,11 +3538,12 @@ r.get('/transcript/:provider/:videoId', transcriptLimiter, async (req, res) => {
} catch (e) {
const msg = String(e?.message || e);
console.warn('[transcript] track fetch failed:', cand.lang, msg);
// One retry after a short pause on transient 429s.
if (msg.includes(':429') && attempt === 0) {
// Backoff exponentiel + jitter sur 429 (buckets YouTube à la minute) :
// 1 retry @1.5s ne suffit pas quand l'IP est limitée.
if (msg.includes(':429') && attempt < 2) {
sawRateLimit = true;
attempt += 1;
await sleep(1500);
await sleep([2000, 8000][attempt - 1] + Math.floor(Math.random() * 1000));
continue;
}
if (msg.includes(':429')) sawRateLimit = true;
@@ -3546,6 +3551,8 @@ r.get('/transcript/:provider/:videoId', transcriptLimiter, async (req, res) => {
}
}
if (lines.length) break;
// Refroidissement entre candidats quand YouTube limite (évite d'aggraver le 429).
if (sawRateLimit) await sleep(700);
}
if (!lines || lines.length === 0) {
// Direct timedtext fetches failed (429 / Sorry pages / impersonation):
+194
View File
@@ -0,0 +1,194 @@
// Couverture HTTP des routes API non couvertes par les autres suites :
// health, search/suggest (400 + fallback), transcript (400), trending (400 + shape),
// ai/status, openapi, auth (register/login/me), preferences, playlists CRUD,
// likes, history search/watch, subscriptions, telemetry, download jobs.
// Offline-safe : aucune assertion sur le contenu provider externe, uniquement
// les formes stables (status, clés JSON). yt-dlp n'est jamais invoqué avec succès.
// Run: npm run test:api
import { describe, it, before, after } from 'node:test';
import assert from 'node:assert/strict';
import fs from 'node:fs';
import path from 'node:path';
import os from 'node:os';
import net from 'node:net';
import { spawn } from 'node:child_process';
const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'newtube-api-cov-'));
const dbPath = path.join(tmpDir, 'cov.db');
const PORT = await new Promise((resolve) => {
const s = net.createServer();
s.listen(0, '127.0.0.1', () => { const p = s.address().port; s.close(() => resolve(p)); });
});
const base = `http://127.0.0.1:${PORT}`;
const server = spawn(process.execPath, ['./server/index.mjs'], {
cwd: path.resolve(import.meta.dirname, '..', '..'),
env: { ...process.env, PORT: String(PORT), NEWTUBE_DB_FILE: dbPath, JWT_SECRET: 'cov-test-secret', NODE_ENV: 'test' },
stdio: ['ignore', 'pipe', 'pipe'],
});
let logs = '';
server.stdout.on('data', (d) => { logs += d; });
server.stderr.on('data', (d) => { logs += d; });
async function waitUp(timeout = 25000) {
const t0 = Date.now();
while (Date.now() - t0 < timeout) {
try { const r = await fetch(`${base}/api/health`); if (r.status < 500) return true; } catch {}
await new Promise((r) => setTimeout(r, 300));
}
return false;
}
const up = await waitUp();
assert.ok(up, `API ne démarre pas:\n${logs.slice(-3000)}`);
const J = async (url, opts = {}) => {
const r = await fetch(url, opts);
const body = await r.json().catch(() => ({}));
return { status: r.status, body };
};
const auth = (t) => ({ Authorization: `Bearer ${t}` });
const uniq = (p) => `${p}_${Date.now()}_${Math.floor(Math.random() * 1e6)}`;
let token;
const user = uniq('covuser');
before(async () => {
const reg = await J(`${base}/api/auth/register`, {
method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ username: user, password: 'Passw0rd!' }),
});
assert.ok([201, 409].includes(reg.status), `register ${reg.status}`);
const login = await J(`${base}/api/auth/login`, {
method: 'POST', headers: { 'content-type': 'application/json' },
body: JSON.stringify({ username: user, password: 'Passw0rd!' }),
});
assert.equal(login.status, 200);
assert.ok(login.body.accessToken);
token = login.body.accessToken;
});
after(() => { try { server.kill(); } catch {} });
describe('lecture publique / santé', () => {
it('GET /api/health -> ok', async () => {
const r = await J(`${base}/api/health`);
assert.equal(r.status, 200); assert.equal(r.body.status, 'ok');
});
it('GET /healthz expose youtube.mode sans secret', async () => {
const r = await J(`${base}/healthz`);
assert.equal(r.status, 200);
assert.equal(r.body.status, 'ok');
assert.ok(r.body.youtube?.mode);
assert.ok(!JSON.stringify(r.body).includes('GEMINI'));
});
it('GET /api/search?q=x -> 400', async () => {
const r = await J(`${base}/api/search?q=x`);
assert.equal(r.status, 400);
});
it('GET /api/search providers inconnus -> fallback registre', async () => {
const r = await J(`${base}/api/search?q=test&providers=xx,yy&pageSize=2`);
assert.equal(r.status, 200);
assert.ok(r.body.providers.length >= 6);
assert.ok(r.body.groups && typeof r.body.groups === 'object');
});
it('GET /api/search/suggest?q=x -> 400', async () => {
const r = await J(`${base}/api/search/suggest?q=x`);
assert.equal(r.status, 400);
});
it('GET /api/transcript provider inconnu -> 400 available:false', async () => {
const r = await J(`${base}/api/transcript/bogus/vid`);
assert.equal(r.status, 400);
assert.equal(r.body.available, false);
});
it('GET /api/details provider inconnu -> 500 details_failed (sans yt-dlp)', async () => {
const r = await J(`${base}/api/details/bogus/vid`);
assert.equal(r.status, 500);
assert.equal(r.body.error, 'details_failed');
});
it('GET /api/trending provider non-yt -> 400', async () => {
const r = await J(`${base}/api/trending?provider=dm`);
assert.equal(r.status, 400);
});
it('GET /api/trending?provider=yt -> shape {provider,items[]}', async () => {
const r = await J(`${base}/api/trending?provider=yt&limit=2`);
assert.equal(r.status, 200);
assert.equal(r.body.provider, 'yt');
assert.ok(Array.isArray(r.body.items));
});
it('GET /api/ai/status -> {ready:boolean}', async () => {
const r = await J(`${base}/api/ai/status`);
assert.equal(r.status, 200);
assert.equal(typeof r.body.ready, 'boolean');
});
it('GET /api/openapi.json documente playlists', async () => {
const r = await J(`${base}/api/openapi.json`);
assert.equal(r.status, 200);
assert.ok(r.body.paths?.['/playlists']);
});
});
describe('auth + espace utilisateur', () => {
it('register/login 400 sans password, 401 mauvais password', async () => {
const r1 = await J(`${base}/api/auth/login`, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ username: user }) });
assert.equal(r1.status, 400);
const r2 = await J(`${base}/api/auth/login`, { method: 'POST', headers: { 'content-type': 'application/json' }, body: JSON.stringify({ username: user, password: 'mauvais' }) });
assert.equal(r2.status, 401);
});
it('protégées sans token -> 401', async () => {
for (const u of ['/api/user/me', '/api/playlists', '/api/user/likes', '/api/subscriptions', '/api/download/jobs']) {
const r = await J(`${base}${u}`);
assert.equal(r.status, 401, u);
}
});
it('GET /api/user/me + preferences round-trip', async () => {
const me = await J(`${base}/api/user/me`, { headers: auth(token) });
assert.equal(me.status, 200); assert.ok(me.body.id);
const p = await J(`${base}/api/user/preferences`, { headers: auth(token) });
assert.equal(p.status, 200);
const w = await J(`${base}/api/user/preferences`, { method: 'PATCH', headers: { ...auth(token), 'content-type': 'application/json' }, body: JSON.stringify({ defaultProviders: ['yt'] }) });
assert.equal(w.status, 200);
});
it('playlists CRUD complet', async () => {
const h = { ...auth(token), 'content-type': 'application/json' };
const c = await J(`${base}/api/playlists`, { method: 'POST', headers: h, body: JSON.stringify({ title: 'Cov' }) });
assert.equal(c.status, 201); const id = c.body.id; assert.ok(id);
const bad = await J(`${base}/api/playlists`, { method: 'POST', headers: h, body: JSON.stringify({}) });
assert.equal(bad.status, 400);
const g = await J(`${base}/api/playlists/${id}`, { headers: auth(token) });
assert.equal(g.status, 200);
const add = await J(`${base}/api/playlists/${id}/videos`, { method: 'POST', headers: h, body: JSON.stringify({ provider: 'youtube', videoId: 'v1', title: 'T', thumbnail: 'http://x/t.jpg' }) });
assert.equal(add.status, 201);
const del = await J(`${base}/api/playlists/${id}/videos/v1?provider=youtube`, { method: 'DELETE', headers: auth(token) });
assert.equal(del.status, 200);
const u = await J(`${base}/api/playlists/${id}`, { method: 'PUT', headers: h, body: JSON.stringify({ title: 'Cov2' }) });
assert.equal(u.status, 200); assert.equal(u.body.title, 'Cov2');
const rm = await fetch(`${base}/api/playlists/${id}`, { method: 'DELETE', headers: auth(token) });
assert.equal(rm.status, 204);
});
it('likes + history search/watch', async () => {
const h = { ...auth(token), 'content-type': 'application/json' };
// title + thumbnail fournis -> pas d'appel yt-dlp d'enrichissement (offline-safe)
const like = await J(`${base}/api/user/likes`, { method: 'POST', headers: h, body: JSON.stringify({ provider: 'youtube', videoId: 'v1', title: 'T', thumbnail: 'http://x/t.jpg' }) });
assert.ok([200, 201].includes(like.status));
const st = await J(`${base}/api/user/likes/status?provider=youtube&videoId=v1`, { headers: auth(token) });
assert.equal(st.status, 200);
const hs = await J(`${base}/api/user/history/search`, { method: 'POST', headers: h, body: JSON.stringify({ query: 'cov' }) });
assert.ok([200, 201].includes(hs.status));
const ls = await J(`${base}/api/user/history/search`, { headers: auth(token) });
assert.equal(ls.status, 200);
const hw = await J(`${base}/api/user/history/watch`, { method: 'POST', headers: h, body: JSON.stringify({ provider: 'youtube', videoId: 'v1' }) });
assert.ok([200, 201].includes(hw.status));
const lw = await J(`${base}/api/user/history/watch`, { headers: auth(token) });
assert.equal(lw.status, 200);
});
it('subscriptions + telemetry + download jobs', async () => {
const h = { ...auth(token), 'content-type': 'application/json' };
const sub = await J(`${base}/api/subscriptions`, { method: 'POST', headers: h, body: JSON.stringify({ provider: 'yt', externalId: 'ch1', title: 'C' }) });
assert.ok([200, 201].includes(sub.status));
const ls = await J(`${base}/api/subscriptions`, { headers: auth(token) });
assert.equal(ls.status, 200);
const tel = await J(`${base}/api/telemetry/events`, { method: 'POST', headers: h, body: JSON.stringify({ event: 'search_submit' }) });
assert.ok([200, 201].includes(tel.status));
const dj = await J(`${base}/api/download/jobs`, { headers: auth(token) });
assert.equal(dj.status, 200);
});
});