From 4bdcd393fea75327ffe31bb1970cf84dac7eee53 Mon Sep 17 00:00:00 2001 From: Bruno Charest Date: Fri, 25 Sep 2026 19:38:26 -0400 Subject: [PATCH] feat(youtube): InnerTube-first search, transcripts and watch-next related (Steps 15-18) - InnerTube layer via pinned youtubei.js 18.1.0 (no quota, no key): search with merged continuations (unlimited pages), watch-next related with LockupView mapping, caption-track discovery - 3-layer dispatcher (YT_SEARCH_MODE, default innertube-first): innertube -> yt-dlp scrape -> official API, graceful errors.yt - Robust yt-dlp binary resolution (YT_DLP_PATH > PATH > bundled) with systematic API fallback (fixes spawn ENOENT in UI) - Transcript: InnerTube caption discovery (YT_TRANSCRIPT_SOURCE), reusing pickTrack/orderedTracks/parseTrackText; yt-dlp fallback kept - Watch: sidebar uses real watch-next related[] (/api/details), title-search fallback for other providers - Cache: memory LRU + SQLite (youtube_search_cache, youtube_metrics), never persist empty pages; /healthz observability; /api/trending - Includes pending Step 15/16 leftovers in same files (suggest, test scripts); unrelated provider adapters left uncommitted --- .env.example | 22 +- .github/workflows/ci.yml | 12 + README.md | 15 +- .../20250926_add_youtube_scrape_cache.sql | 18 + docker-compose/.env.example | 17 + docs/transcript_architecture_dev.md | 769 ++++++++++++++++++ package-lock.json | 36 + package.json | 12 +- server/db.mjs | 71 ++ server/index.mjs | 461 ++++++++++- server/providers/channel-content.mjs | 35 +- server/providers/youtube-common.mjs | 174 ++++ server/providers/youtube-innertube.mjs | 331 ++++++++ server/providers/youtube-scrape.mjs | 197 +++++ server/providers/youtube.mjs | 411 ++++++---- server/tests/suggest.test.mjs | 130 +++ server/tests/transcript.test.mjs | 212 +++++ server/tests/youtube-innertube.test.mjs | 131 +++ server/tests/youtube-scrape.test.mjs | 98 +++ server/transcript.mjs | 309 +++++++ src/app/search/suggest.service.ts | 72 ++ src/app/search/suggest.spec.ts | 103 +++ src/app/search/suggest.util.ts | 72 ++ src/components/watch/watch.component.ts | 290 ++++++- todo.md | 282 +++++++ 25 files changed, 4112 insertions(+), 168 deletions(-) create mode 100644 db/migrations/20250926_add_youtube_scrape_cache.sql create mode 100644 docs/transcript_architecture_dev.md create mode 100644 server/providers/youtube-common.mjs create mode 100644 server/providers/youtube-innertube.mjs create mode 100644 server/providers/youtube-scrape.mjs create mode 100644 server/tests/suggest.test.mjs create mode 100644 server/tests/transcript.test.mjs create mode 100644 server/tests/youtube-innertube.test.mjs create mode 100644 server/tests/youtube-scrape.test.mjs create mode 100644 server/transcript.mjs create mode 100644 src/app/search/suggest.service.ts create mode 100644 src/app/search/suggest.spec.ts create mode 100644 src/app/search/suggest.util.ts diff --git a/.env.example b/.env.example index 4f9e0ce..dea25cd 100644 --- a/.env.example +++ b/.env.example @@ -1,8 +1,28 @@ # Configuration des clés API pour les adaptateurs de recherche # Copiez ce fichier en .env et remplissez les valeurs -# YouTube API Key (obligatoire pour la recherche YouTube) +# --- YouTube hybride Step 17-18 : InnerTube (sans clé) > scrape yt-dlp > API officielle --- +# Mode : innertube-first (défaut, 0 quota, pagination illimitée) | innertube-only +# | scrape-first | api-first | scrape-only | api-only +YT_SEARCH_MODE=innertube-first +# Localisation InnerTube (gl=region, hl=langue) +YT_INNERTUBE_GL=FR +YT_INNERTUBE_HL=fr +# Découverte des pistes de sous-titres YouTube : innertube-first (défaut, +# sans spawn yt-dlp) | ytdlp-only | innertube-only +YT_TRANSCRIPT_SOURCE=innertube-first +# Cache scrape (30 min) + binaire yt-dlp (frais requis contre le bot-check) +YT_SCRAPE_TTL_MS=1800000 +YT_DLP_TIMEOUT_MS=20000 +# YT_DLP_PATH=/usr/local/bin/yt-dlp +# Anti-ban : cookies exportés (volume :ro en Docker) + PO-Token + proxy sortant +# YT_COOKIES_FILE=/cookies/youtube.txt +# YT_PO_TOKEN= +# YT_EGRESS_PROXY=http://proxy:8080 + +# YouTube API Key (fallback uniquement en scrape-first, obligatoire en api-only) # Obtenez une clé ici : https://console.developers.google.com/ +# Accepte YOUTUBE_API_KEYS en CSV ('k1,k2') ou JSON ('["k1","k2"]') avec rotation auto. YOUTUBE_API_KEY=your_youtube_api_key_here # Twitch Client ID (optionnel, pour une recherche Twitch plus complète) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ecde92e..fda1401 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -42,3 +42,15 @@ jobs: - name: E2E scenarios (search UX) run: npm run test:search-e2e + + - name: Suggest typeahead (Step 15) + run: npm run test:suggest + + - name: Transcripts (Step 16) + run: npm run test:transcript + + - name: YouTube scrape-first (Step 17, offline) + run: npm run test:ytscrape + + - name: YouTube InnerTube (Step 18, offline) + run: npm run test:ytinnertube diff --git a/README.md b/README.md index 7871f26..564e59b 100644 --- a/README.md +++ b/README.md @@ -56,20 +56,28 @@ Un seul champ, tous les fournisseurs — avec filtres, raccourcis et deep-links * **Deep-links** : `/#/search?q=…&providers=yt,ru` relance la recherche filtrée — partageable * **Fallback préférence** : URL sans `providers` → préférence `defaultProviders` de l’utilisateur → provider actif * **Accessibilité** : focus trap dans les modals, Esc pour fermer, aria-combobox sur le champ, focus restauré à la fermeture +* **Typeahead requête** : sous l’input, suggestions de requêtes (recherches récentes 🕘 + groupes par provider `YT/DM/…`, sous-chaîne surlignée) — `GET /api/search/suggest?q=…&providers=…&limit=…` (min 2 caractères, debounce 250 ms, cache 5 min, dégradation `[]` par provider) ; ↑/↓/Enter/Tab/Esc, priorité au popover `@` ### Endpoints API concernés -* `GET /api/search?q=…&providers=yt,dm` — fan-out parallèle, réponse groupée par provider +* `GET /api/search?q=…&providers=yt,dm` — fan-out parallèle, réponse groupée par provider (`page`/`pageSize`/`sort` ; YT sans quota via InnerTube + continuations) +* `GET /api/search/suggest?q=…&providers=yt,dm&limit=10` — typeahead `{ q, groups: { yt: string[], … } }` (cache 5 min, rate-limit) +* `GET /api/details/youtube/:videoId` — métadonnées + `related[]` (watch-next InnerTube, `?related=0` pour désactiver) +* `GET /api/trending?provider=yt&limit=…` — tendances YT sans clé +* `GET /healthz` (alias `/api/healthz`) — mode YT, binaire yt-dlp `binOk`, cache, métriques quota/jour, clés +* `GET /api/transcript/:provider/:videoId?lang=&instance=&slug=&sourceUrl=` — transcript `{ lang, available, languages, lines: [{ t, dur, text }] }` (cache 24 h, rate-limit 10/min ; absent → 200 `{ available: false }`, échec → 502 ; YouTube : découverte des pistes via InnerTube, `YT_TRANSCRIPT_SOURCE`) * `GET/PATCH /api/user/preferences` — `defaultProviders` (tableau JSON, sanitizé serveur) -* `POST /api/telemetry/events` — événements UX anonymes (whitelist : `search_submit`, `provider_picker_open`, `provider_apply`, `at_autocomplete_use`, `quick_menu_open`) +* `POST /api/telemetry/events` — événements UX anonymes (whitelist : `search_submit`, `provider_picker_open`, `provider_apply`, `at_autocomplete_use`, `quick_menu_open`, `suggest_shown`, `suggest_used`) ### Tests ```bash npm run test:search # unitaires SearchService + parsing @ + picker npm run test:search-e2e # scénarios e2e (serveur réel isolé) +npm run test:suggest # typeahead : parsing/dédup + contrat /api/search/suggest +npm run test:transcript # transcripts : parseurs json3/vtt + contrat /api/transcript npm run test:preferences # persistance defaultProviders npm run test:telemetry # télémétrie minimale ``` @@ -242,7 +250,8 @@ Ajoutez au besoin `log-driver`, `log-opts`, `default-address-pools`, etc. * ✅ **Téléchargements** intégrés — file d'attente **persistée en SQLite** (survit aux redémarrages API), jobs **par utilisateur** (répertoires isolés, ownership sur status/fichier/cancel), **quota de stockage** configurable (`DOWNLOAD_STORAGE_QUOTA_BYTES`, fenêtre `DOWNLOAD_QUOTA_WINDOW_MS`), **reprise** des jobs échoués/interrrompus (bouton Réessayer), **page Bibliothèque > Téléchargements** (filtres par état, progression live, quota), nettoyage auto des fichiers orphelins au boot * 🔧 Variables : `DOWNLOAD_MAX_CONCURRENT` (2), `DOWNLOAD_STORAGE_QUOTA_BYTES` (5 GiB), `DOWNLOAD_QUOTA_WINDOW_MS` (30 j), `DOWNLOAD_PROVIDERS` (peertube,odysee) * ⏳ **Import/Export** playlists (JSON / OPML-like) -* ⏳ **Sous-titres & transcripts** (si dispo API; fallback parsing) +* ✅ **Sous-titres & transcripts** — `GET /api/transcript/:provider/:videoId` (yt-dlp `subtitles`/`automatic_captions`, parsing `json3`/`vtt`, cache 24 h, rate-limit), panneau **Transcript** sur la page Watch (sélecteur de langue, dégradation propre si indisponible) +* ✅ **YouTube sans quota (InnerTube façon SmartTube)** — `youtubei.js` pinné, chaîne `innertube → scrape yt-dlp → API officielle` (`YT_SEARCH_MODE`, défaut `innertube-first`), **pagination illimitée** via continuations (scroll infini, `page=2,3…`), **vidéos connexes** watch-next dans `GET /api/details/youtube/:videoId` → sidebar Watch, `GET /api/trending?provider=yt`, cache mémoire + SQLite, anti-ban (`YT_COOKIES_FILE`, `YT_PO_TOKEN`, `YT_EGRESS_PROXY`), observabilité `/healthz` (mode, binaire `binOk`, cache, quota jour, clés) * ⏳ **PWA** (installable, offline cache des métadonnées) * ⏳ **Chromecast / AirPlay** * ⏳ Mode “TV” diff --git a/db/migrations/20250926_add_youtube_scrape_cache.sql b/db/migrations/20250926_add_youtube_scrape_cache.sql new file mode 100644 index 0000000..2d81e88 --- /dev/null +++ b/db/migrations/20250926_add_youtube_scrape_cache.sql @@ -0,0 +1,18 @@ +-- Step 17 : cache persistant recherche YouTube (scrape + API) + métriques quota journalières. +CREATE TABLE IF NOT EXISTS youtube_search_cache ( + q_hash TEXT PRIMARY KEY, + q TEXT NOT NULL, + payload_json TEXT NOT NULL, + source TEXT NOT NULL DEFAULT 'scrape', + created_at INTEGER NOT NULL, + expires_at INTEGER NOT NULL +); +CREATE INDEX IF NOT EXISTS idx_yt_cache_exp ON youtube_search_cache(expires_at); + +CREATE TABLE IF NOT EXISTS youtube_metrics ( + day TEXT PRIMARY KEY, + scrape_calls INTEGER NOT NULL DEFAULT 0, + api_calls INTEGER NOT NULL DEFAULT 0, + quota_units INTEGER NOT NULL DEFAULT 0, + updated_at TEXT NOT NULL +); diff --git a/docker-compose/.env.example b/docker-compose/.env.example index 32db9c5..9484fc4 100644 --- a/docker-compose/.env.example +++ b/docker-compose/.env.example @@ -15,6 +15,23 @@ TWITCH_CLIENT_SECRET=votre_client_secret_twitch_ici # Configuration du cache (en millisecondes) YT_CACHE_TTL_MS=3600000 # 1 heure par défaut +# --- YouTube hybride : InnerTube (sans clé) > scrape yt-dlp > API officielle --- +# Mode : innertube-first (défaut, 0 quota, pagination illimitée) | innertube-only +# | scrape-first | api-first | scrape-only | api-only +YT_SEARCH_MODE=innertube-first +YT_INNERTUBE_GL=FR +YT_INNERTUBE_HL=fr +# Sous-titres YouTube : innertube-first (défaut) | ytdlp-only | innertube-only +YT_TRANSCRIPT_SOURCE=innertube-first +# Cache scrape/pagination (30 min) + timeout yt-dlp +YT_SCRAPE_TTL_MS=1800000 +YT_DLP_TIMEOUT_MS=20000 +# Anti-ban (optionnel) : binaire explicite, cookies exportés, PO-Token, proxy sortant +# YT_DLP_PATH=/usr/local/bin/yt-dlp +# YT_COOKIES_FILE=/cookies/youtube.txt +# YT_PO_TOKEN= +# YT_EGRESS_PROXY=http://proxy:8080 + # Téléchargements : providers autorisés (liste CSV) + concurrence max par user DOWNLOAD_PROVIDERS=youtube,dailymotion,twitch,peertube,odysee,rumble DOWNLOAD_MAX_CONCURRENT=2 diff --git a/docs/transcript_architecture_dev.md b/docs/transcript_architecture_dev.md new file mode 100644 index 0000000..6717d11 --- /dev/null +++ b/docs/transcript_architecture_dev.md @@ -0,0 +1,769 @@ +# Guide d'architecture et de développement — Transcripts multi-fournisseurs pour NewTube + +**Version :** 1.0 +**Date :** 2026-09-25 +**Statut :** Proposition d'implémentation +**Périmètre :** Ajout de la fonctionnalité « Transcripts / Sous-titres » sur NewTube, pour les 6 fournisseurs supportés. + +--- + +## Table des matières + +1. [Contexte](#1-contexte) +2. [Objectifs et non-objectifs](#2-objectifs-et-non-objectifs) +3. [Architecture globale](#3-architecture-globale) +4. [Backend](#4-backend) +5. [Frontend](#5-frontend) +6. [Référence API](#6-référence-api) +7. [Tests](#7-tests) +8. [Développement pas à pas](#8-développement-pas-à-pas) +9. [Déploiement et configuration](#9-déploiement-et-configuration) +10. [Risques et mitigations](#10-risques-et-mitigations) +11. [Alternatives écartées](#11-alternatives-écartées) +12. [Roadmap](#12-roadmap) +13. [Annexes](#13-annexes) + +--- + +## 1. Contexte + +NewTube est un agrégateur multi-fournisseurs qui s'appuie déjà sur `yt-dlp` via `youtube-dl-exec` pour extraire les métadonnées et les formats de téléchargement. + +Fournisseurs supportés : + +- YouTube +- Dailymotion +- Twitch +- PeerTube +- Odysee +- Rumble + +L'analyse du code existant montre que : + +- `server/index.mjs:10` importe `youtube-dl-exec`. +- `providerUrlFrom()` (`server/index.mjs:541`) construit une URL normalisée pour les 6 fournisseurs. +- Les routes `/api/details/:provider/:videoId` (`l.988`) et `/api/download/.../formats` (`l.1061`) utilisent déjà `youtubedl(url, { dumpSingleJson, skipDownload })`. +- `yt-dlp` renvoie dans ce même JSON les champs `subtitles` et `automatic_captions`. +- Côté UI, `watch.component.html:154` et `:165` fournissent le motif du panneau « Download » à copier pour un panneau « Transcript ». +- Le README contient déjà une ligne roadmap : `⏳ Sous-titres & transcripts` (`ligne 245`). + +La fonctionnalité peut donc être ajoutée **sans nouveau registre de providers**, **sans adaptateur par fournisseur**, et **sans migration de base de données**. + +--- + +## 2. Objectifs et non-objectifs + +### 2.1 Objectifs + +- Extraire les sous-titres manuels et automatiques via `yt-dlp`. +- Supporter plusieurs langues avec un sélecteur. +- Offrir une dégradation propre quand aucune piste n'existe. +- Mettre en cache les transcripts (immuables) pour éviter les appels répétés. +- Limiter le débit pour éviter les erreurs HTTP 429 de YouTube. +- Ajouter une UI simple dans la page « Watch ». +- Rester compatible avec les 6 fournisseurs sans code spécifique par plateforme. + +### 2.2 Non-objectifs + +- Traduction automatique des transcripts. +- Transcription audio via LLM ou service tiers. +- Téléchargement de la vidéo complète. +- Authentification OAuth. +- Seek avancé dans la vidéo depuis le transcript (optionnel, phase 2). +- Support garanti des sous-titres sur Twitch, Odysee et Rumble. + +--- + +## 3. Architecture globale + +### 3.1 Schéma + +```mermaid +flowchart LR + A[Client Watch] -->|GET /api/transcript/:provider/:videoId| B[API NewTube] + B --> C{Cache ?} + C -->|Oui| D[Retour JSON] + C -->|Non| E[providerUrlFrom] + E --> F[yt-dlp dumpSingleJson] + F --> G[subtitles / automatic_captions] + G --> H[pickTrack] + H --> I[Fetch piste json3/vtt] + I --> J[parseJson3 / parseVtt] + J --> K[Mise en cache] + K --> D + D --> A +``` + +### 3.2 Composants + +| Composant | Fichier | Rôle | +|---|---|---| +| Module transcript | `server/transcript.mjs` | Fonctions pures : sélection de piste, parsing json3/vtt | +| Route API | `server/index.mjs` | Endpoint `/api/transcript/...`, cache, rate-limit | +| UI Watch | `watch.component.ts/html` | Bouton, panneau, sélecteur de langue, affichage | +| Tests | `server/tests/transcript.test.mjs` | Tests unitaires et d'intégration | +| Script npm | `package.json` | `test:transcript` | +| Documentation | `README.md` | Case roadmap cochée + bump de version | + +### 3.3 Principe directeur + +> **Rung 2 : réutiliser, pas réinventer.** + +Une seule route, un seul module de parsing, une seule UI. La capacité dépend de la plateforme, mais le code ne change pas. + +--- + +## 4. Backend + +### 4.1 Module `server/transcript.mjs` + +Fonctions pures, testables sans réseau ni base de données. + +#### 4.1.1 `pickTrack(json, lang)` + +Sélectionne la meilleure piste disponible. + +Priorités : + +1. Sous-titres manuels (`subtitles`) dans la langue demandée. +2. Sous-titres manuels dans une langue proche (`fr-*`). +3. Sous-titres automatiques (`automatic_captions`) dans la langue demandée. +4. Sous-titres automatiques dans une langue proche. +5. Première piste disponible. + +```js +export function pickTrack(json, lang = 'fr') { + const manual = json.subtitles || {}; + const auto = json.automatic_captions || {}; + const all = { ...auto, ...manual }; + const languages = Object.keys(all); + + if (!languages.length) { + return { track: null, languages: [], lang: null }; + } + + const findLang = (dict, code) => + dict[code] || Object.keys(dict).find(k => k.startsWith(code + '-')); + + const chosenLang = + findLang(manual, lang) ? lang : + findLang(auto, lang) ? lang : + languages[0]; + + const track = + manual[chosenLang] || + auto[chosenLang] || + manual[languages[0]] || + auto[languages[0]]; + + return { track, languages, lang: chosenLang }; +} +``` + +#### 4.1.2 `parseJson3(data)` + +Convertit le format `json3` de YouTube en lignes normalisées. + +```js +export function parseJson3(data) { + const events = data.events || []; + return events + .filter(e => e.segs) + .map(e => ({ + t: (e.tStartMs || 0) / 1000, + dur: (e.dDurationMs || 0) / 1000, + text: e.segs.map(s => s.utf8).join('').trim() + })) + .filter(line => line.text); +} +``` + +#### 4.1.3 `parseVtt(text)` + +Convertit un fichier VTT en lignes normalisées. + +```js +export function parseVtt(text) { + const lines = []; + const blocks = text.split(/\n\n+/); + + for (const block of blocks) { + const match = block.match(/(\d{2}):(\d{2}):(\d{2})\.(\d{3}) --> (\d{2}):(\d{2}):(\d{2})\.(\d{3})/); + if (!match) continue; + + const start = + parseInt(match[1]) * 3600 + + parseInt(match[2]) * 60 + + parseInt(match[3]) + + parseInt(match[4]) / 1000; + + const end = + parseInt(match[5]) * 3600 + + parseInt(match[6]) * 60 + + parseInt(match[7]) + + parseInt(match[8]) / 1000; + + const content = block + .split('\n') + .filter(line => !line.includes('-->') && !/^\d+$/.test(line)) + .join(' ') + .trim(); + + if (content) { + lines.push({ t: start, dur: end - start, text: content }); + } + } + + return lines; +} +``` + +#### 4.1.4 Structure de retour normalisée + +```json +{ + "lang": "fr", + "available": true, + "languages": ["fr", "en", "es"], + "lines": [ + { "t": 0, "dur": 2.5, "text": "Bonjour à tous" }, + { "t": 2.5, "dur": 3.1, "text": "Bienvenue dans cette vidéo" } + ] +} +``` + +En l'absence de piste : + +```json +{ + "lang": null, + "available": false, + "languages": [], + "lines": [] +} +``` + +--- + +### 4.2 Route API dans `server/index.mjs` + +#### 4.2.1 Signature + +```http +GET /api/transcript/:provider/:videoId?lang=&instance=&slug=&sourceUrl= +``` + +#### 4.2.2 Flux d'exécution + +1. Valider le provider. +2. Construire la clé de cache : `transcript:${provider}:${videoId}:${lang}`. +3. Vérifier le cache. +4. Si absent : + - `providerUrlFrom()` construit l'URL. + - `youtubedl(url, { dumpSingleJson: true, skipDownload: true })`. + - `pickTrack(json, lang)`. + - Si aucune piste : répondre `{ available: false }` avec HTTP 200. + - Sinon : `fetch(track.url)`. + - Parser selon le format (`json3` ou `vtt`). + - Mettre en cache. +5. Répondre. + +#### 4.2.3 Pseudo-code + +```js +app.get('/api/transcript/:provider/:videoId', channelsLimiter, async (req, res) => { + const { provider, videoId } = req.params; + const { lang = 'fr', instance, slug, sourceUrl } = req.query; + + const cacheKey = `transcript:${provider}:${videoId}:${lang}`; + const cached = transcriptCache.get(cacheKey); + if (cached) return res.json(cached); + + try { + const url = providerUrlFrom(provider, videoId, { instance, slug, sourceUrl }); + const json = await youtubedl(url, { + dumpSingleJson: true, + skipDownload: true, + }); + + const { track, languages, lang: chosenLang } = pickTrack(json, lang); + + if (!track) { + const empty = { lang: null, available: false, languages: [], lines: [] }; + transcriptCache.set(cacheKey, empty, TTL_LONG); + return res.json(empty); + } + + const response = await fetch(track.url); + const text = await response.text(); + + const lines = track.ext === 'json3' + ? parseJson3(JSON.parse(text)) + : parseVtt(text); + + const result = { + lang: chosenLang, + available: true, + languages, + lines, + }; + + transcriptCache.set(cacheKey, result, TTL_LONG); + res.json(result); + } catch (err) { + console.error('Transcript error:', err); + res.status(502).json({ available: false, error: 'transcript_fetch_failed' }); + } +}); +``` + +--- + +### 4.3 Cache + +- **Type :** `Map` avec TTL, comme `ytCache` (`server/index.mjs:838`). +- **Clé :** `transcript:${provider}:${videoId}:${lang}`. +- **TTL :** long (ex. 24 h) car les transcripts sont immuables. +- **Évolution :** pour un déploiement multi-instances, remplacer par Redis ou Memcached. + +```js +const transcriptCache = { + data: new Map(), + get(key) { + const entry = this.data.get(key); + if (!entry) return null; + if (Date.now() > entry.expires) { + this.data.delete(key); + return null; + } + return entry.value; + }, + set(key, value, ttlMs) { + this.data.set(key, { value, expires: Date.now() + ttlMs }); + } +}; +``` + +--- + +### 4.4 Rate limiting + +- Réutiliser le motif `channelsLimiter`. +- Limiter par IP, par exemple 10 requêtes / minute sur `/api/transcript/...`. +- En cas de HTTP 429 de YouTube : + - Ne pas réessayer immédiatement. + - Attendre 20 s minimum. + - Utiliser un backoff exponentiel. + - Si le problème persiste, activer le fallback. + +--- + +### 4.5 Fallback en cas de 429 persistant + +Si le endpoint `timedtext` de YouTube renvoie trop de 429 : + +1. Laisser `yt-dlp` écrire le VTT dans un dossier temporaire. +2. Lire le fichier. +3. Parser avec `parseVtt()`. + +Pattern déjà utilisé pour les downloads : `youtubedl.exec`. + +```js +const tmp = path.join(os.tmpdir(), `transcript-${Date.now()}.vtt`); +await youtubedl.exec(url, { + writeAutoSub: true, + subLang: lang, + subFormat: 'vtt', + output: tmp, +}); +const text = await fs.readFile(tmp, 'utf8'); +const lines = parseVtt(text); +``` + +--- + +## 5. Frontend + +### 5.1 UI dans `watch.component` + +#### 5.1.1 Bouton + +Ajouter un bouton « Transcript » à côté du bouton « Download ». + +```html + +``` + +#### 5.1.2 Panneau + +Copier le motif du panneau Download (`watch.component.html:165`). + +```html +
+
+

Transcript

+ +
+ +
+ Chargement… +
+ +
+ Aucun sous-titre disponible pour cette vidéo. +
+ +
+
+ {{ line.t | duration }} + {{ line.text }} +
+
+
+``` + +#### 5.1.3 Composant TypeScript + +```ts +transcriptOpen = false; +transcriptLoading = false; +transcriptAvailable = false; +transcriptLanguages: string[] = []; +transcriptLines: { t: number; dur: number; text: string }[] = []; +selectedLang = 'fr'; + +toggleTranscript() { + this.transcriptOpen = !this.transcriptOpen; + if (this.transcriptOpen && !this.transcriptLines.length) { + this.loadTranscript(); + } +} + +async loadTranscript() { + this.transcriptLoading = true; + const res = await fetch( + `/api/transcript/${this.provider}/${this.videoId}?lang=${this.selectedLang}` + ); + const data = await res.json(); + + this.transcriptAvailable = data.available; + this.transcriptLanguages = data.languages || []; + this.transcriptLines = data.lines || []; + this.transcriptLoading = false; +} +``` + +#### 5.1.4 Gestion de l'absence de sous-titres + +- Si `available: false`, afficher un message clair. +- Le bouton peut rester visible mais le panneau indique l'absence. +- Ne jamais casser la page Watch. + +--- + +### 5.2 Variante UI #1 — recommandée + +- Affichage du transcript. +- Sélecteur de langue. +- Pas de seek. + +**Avantages :** simple, robuste, peu de code. + +--- + +### 5.3 Variante UI #2 — optionnelle + +- Comme #1, plus : clic sur une ligne = seek dans la vidéo. + +**Implémentation :** + +- Player natif : `seekBy` dans `video-player.component.ts:112`. +- Iframe YouTube : `seekTo` dans `iframe-progress.service.ts:104`. + +**Inconvénient :** le seek iframe n'est attaché que sous certaines conditions. Il faut brancher par provider. Sensiblement plus de code. + +**Recommandation :** garder #2 pour une phase ultérieure. + +--- + +## 6. Référence API + +### 6.1 Endpoint + +```http +GET /api/transcript/:provider/:videoId +``` + +### 6.2 Paramètres + +| Nom | Type | Requis | Description | +|---|---|---|---| +| `provider` | string | Oui | `youtube`, `dailymotion`, `twitch`, `peertube`, `odysee`, `rumble` | +| `videoId` | string | Oui | Identifiant de la vidéo | +| `lang` | string | Non | Langue souhaitée (ex. `fr`, `en`). Défaut : `fr` | +| `instance` | string | Non | Pour PeerTube | +| `slug` | string | Non | Pour PeerTube | +| `sourceUrl` | string | Non | Pour Odysee / Rumble | + +### 6.3 Réponse succès + +```json +{ + "lang": "fr", + "available": true, + "languages": ["fr", "en"], + "lines": [ + { "t": 0, "dur": 2.5, "text": "Bonjour" }, + { "t": 2.5, "dur": 3.1, "text": "Bienvenue" } + ] +} +``` + +### 6.4 Réponse absence de sous-titres + +```json +{ + "lang": null, + "available": false, + "languages": [], + "lines": [] +} +``` + +HTTP 200. + +### 6.5 Réponse erreur + +```json +{ + "available": false, + "error": "transcript_fetch_failed" +} +``` + +HTTP 502. + +--- + +## 7. Tests + +### 7.1 Tests unitaires + +Fichier : `server/tests/transcript.test.mjs` + +- `pickTrack()` : + - manuel prioritaire sur auto. + - `fr` → `fr-*` → autre langue. + - aucun track → `available: false`. +- `parseJson3()` : + - events valides. + - events sans `segs`. + - texte vide filtré. +- `parseVtt()` : + - blocs valides. + - timestamps corrects. + - contenu multi-lignes. + +### 7.2 Tests d'intégration + +- Mocker `youtubedl` pour renvoyer un JSON avec `subtitles`. +- Vérifier la route `/api/transcript/...`. +- Vérifier le cache : deuxième appel ne déclenche pas `youtubedl`. +- Vérifier le rate-limit. + +### 7.3 Tests manuels + +| Provider | Vidéo testée | Résultat attendu | +|---|---|---| +| YouTube | Vidéo avec auto-captions | `available: true`, ~100 langues | +| Dailymotion | Vidéo avec pistes | `available: true` | +| PeerTube | Vidéo framatube | souvent `available: false` | +| Twitch | VOD | `available: false` | +| Odysee | Vidéo | `available: false` | +| Rumble | Vidéo | `available: false` | + +### 7.4 Script npm + +```json +{ + "scripts": { + "test:transcript": "node --test server/tests/transcript.test.mjs" + } +} +``` + +--- + +## 8. Développement pas à pas + +### Checklist + +- [ ] Créer `server/transcript.mjs` avec `pickTrack`, `parseJson3`, `parseVtt`. +- [ ] Ajouter la route `GET /api/transcript/:provider/:videoId` dans `server/index.mjs`. +- [ ] Ajouter le cache `transcriptCache` avec TTL long. +- [ ] Ajouter le rate-limit sur le motif `channelsLimiter`. +- [ ] Ajouter la gestion des erreurs 429 et le fallback temp dir. +- [ ] Créer `server/tests/transcript.test.mjs`. +- [ ] Ajouter `test:transcript` dans `package.json`. +- [ ] Ajouter le bouton et le panneau dans `watch.component.html`. +- [ ] Ajouter la logique dans `watch.component.ts`. +- [ ] Mettre à jour le README : cocher `⏳ Sous-titres & transcripts`. +- [ ] Bumper la version selon la règle du projet. +- [ ] Tester manuellement sur YouTube et Dailymotion. +- [ ] Vérifier la dégradation propre sur Twitch, Odysee, Rumble. + +**Volume estimé :** 150 à 200 lignes, 4 fichiers. + +--- + +## 9. Déploiement et configuration + +### 9.1 Variables d'environnement + +| Variable | Description | Défaut | +|---|---|---| +| `YTDLP_PATH` | Chemin vers `yt-dlp` | `yt-dlp` | +| `TRANSCRIPT_CACHE_TTL` | TTL du cache en ms | `86400000` (24 h) | +| `TRANSCRIPT_RATE_LIMIT` | Requêtes / minute / IP | `10` | +| `TRANSCRIPT_FALLBACK_TMP` | Activer le fallback temp dir | `false` | + +### 9.2 Mise à jour de `yt-dlp` + +- `yt-dlp` évolue vite. +- Prévoir une mise à jour régulière. +- Surveiller les breaking changes. + +### 9.3 Monitoring + +- Compter les HTTP 429 sur `timedtext`. +- Compter les `available: false` par provider. +- Mesurer le temps de réponse de `/api/transcript/...`. +- Alerter si le taux d'échec dépasse un seuil. + +--- + +## 10. Risques et mitigations + +| Risque | Impact | Mitigation | +|---|---|---| +| ToS des plateformes | Élevé | Usage personnel / auto-hébergé, ne pas revendre | +| HTTP 429 YouTube | Moyen | Cache long, rate-limit, fallback temp dir | +| `yt-dlp` cassé | Élevé | Mise à jour régulière, tests de non-régression | +| Providers sans sous-titres | Faible | Dégradation propre `available: false` | +| Cache mémoire non partagé | Moyen | Redis en production multi-instances | +| Seek iframe complexe | Faible | Reporter en phase 2 | + +--- + +## 11. Alternatives écartées + +| Alternative | Raison du rejet | +|---|---| +| YouTube Data API `captions.download` | Nécessite OAuth du propriétaire | +| Un implémentation par provider | 6 chemins de code pour le même résultat | +| Fetch navigateur `timedtext` | CORS non garanti + casse le proxy clé API | +| Services tiers / LLM de transcription | Dépendance, coût, vie privée, YAGNI | +| Scrapers custom par plateforme | Maintenance impossible | + +--- + +## 12. Roadmap + +### Phase 1 — Base (recommandée) + +- Route `/api/transcript/...` +- Module `transcript.mjs` +- UI #1 : affichage + sélecteur de langue +- Cache + rate-limit +- Tests + +### Phase 2 — Confort + +- UI #2 : clic sur une ligne = seek +- Branchement par provider +- Meilleure gestion des timecodes + +### Phase 3 — Scalabilité + +- Cache Redis +- Proxies rotatifs +- Monitoring avancé +- Support de nouveaux formats + +--- + +## 13. Annexes + +### 13.1 Format `json3` + +Exemple simplifié : + +```json +{ + "events": [ + { + "tStartMs": 0, + "dDurationMs": 2500, + "segs": [{ "utf8": "Bonjour" }] + }, + { + "tStartMs": 2500, + "dDurationMs": 3100, + "segs": [{ "utf8": "Bienvenue" }] + } + ] +} +``` + +### 13.2 Format VTT + +Exemple simplifié : + +```vtt +WEBVTT + +00:00:00.000 --> 00:00:02.500 +Bonjour + +00:00:02.500 --> 00:00:05.600 +Bienvenue +``` + +### 13.3 Exemple de réponse complète + +```json +{ + "lang": "fr", + "available": true, + "languages": ["fr", "en", "es"], + "lines": [ + { "t": 0, "dur": 2.5, "text": "Bonjour à tous" }, + { "t": 2.5, "dur": 3.1, "text": "Bienvenue dans cette vidéo" }, + { "t": 5.6, "dur": 4.2, "text": "Aujourd'hui, nous allons voir..." } + ] +} +``` + +### 13.4 Arborescence des fichiers modifiés + +```text +server/ + index.mjs # route + cache + rate-limit + transcript.mjs # fonctions pures + tests/ + transcript.test.mjs # tests +watch.component.ts # logique UI +watch.component.html # panneau Transcript +package.json # script test:transcript +README.md # roadmap + version +``` + +--- + +**Fin du guide.** diff --git a/package-lock.json b/package-lock.json index 188435e..14e9534 100644 --- a/package-lock.json +++ b/package-lock.json @@ -34,6 +34,7 @@ "rxjs": "^7.8.2", "tailwindcss": "latest", "youtube-dl-exec": "^3.0.0", + "youtubei.js": "^18.1.0", "zone.js": "~0.15.1" }, "devDependencies": { @@ -965,6 +966,12 @@ "node": ">=6.9.0" } }, + "node_modules/@bufbuild/protobuf": { + "version": "2.15.0", + "resolved": "https://registry.npmjs.org/@bufbuild/protobuf/-/protobuf-2.15.0.tgz", + "integrity": "sha512-DAheWUkVr/SJTWCc+lg9dhY0eN4SaWlf4+bG1KzHeXbnqt0AfB/NX0Z+VunGlM1ki1B4zVvye27MpKh/svySUA==", + "license": "(Apache-2.0 AND BSD-3-Clause)" + }, "node_modules/@cspotcode/source-map-support": { "version": "0.8.1", "resolved": "https://registry.npmjs.org/@cspotcode/source-map-support/-/source-map-support-0.8.1.tgz", @@ -5672,6 +5679,12 @@ } } }, + "node_modules/fflate": { + "version": "0.8.3", + "resolved": "https://registry.npmjs.org/fflate/-/fflate-0.8.3.tgz", + "integrity": "sha512-tbZNuJrLwGUp3zshBtdy4W+ORxZuIh8a5ilyIEQDC5rY1f3U20JMry0Ll3WBzU58EZKsEuJFXhb5gwv8CsPvgA==", + "license": "MIT" + }, "node_modules/ffmpeg-static": { "version": "5.2.0", "resolved": "https://registry.npmjs.org/ffmpeg-static/-/ffmpeg-static-5.2.0.tgz", @@ -6943,6 +6956,15 @@ "integrity": "sha512-abv/qOcuPfk3URPfDzmZU1LKmuw8kT+0nIHvKrKgFrwifol/doWcdA4ZqsWQ8ENrFKkd67Mfpo/LovbIUsbt3w==", "license": "MIT" }, + "node_modules/meriyah": { + "version": "7.3.3", + "resolved": "https://registry.npmjs.org/meriyah/-/meriyah-7.3.3.tgz", + "integrity": "sha512-uE5cnoNj+UYhoMdZDuymCzr5TzEuE3ZF2C4kn6Z76rLhkwnvZ/+6dUZnM41T3fifyuhW0+n8vK7eq6DKJxUfPA==", + "license": "ISC", + "engines": { + "node": ">=20.0.0" + } + }, "node_modules/methods": { "version": "1.1.2", "resolved": "https://registry.npmjs.org/methods/-/methods-1.1.2.tgz", @@ -9992,6 +10014,20 @@ "node": ">= 18" } }, + "node_modules/youtubei.js": { + "version": "18.1.0", + "resolved": "https://registry.npmjs.org/youtubei.js/-/youtubei.js-18.1.0.tgz", + "integrity": "sha512-4Goo3ZeO6/kE8Mq67eC8Z9K1rijeqRwyKxf/YM4lqf8bB4r/s8JC0V3YFx7+l/AJR3LnAsulTsiIj+AFe0cduA==", + "funding": [ + "https://github.com/sponsors/LuanRT" + ], + "license": "MIT", + "dependencies": { + "@bufbuild/protobuf": "^2.0.0", + "fflate": "^0.8.2", + "meriyah": "^7.3.1" + } + }, "node_modules/zod": { "version": "3.25.76", "resolved": "https://registry.npmjs.org/zod/-/zod-3.25.76.tgz", diff --git a/package.json b/package.json index cdad382..21802a3 100644 --- a/package.json +++ b/package.json @@ -15,7 +15,12 @@ "test:telemetry": "node ./server/tests/telemetry.test.mjs", "test:downloads": "node ./server/tests/download_jobs.test.mjs", "test:subscriptions": "node --loader ts-node/esm --experimental-specifier-resolution=node src/services/subscriptions.service.spec.ts", - "test:search": "node --loader ts-node/esm --experimental-specifier-resolution=node src/app/search/search.service.spec.ts && node --loader ts-node/esm --experimental-specifier-resolution=node src/app/search/search-components.spec.ts" + "test:search": "node --loader ts-node/esm --experimental-specifier-resolution=node src/app/search/search.service.spec.ts && node --loader ts-node/esm --experimental-specifier-resolution=node src/app/search/search-components.spec.ts", + "test:suggest": "node --loader ts-node/esm --experimental-specifier-resolution=node src/app/search/suggest.spec.ts && node ./server/tests/suggest.test.mjs", + "test:transcript": "node --test server/tests/transcript.test.mjs", + "test:ytscrape": "node server/tests/youtube-scrape.test.mjs", + "test:ytinnertube": "node server/tests/youtube-innertube.test.mjs", + "ytdlp:update": "yt-dlp -U || python3 -m yt_dlp -U || echo \"yt-dlp update: installez yt-dlp puis relancez\"" }, "dependencies": { "@angular/build": "^20.1.0", @@ -44,12 +49,13 @@ "rxjs": "^7.8.2", "tailwindcss": "latest", "youtube-dl-exec": "^3.0.0", + "youtubei.js": "18.1.0", "zone.js": "~0.15.1" }, "devDependencies": { "@types/node": "^22.14.0", + "ts-node": "^10.9.2", "typescript": "~5.8.2", - "vite": "^6.2.0", - "ts-node": "^10.9.2" + "vite": "^6.2.0" } } diff --git a/server/db.mjs b/server/db.mjs index d318bea..9772c9e 100644 --- a/server/db.mjs +++ b/server/db.mjs @@ -998,3 +998,74 @@ function subscriptionRowToDto(row) { channel, }; } + +// -------------------- Step 17 : cache persistant recherche YouTube -------------------- +function ensureYoutubeCacheTables() { + try { + db.exec(`CREATE TABLE IF NOT EXISTS youtube_search_cache ( + q_hash TEXT PRIMARY KEY, q TEXT NOT NULL, payload_json TEXT NOT NULL, + source TEXT NOT NULL DEFAULT 'scrape', created_at INTEGER NOT NULL, expires_at INTEGER NOT NULL + );`); + db.exec(`CREATE INDEX IF NOT EXISTS idx_yt_cache_exp ON youtube_search_cache(expires_at);`); + db.exec(`CREATE TABLE IF NOT EXISTS youtube_metrics ( + day TEXT PRIMARY KEY, scrape_calls INTEGER NOT NULL DEFAULT 0, + api_calls INTEGER NOT NULL DEFAULT 0, quota_units INTEGER NOT NULL DEFAULT 0, updated_at TEXT NOT NULL + );`); + } catch {} +} +ensureYoutubeCacheTables(); + +export function getCachedYoutubeSearch(qHash) { + try { + ensureYoutubeCacheTables(); + const row = db.prepare(`SELECT payload_json AS payload, source, expires_at AS exp FROM youtube_search_cache WHERE q_hash = ?`).get(qHash); + if (!row) return null; + if (Date.now() >= Number(row.exp || 0)) { + try { db.prepare(`DELETE FROM youtube_search_cache WHERE q_hash = ?`).run(qHash); } catch {} + return null; + } + try { return { items: JSON.parse(String(row.payload || '[]')), source: row.source }; } catch { return null; } + } catch { return null; } +} + +export function setCachedYoutubeSearch(qHash, q, items, source, ttlMs) { + try { + ensureYoutubeCacheTables(); + const now = Date.now(); + db.prepare(`INSERT INTO youtube_search_cache (q_hash, q, payload_json, source, created_at, expires_at) + VALUES (?, ?, ?, ?, ?, ?) + ON CONFLICT(q_hash) DO UPDATE SET q=excluded.q, payload_json=excluded.payload_json, + source=excluded.source, created_at=excluded.created_at, expires_at=excluded.expires_at`) + .run(qHash, String(q || '').slice(0, 300), JSON.stringify(items || []), source, now, now + ttlMs); + // Cap : garde 2000 entrées les plus fraîches + try { db.exec(`DELETE FROM youtube_search_cache WHERE q_hash NOT IN (SELECT q_hash FROM youtube_search_cache ORDER BY expires_at DESC LIMIT 2000)`); } catch {} + } catch {} +} + +export function pruneYoutubeCache() { + try { db.prepare(`DELETE FROM youtube_search_cache WHERE expires_at <= ?`).run(Date.now()); } catch {} +} + +export function incYoutubeMetrics({ scrapeCalls = 0, apiCalls = 0, quotaUnits = 0 } = {}) { + try { + ensureYoutubeCacheTables(); + const day = new Date().toISOString().slice(0, 10); + db.prepare(`INSERT INTO youtube_metrics (day, scrape_calls, api_calls, quota_units, updated_at) + VALUES (?, ?, ?, ?, ?) + ON CONFLICT(day) DO UPDATE SET scrape_calls = scrape_calls + ?, api_calls = api_calls + ?, + quota_units = quota_units + ?, updated_at = excluded.updated_at`) + .run(day, scrapeCalls, apiCalls, quotaUnits, new Date().toISOString(), scrapeCalls, apiCalls, quotaUnits); + } catch {} +} + +export function getYoutubeMetricsToday() { + try { + ensureYoutubeCacheTables(); + const day = new Date().toISOString().slice(0, 10); + return db.prepare(`SELECT * FROM youtube_metrics WHERE day = ?`).get(day) || { day, scrape_calls: 0, api_calls: 0, quota_units: 0 }; + } catch { return { scrape_calls: 0, api_calls: 0, quota_units: 0 }; } +} + +export function countYoutubeCacheRows() { + try { return db.prepare(`SELECT COUNT(1) AS n FROM youtube_search_cache`).get()?.n || 0; } catch { return 0; } +} diff --git a/server/index.mjs b/server/index.mjs index fa38613..07f7e40 100644 --- a/server/index.mjs +++ b/server/index.mjs @@ -7,12 +7,16 @@ import bcrypt from 'bcryptjs'; import jwt from 'jsonwebtoken'; import fs from 'node:fs'; import path from 'node:path'; -import youtubedl from 'youtube-dl-exec'; +import youtubedlPkg, { create as createYtDlp } from 'youtube-dl-exec'; +import { execFile as execFileCb } from 'node:child_process'; +import { promisify } from 'node:util'; +import { fileURLToPath as serverFileURLToPath } from 'node:url'; import ffmpegPath from 'ffmpeg-static'; import * as cheerio from 'cheerio'; import axios from 'axios'; import rumbleRouter from './rumble.mjs'; import { providerRegistry, validateProviders } from './providers/registry.mjs'; +import { pickTrack, parseTrackText, parseVtt, orderedTracks, normalizeTranscriptProvider, transcriptTrackExt, looksLikeHtmlError } from './transcript.mjs'; import { getUserByUsername, getUserById, @@ -73,9 +77,53 @@ import { } from './db.mjs'; import { getChannelAdapter, setTwitchTokenProvider } from './providers/channel-registry.mjs'; import { fetchChannelContent } from './providers/channel-content.mjs'; +import { getSearchMode, getYtDlpBin, hasCookiesFile, metricsSnapshot } from './providers/youtube-common.mjs'; +import { ytScrapeCacheStats } from './providers/youtube.mjs'; const app = express(); const PORT = Number(process.env.PORT || 4000); +// yt-dlp: prefer the newest binary available. The copy bundled with +// youtube-dl-exec goes stale (YouTube then answers "The page needs to be +// reloaded" to dump-single-json), while a system install is usually fresher. +// `YT_DLP_PATH` wins, then `yt-dlp` on PATH, then the bundled binary. +const execFileAsync = promisify(execFileCb); +async function binaryVersion(bin) { + try { + const { stdout } = await execFileAsync(bin, ['--version'], { timeout: 15000 }); + return String(stdout || '').trim().split('\n')[0].trim(); + } catch { + return null; + } +} +let youtubedl = youtubedlPkg; +let ytDlpInfo = 'bundled'; +try { + let bundledBin = null; + try { + const u = new URL('../node_modules/youtube-dl-exec/bin/', import.meta.url); + const cand = path.join(String(serverFileURLToPath(u)), process.platform === 'win32' ? 'yt-dlp.exe' : 'yt-dlp'); + if (fs.existsSync(cand)) bundledBin = cand; + } catch {} + const bundledVer = bundledBin ? await binaryVersion(bundledBin) : null; + const candidates = []; + if (process.env.YT_DLP_PATH) candidates.push(process.env.YT_DLP_PATH); + candidates.push(process.platform === 'win32' ? 'yt-dlp.exe' : 'yt-dlp', 'yt-dlp'); + let best = null; + for (const cand of candidates) { + if (!cand) continue; + const ver = await binaryVersion(cand); + if (ver && (!best || ver > best.ver)) best = { bin: cand, ver }; + } + if (best && (!bundledVer || best.ver >= bundledVer)) { + youtubedl = createYtDlp(best.bin); + ytDlpInfo = `${best.bin} (${best.ver})`; + } else if (bundledVer) { + ytDlpInfo = `bundled (${bundledVer})`; + } +} catch (e) { + console.warn('[config] yt-dlp binary selection failed, using bundled:', e?.message || e); +} +console.log(`[config] yt-dlp: ${ytDlpInfo}`); const IS_PROD = String(process.env.NODE_ENV || '').toLowerCase() === 'production'; const JWT_SECRET = process.env.JWT_SECRET || 'dev-secret-change-me'; if (!process.env.JWT_SECRET) { @@ -1006,6 +1054,14 @@ r.get('/peertube/:instance/*', async (req, res) => { }); // -------------------- Generic video details (GET) -------------------- +// Cache mémoire pour les vidéos connexes InnerTube (watch-next) : TTL 1h, LRU 200. +const YT_RELATED_TTL_MS = Number(process.env.YT_RELATED_TTL_MS || 60 * 60 * 1000); +const ytRelatedCache = new Map(); +function ytRelatedCacheSet(key, items) { + if (ytRelatedCache.has(key)) ytRelatedCache.delete(key); + ytRelatedCache.set(key, { ts: Date.now(), items }); + while (ytRelatedCache.size > 200) { const o = ytRelatedCache.keys().next().value; if (o === undefined) break; ytRelatedCache.delete(o); } +} // Returns metadata such as title, description, uploader, thumbnail, duration and views for a provider/videoId // Supports query params similar to download endpoints: instance (PeerTube), slug (Odysee), sourceUrl (direct) r.get('/details/:provider/:videoId', async (req, res) => { @@ -1071,6 +1127,22 @@ r.get('/details/:provider/:videoId', async (req, res) => { url, type: 'video', }; + // Step 18 : vidéos connexes façon SmartTube (watch-next InnerTube, best-effort, 0 quota). + // N'implique que YouTube ; toute erreur -> `related: []`, la réponse reste 200. + if (String(provider) === 'youtube' && req.query.related !== '0') { + try { + const { getRelatedViaInnerTube } = await import('./providers/youtube-innertube.mjs'); + const relKey = `related:${videoId}`; + const cached = ytRelatedCache.get(relKey); + if (cached && (Date.now() - cached.ts) < YT_RELATED_TTL_MS) { + out.related = cached.items; + } else { + const items = await getRelatedViaInnerTube(videoId, 24).catch(() => []); + out.related = items; + ytRelatedCacheSet(relKey, items); + } + } catch { out.related = []; } + } return res.json(out); } catch (e) { return res.status(500).json({ error: 'details_failed', details: String(e?.message || e) }); @@ -1684,7 +1756,7 @@ r.post('/telemetry/events', authMiddleware, telemetryLimiter, (req, res) => { const { event, meta } = req.body || {}; if (!event || typeof event !== 'string') return res.status(400).json({ error: 'event_required' }); // Whitelist known event names to keep the table clean - const allowed = new Set(['search_submit', 'provider_picker_open', 'provider_apply', 'at_autocomplete_use', 'quick_menu_open']); + const allowed = new Set(['search_submit', 'provider_picker_open', 'provider_apply', 'at_autocomplete_use', 'quick_menu_open', 'suggest_shown', 'suggest_used']); if (!allowed.has(event)) return res.status(400).json({ error: 'unknown_event' }); const row = insertTelemetryEvent({ userId: req.user.id, event, meta: (meta && typeof meta === 'object') ? meta : null }); return res.status(201).json(row || { ok: true }); @@ -1704,7 +1776,7 @@ r.get('/telemetry/events', authMiddleware, (req, res) => { r.get('/telemetry/summary', authMiddleware, (req, res) => { const since = typeof req.query.since === 'string' ? req.query.since : undefined; const events = {}; - for (const name of ['search_submit', 'provider_picker_open', 'provider_apply', 'at_autocomplete_use', 'quick_menu_open']) { + for (const name of ['search_submit', 'provider_picker_open', 'provider_apply', 'at_autocomplete_use', 'quick_menu_open', 'suggest_shown', 'suggest_used']) { events[name] = countTelemetryEvents({ event: name, since }); } return res.json({ events }); @@ -1944,6 +2016,53 @@ r.get('/img/odysee', async (req, res) => { app.use('/api', r); // Health endpoint for container checks app.get('/api/health', (_req, res) => res.json({ status: 'ok' })); +// Step 17 : observabilité YouTube (mode, yt-dlp, cache, quota). Aucun secret exposé. +app.get(['/healthz', '/api/healthz'], async (_req, res) => { + try { + const [{ getYoutubeMetricsToday, countYoutubeCacheRows }, common] = + await Promise.all([import('./db.mjs'), import('./providers/youtube-common.mjs')]); + let ytdlpVersion = null; + let resolvedBin = null; + try { + resolvedBin = await common.resolveYtDlpBin(); + const { stdout } = await execFileAsync(resolvedBin, ['--version'], { timeout: 10000 }); + ytdlpVersion = String(stdout || '').trim().split('\n')[0].trim() || null; + } catch {} + const keys = common.getYouTubeKeys(); + const banned = []; + try { for (const [k, until] of ytKeyBans || []) if (Date.now() < until) banned.push(`...${String(k).slice(-4)}`); } catch {} + res.json({ + status: 'ok', + youtube: { + mode: getSearchMode(), + ytdlp: { bin: resolvedBin || getYtDlpBin(), version: ytdlpVersion, info: ytDlpInfo, binOk: Boolean(ytdlpVersion) }, + antiban: { + cookiesFile: hasCookiesFile(), + poToken: Boolean(String(process.env.YT_PO_TOKEN || '').trim()), + egressProxy: Boolean(String(process.env.YT_EGRESS_PROXY || '').trim()), + }, + cache: { ...ytScrapeCacheStats(), sqliteRows: countYoutubeCacheRows() }, + metrics: { ...metricsSnapshot(), today: getYoutubeMetricsToday() }, + keys: { count: keys.length, banned }, + }, + }); + } catch (e) { + res.status(500).json({ status: 'error', error: String(e?.message || e) }); + } +}); +// Step 17 : trending YouTube sans clé (scrape) avec fallback [] propre. +app.get('/api/trending', async (req, res) => { + try { + const provider = String(req.query.provider || 'yt'); + const limit = Math.min(50, Math.max(1, Number(req.query.limit || 24))); + if (provider !== 'yt') return res.status(400).json({ error: 'only yt supported in phase 1' }); + const { getTrendingViaScrape } = await import('./providers/youtube-scrape.mjs'); + const items = await getTrendingViaScrape(limit); + return res.json({ provider, items }); + } catch (e) { + return res.json({ provider: 'yt', items: [], error: String(e?.code || e?.message || 'trending_failed') }); + } +}); // Alias to support Angular dev proxy paths in both dev and production builds app.use('/proxy/api', r); // Mount dedicated Rumble router (browse, search, video) @@ -2170,6 +2289,342 @@ app.get('/api/search', async (req, res) => { } }); +// -------------------- Query typeahead suggestions (Step 15) -------------------- +// GET /api/search/suggest?q=…&providers=yt,dm&limit=10 -> { q, groups: { yt: string[], dm: string[] } } +// Fan-out over provider `suggest()` handlers; providers without one degrade to [] (never 500). +const SUGGEST_CACHE_TTL_MS = Number(process.env.SUGGEST_CACHE_TTL_MS || 5 * 60 * 1000); +const SUGGEST_CACHE_MAX_ENTRIES = 500; +/** @type {Map} */ +const suggestCache = new Map(); +function suggestCacheGet(key) { + const hit = suggestCache.get(key); + if (!hit) return null; + if ((Date.now() - hit.ts) >= SUGGEST_CACHE_TTL_MS) { + suggestCache.delete(key); + return null; + } + // LRU refresh + suggestCache.delete(key); + suggestCache.set(key, hit); + return hit.data; +} +function suggestCacheSet(key, data) { + if (suggestCache.has(key)) suggestCache.delete(key); + suggestCache.set(key, { ts: Date.now(), data }); + while (suggestCache.size > SUGGEST_CACHE_MAX_ENTRIES) { + const oldest = suggestCache.keys().next().value; + suggestCache.delete(oldest); + } +} +const suggestLimiter = rateLimit({ + windowMs: 60 * 1000, + max: Number(process.env.SUGGEST_RATE_LIMIT || 60), + standardHeaders: true, + legacyHeaders: false, +}); +app.get('/api/search/suggest', suggestLimiter, async (req, res) => { + try { + const rawQ = typeof req.query.q === 'string' ? req.query.q : ''; + const q = rawQ.trim(); + if (q.length < 2) { + return res.status(400).json({ error: 'q is required and must be at least 2 characters long' }); + } + const limit = Math.min(20, Math.max(1, Number(req.query.limit || 10))); + const requested = typeof req.query.providers === 'string' ? String(req.query.providers) : ''; + const validProviders = validateProviders(requested); + const cacheKey = `suggest:${validProviders.join(',')}:${q.toLowerCase()}:${limit}`; + const cached = suggestCacheGet(cacheKey); + if (cached) return res.json(cached); + const results = await Promise.allSettled( + validProviders.map((providerId) => { + const mod = providerRegistry[providerId]; + if (!mod || typeof mod.suggest !== 'function') return Promise.resolve([]); + return Promise.resolve().then(() => mod.suggest(q, { limit })); + }) + ); + const groups = {}; + results.forEach((result, index) => { + const providerId = validProviders[index]; + if (result.status === 'fulfilled' && Array.isArray(result.value)) { + groups[providerId] = result.value + .map((s) => String(s ?? '').trim()) + .filter(Boolean) + .slice(0, limit); + } else { + groups[providerId] = []; + } + }); + const data = { q, groups }; + suggestCacheSet(cacheKey, data); + return res.json(data); + } catch (e) { + return res.status(500).json({ error: 'suggest_failed', details: String(e?.message || e) }); + } +}); + +// -------------------- Video transcripts (Step 16, Phase 1) -------------------- +// GET /api/transcript/:provider/:videoId?lang=&instance=&slug=&sourceUrl= +// -> { lang, available, languages, lines: [{ t, dur, text }] } +// One endpoint / one parser / one UI whatever the provider (yt-dlp subtitles). +const TRANSCRIPT_CACHE_TTL_MS = Number(process.env.TRANSCRIPT_CACHE_TTL || 24 * 60 * 60 * 1000); +const TRANSCRIPT_CACHE_MAX_ENTRIES = 200; +/** @type {Map} */ +const transcriptCache = new Map(); +function transcriptCacheGet(key) { + const hit = transcriptCache.get(key); + if (!hit) return null; + if ((Date.now() - hit.ts) >= TRANSCRIPT_CACHE_TTL_MS) { + transcriptCache.delete(key); + return null; + } + transcriptCache.delete(key); + transcriptCache.set(key, hit); + return hit.data; +} +function transcriptCacheSet(key, data) { + if (transcriptCache.has(key)) transcriptCache.delete(key); + transcriptCache.set(key, { ts: Date.now(), data }); + while (transcriptCache.size > TRANSCRIPT_CACHE_MAX_ENTRIES) { + const oldest = transcriptCache.keys().next().value; + transcriptCache.delete(oldest); + } +} +const transcriptLimiter = rateLimit({ + windowMs: 60 * 1000, + max: Number(process.env.TRANSCRIPT_RATE_LIMIT || 10), + standardHeaders: true, + legacyHeaders: false, + // Always answer JSON (default handler sends an HTML/text page, which the + // Angular HttpClient cannot parse as JSON and surfaces as a raw SyntaxError). + handler: (req, res) => { + return res.status(429).json({ available: false, error: 'rate_limited' }); + }, +}); + +/** Providers known NOT to expose subtitle tracks via yt-dlp (no transcript possible). */ +const TRANSCRIPT_UNSUPPORTED_PROVIDERS = new Set(['twitch', 'odysee', 'rumble']); + +/** Download subtitles via yt-dlp (handles YouTube impersonation + 429-prone + * translated tracks). Tries `langs` in order, returns the first non-empty + * parsed lines with the language that worked, or null. Bounded by a timeout + * (yt-dlp subtitle downloads can hang on .part files); partial results on + * disk are still scanned when yt-dlp exits non-zero. */ +async function transcriptViaYtDlp(url, langs) { + const osMod = await import('node:os'); + const dir = await fs.promises.mkdtemp(path.join(osMod.tmpdir(), 'newtube-transcript-')); + const scanDir = async (wanted) => { + let files = []; + try { + files = (await fs.promises.readdir(dir)).filter((f) => /\.vtt$/i.test(f) && !/\.part$/i.test(f)); + } catch { return null; } + // Ignore stale/empty files + const nonEmpty = []; + for (const f of files) { + try { + const st = await fs.promises.stat(path.join(dir, f)); + if (st.size > 0) nonEmpty.push(f); + } catch {} + } + // Prefer requested languages first + nonEmpty.sort((a, b) => { + const la = a.toLowerCase(), lb = b.toLowerCase(); + const ia = wanted.findIndex((w) => la.includes(`.${w.toLowerCase()}.`) || la.endsWith(`.${w.toLowerCase()}.vtt`)); + const ib = wanted.findIndex((w) => lb.includes(`.${w.toLowerCase()}.`) || lb.endsWith(`.${w.toLowerCase()}.vtt`)); + return (ia === -1 ? 99 : ia) - (ib === -1 ? 99 : ib); + }); + for (const file of nonEmpty) { + try { + const text = await fs.promises.readFile(path.join(dir, file), 'utf8'); + const lines = parseVtt(text); + if (lines.length) { + const m = /\.([a-z]{2,3}(?:-[a-z]{2,4})?)\.vtt$/i.exec(file); + return { lines, lang: m ? m[1] : null }; + } + } catch {} + } + return null; + }; + try { + const wanted = Array.from(new Set((langs || []).map((l) => String(l || '').split('-')[0]).filter(Boolean))); + if (!wanted.includes('en')) wanted.push('en'); + const subLangs = wanted.slice(0, 4).join(','); + const child = youtubedl(url, { + writeSub: true, + writeAutoSub: true, + subLangs, + subFormat: 'vtt/best', + skipDownload: true, + noWarnings: true, + noCheckCertificates: true, + noPlaylist: true, + output: path.join(dir, '%(id)s'), + }); + const timer = setTimeout(() => { try { child.kill('SIGKILL'); } catch {} }, 90000); + try { + await child; + } catch (e) { + // Non-zero exit (e.g. one language 429'd) — partial files may still exist. + console.warn('[transcript] yt-dlp subtitle download exited non-zero:', String(e?.message || e).slice(0, 200)); + } finally { + clearTimeout(timer); + } + return await scanDir(wanted); + } catch (e) { + console.warn('[transcript] yt-dlp subtitle download failed:', e?.message || e); + try { + const wanted = Array.from(new Set((langs || []).map((l) => String(l || '').split('-')[0]).filter(Boolean))); + return await scanDir(wanted); + } catch { return null; } + } finally { + try { await fs.promises.rm(dir, { recursive: true, force: true }); } catch {} + } +} + +/** Fetch one timedtext track URL and parse it (any format). Returns [] on any failure. */ +async function fetchTimedTextLines(track) { + const resp = await fetch(String(track.url), { + headers: { + 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36', + Accept: 'application/json, text/vtt, text/*;q=0.9, */*;q=0.8', + 'Accept-Language': 'en-US,en;q=0.9,fr;q=0.8', + }, + signal: AbortSignal.timeout(15000), + }); + if (!resp.ok) throw new Error(`track_fetch_failed:${resp.status}`); + const text = await resp.text(); + if (!text || looksLikeHtmlError(text)) throw new Error('track_fetch_failed:html_error_page'); + return parseTrackText(text, transcriptTrackExt(track)); +} + +app.get('/api/transcript/:provider/:videoId', transcriptLimiter, async (req, res) => { + try { + const { provider, videoId } = req.params; + const lang = String(req.query.lang || 'fr').slice(0, 12) || 'fr'; + const normalized = normalizeTranscriptProvider(provider); + if (!normalized || !videoId) { + return res.status(400).json({ available: false, error: 'invalid_provider_or_video' }); + } + const cacheKey = `transcript:${normalized}:${videoId}:${lang.toLowerCase()}`; + const cached = transcriptCacheGet(cacheKey); + if (cached) return res.json(cached); + // Source de découverte des pistes : InnerTube d'abord pour YouTube + // (mêmes URLs timedtext, sans spawn yt-dlp), yt-dlp sinon/en secours. + // YT_TRANSCRIPT_SOURCE=innertube-first (défaut) | ytdlp-only | innertube-only + const transcriptSource = String(process.env.YT_TRANSCRIPT_SOURCE || 'innertube-first').trim().toLowerCase(); + const transcriptUrl = providerUrlFrom(normalized, String(videoId), { + instance: req.query.instance || undefined, + slug: req.query.slug || undefined, + sourceUrl: req.query.sourceUrl || undefined, + }); + let meta = null; + if (normalized === 'youtube' && transcriptSource !== 'ytdlp-only') { + try { + const { getCaptionTracksViaInnerTube } = await import('./providers/youtube-innertube.mjs'); + const cap = await getCaptionTracksViaInnerTube(String(videoId)); + if (cap.trackCount > 0) { + meta = { subtitles: cap.subtitles, automatic_captions: cap.automatic_captions }; + console.log(`[transcript] source=innertube tracks=${cap.trackCount} langs=${cap.languages.join(',')}`); + } else { + // InnerTube fait foi (même backend que le lecteur) : 0 piste = pas + // de sous-titres, sans payer un dump yt-dlp complet. + console.log('[transcript] source=innertube tracks=0 -> no_subtitles'); + const empty = { lang: null, available: false, languages: [], lines: [], reason: 'no_subtitles' }; + transcriptCacheSet(cacheKey, empty); + return res.json(empty); + } + } catch (e) { + console.warn('[transcript] innertube discovery failed, fallback yt-dlp:', e?.code || String(e?.message || e).slice(0, 120)); + if (transcriptSource === 'innertube-only') { + return res.status(502).json({ available: false, languages: [], lines: [], error: 'transcript_temporarily_unavailable', retryable: true }); + } + } + } + if (!meta) { + try { + const raw = await youtubedl(transcriptUrl, { dumpSingleJson: true, skipDownload: true, noWarnings: true, noCheckCertificates: true }); + meta = (typeof raw === 'string') ? JSON.parse(raw || '{}') : (raw || {}); + } catch (e) { + console.error('[transcript] yt-dlp failed:', e?.message || e); + return res.status(502).json({ available: false, languages: [], lines: [], error: 'transcript_temporarily_unavailable', retryable: true }); + } + } + const { track, languages, lang: chosenLang } = pickTrack(meta, lang); + if (!track) { + // Definitive absence: distinguish "provider never exposes subtitles" + // (twitch/odysee/rumble) from "this video has none" — cacheable 200s. + const reason = TRANSCRIPT_UNSUPPORTED_PROVIDERS.has(normalized) ? 'provider_unsupported' : 'no_subtitles'; + const empty = { lang: null, available: false, languages: languages || [], lines: [], reason }; + transcriptCacheSet(cacheKey, empty); + return res.json(empty); + } + // Try candidate tracks in order: requested language, then original (`en`), + // then everything else. Translated tracks are often rate-limited while the + // original still works — never fail on the first track alone. + // Bounded: hammering dozens of timedtext URLs only worsens YouTube 429s. + const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); + let lines = []; + let workedLang = chosenLang; + let sawRateLimit = false; + for (const cand of orderedTracks(meta, lang).slice(0, 5)) { + let attempt = 0; + for (;;) { + try { + const parsed = await fetchTimedTextLines(cand.track); + if (parsed && parsed.length) { + lines = parsed; + workedLang = cand.lang || chosenLang; + } + break; + } catch (e) { + const msg = String(e?.message || e); + console.warn('[transcript] track fetch failed:', cand.lang, msg); + // One retry after a short pause on transient 429s. + if (msg.includes(':429') && attempt === 0) { + sawRateLimit = true; + attempt += 1; + await sleep(1500); + continue; + } + if (msg.includes(':429')) sawRateLimit = true; + break; + } + } + if (lines.length) break; + } + if (!lines || lines.length === 0) { + // Direct timedtext fetches failed (429 / Sorry pages / impersonation): + // let yt-dlp download the subtitles instead (handles all of the above). + try { + const viaDlp = await transcriptViaYtDlp(transcriptUrl, [lang, chosenLang].filter(Boolean)); + if (viaDlp && viaDlp.lines && viaDlp.lines.length) { + lines = viaDlp.lines; + if (viaDlp.lang) workedLang = viaDlp.lang; + } + } catch {} + } + if (!lines || lines.length === 0) { + // Subtitle tracks exist but their content could not be retrieved or + // parsed (YouTube 429 / Sorry pages / impersonation): the video HAS + // subtitles, so this is transient — 502 with languages + retryable flag, + // never cached as "no subtitles". + return res.status(502).json({ + available: false, + languages: languages || [], + lines: [], + error: 'transcript_temporarily_unavailable', + retryable: true, + rateLimited: sawRateLimit || undefined, + }); + } + const result = { lang: workedLang, available: true, languages: languages || [], lines }; + transcriptCacheSet(cacheKey, result); + return res.json(result); + } catch (e) { + console.error('[transcript] unexpected error:', e?.message || e); + return res.status(502).json({ available: false, languages: [], lines: [], error: 'transcript_temporarily_unavailable', retryable: true }); + } +}); + // -------------------- Static Frontend (Angular build) -------------------- const distRoot = path.join(process.cwd(), 'dist'); const distBrowser = path.join(distRoot, 'browser'); diff --git a/server/providers/channel-content.mjs b/server/providers/channel-content.mjs index 664b725..c5a3129 100644 --- a/server/providers/channel-content.mjs +++ b/server/providers/channel-content.mjs @@ -63,6 +63,16 @@ async function resolveYouTubeChannelId(externalId) { const raw = String(externalId || '').trim(); if (/^UC[\w-]{20,}$/.test(raw)) return raw; const handle = raw.replace(/^@/, ''); + // 0) scrape sans clé (Step 17) : ne consomme aucun quota + try { + const { getSearchMode } = await import('./youtube-common.mjs'); + const mode = getSearchMode(); + if (mode !== 'api-only') { + const { resolveChannelIdViaScrape } = await import('./youtube-scrape.mjs'); + const id = await resolveChannelIdViaScrape(raw); + if (id && /^UC[\w-]{20,}$/.test(id)) return id; + } + } catch {} // 1) channels?forHandle (fonctionne encore pour beaucoup de chaînes) try { const data = await ytGet('channels', { part: 'id', forHandle: handle }); @@ -131,9 +141,32 @@ function ytTokenFor(key, page) { } async function ytContent(externalId, { type, page, limit, sort, q }) { - const channelId = await resolveYouTubeChannelId(externalId); const perPage = Math.min(Math.max(1, Number(limit || 24)), 50); const pageNum = Math.max(1, Number(page || 1)); + // Step 17 : scrape-first sans clé (0 quota). Fallback API si bot-check/timeout. + try { + const { getSearchMode } = await import('./youtube-common.mjs'); + const mode = getSearchMode(); + if (mode !== 'api-only') { + const { fetchChannelViaScrape } = await import('./youtube-scrape.mjs'); + // externalId brut (handle ou UC...) : le scrape gère les deux formes + const scraped = await fetchChannelViaScrape(externalId, { type, page: pageNum, limit: perPage }); + if (Array.isArray(scraped?.items) && scraped.items.length) { + if (mode === 'scrape-only') return { ...scraped, total: null }; + // scrape-first : retour direct si non vide + return { ...scraped, total: null }; + } + // vide -> on tente l'API (chaîne à faible volume ou tab non supporté en scrape) + if (mode === 'scrape-only') return { items: [], nextPage: null }; + } + } catch (e) { + console.warn('[channel-content/yt] scrape failed, fallback api:', e?.code || e?.message || e); + try { + const { getSearchMode } = await import('./youtube-common.mjs'); + if (getSearchMode() === 'scrape-only') return { items: [], nextPage: null }; + } catch {} + } + const channelId = await resolveYouTubeChannelId(externalId); if (type === 'playlists') { const key = ['pl', channelId, perPage].join('|'); const token = ytTokenFor(key, pageNum); diff --git a/server/providers/youtube-common.mjs b/server/providers/youtube-common.mjs new file mode 100644 index 0000000..795d979 --- /dev/null +++ b/server/providers/youtube-common.mjs @@ -0,0 +1,174 @@ +// Helpers partagés YouTube : clés API, mode de recherche, anti-ban yt-dlp, métriques. +// Extrait de youtube.mjs + index.mjs pour éviter la duplication (Step 17 P0). +import fs from 'node:fs'; +import path from 'node:path'; +import crypto from 'node:crypto'; +import { execFile as execFileCb } from 'node:child_process'; +import { promisify } from 'node:util'; + +export function getYouTubeKeys() { + const keys = []; + try { + const raw = process.env.YOUTUBE_API_KEYS; + if (raw && String(raw).trim() && !['undefined', 'null'].includes(String(raw).trim())) { + const s = String(raw).trim(); + if (s.startsWith('[')) { + try { + const arr = JSON.parse(s); + if (Array.isArray(arr)) keys.push(...arr.map((v) => String(v || '').trim()).filter(Boolean)); + } catch {} + } else { + keys.push(...s.split(',').map((v) => String(v || '').trim()).filter(Boolean)); + } + } + } catch {} + try { + const single = process.env.YOUTUBE_API_KEY; + if (single && String(single).trim()) keys.push(String(single).trim()); + } catch {} + return Array.from(new Set(keys.filter(Boolean))); +} + +export function isKeyFailure(status, data) { + try { + const reason = data?.error?.errors?.[0]?.reason || ''; + const message = String(data?.error?.message || ''); + if (status === 400 && (reason === 'API_KEY_INVALID' || /api key (expired|invalid)/i.test(message))) return true; + if (status === 403 && /quota|rateLimit|dailyLimit|userRateLimit/i.test(`${reason} ${message}`)) return true; + } catch {} + return false; +} + +/** Mode de recherche YT : innertube-first (défaut) | innertube-only | scrape-first | api-first | scrape-only | api-only */ +export const YT_SEARCH_MODES = ['innertube-first', 'innertube-only', 'scrape-first', 'api-first', 'scrape-only', 'api-only']; +export function getSearchMode() { + const m = String(process.env.YT_SEARCH_MODE || 'innertube-first').trim().toLowerCase(); + if (YT_SEARCH_MODES.includes(m)) return m; + return 'innertube-first'; +} + +export function getScrapeTtlMs() { + const v = Number(process.env.YT_SCRAPE_TTL_MS || 30 * 60 * 1000); + return Number.isFinite(v) && v > 0 ? v : 30 * 60 * 1000; +} + +export function getYtDlpTimeoutMs() { + const v = Number(process.env.YT_DLP_TIMEOUT_MS || 20000); + return Number.isFinite(v) && v >= 5000 ? v : 20000; +} + +export function getYtDlpBin() { + const p = String(process.env.YT_DLP_PATH || '').trim(); + if (p) return p; + return process.platform === 'win32' ? 'yt-dlp.exe' : 'yt-dlp'; +} + +/** + * Résolution robuste du binaire yt-dlp (même ordre que server/index.mjs) : + * 1. YT_DLP_PATH (si le fichier existe) + * 2. `yt-dlp[.exe]` sur le PATH (probé via --version) + * 3. binaire bundled de `youtube-dl-exec` (node_modules/.../bin/, utilisé en Docker) + * Résultat mis en cache process-wide. Throw avec code `yt_scrape_no_binary` si introuvable. + */ +let _resolvedBin = null; +export function resetYtDlpBinCache() { _resolvedBin = null; } + +function bundledYtDlpPath() { + try { + const exe = process.platform === 'win32' ? 'yt-dlp.exe' : 'yt-dlp'; + // server/providers/ -> server/../node_modules/youtube-dl-exec/bin/ + const cand = path.resolve(path.dirname(new URL(import.meta.url).pathname.replace(/^\/([A-Za-z]:)/, '$1')), '..', '..', 'node_modules', 'youtube-dl-exec', 'bin', exe); + if (fs.existsSync(cand)) return cand; + } catch {} + return null; +} + +export function getBundledYtDlpPath() { return bundledYtDlpPath(); } + +async function probeBin(bin) { + try { + const execFileAsync = promisify(execFileCb); + await execFileAsync(bin, ['--version'], { timeout: 10000 }); + return true; + } catch (e) { + // ENOENT = binaire absent ; autre erreur (timeout...) = présent mais KO -> on le garde quand même + return e?.code !== 'ENOENT' && e?.errno !== 'ENOENT'; + } +} + +export async function resolveYtDlpBin() { + if (_resolvedBin) return _resolvedBin; + // 1) YT_DLP_PATH explicite + try { + const p = String(process.env.YT_DLP_PATH || '').trim(); + if (p && fs.existsSync(p)) { + if (await probeBin(p)) { _resolvedBin = p; return p; } + } else if (p) { + console.warn(`[YT] YT_DLP_PATH introuvable : ${p} (fallback PATH/bundled)`); + } + } catch {} + // 2) PATH + const exe = process.platform === 'win32' ? 'yt-dlp.exe' : 'yt-dlp'; + for (const cand of [exe, 'yt-dlp']) { + try { + if (await probeBin(cand)) { _resolvedBin = cand; return cand; } + } catch {} + } + // 3) bundled youtube-dl-exec + const bundled = bundledYtDlpPath(); + if (bundled && await probeBin(bundled)) { _resolvedBin = bundled; return bundled; } + throw Object.assign( + new Error('yt-dlp introuvable (PATH, YT_DLP_PATH ni binaire bundled). Installez yt-dlp ou définissez YT_DLP_PATH.'), + { ytStatus: 503, code: 'yt_scrape_no_binary' }, + ); +} + +/** + * Args anti-ban communs pour yt-dlp (cookies, PO-Token, proxy). + * Ne jamais logger les valeurs (cookies path exclu des logs verbeux). + */ +export function buildYtDlpExtraArgs() { + const args = []; + try { + const cookies = String(process.env.YT_COOKIES_FILE || '').trim(); + if (cookies && fs.existsSync(cookies)) { + args.push('--cookies', cookies); + } + } catch {} + try { + const po = String(process.env.YT_PO_TOKEN || '').trim(); + if (po) args.push('--extractor-args', `youtube:po_token=${po}`); + } catch {} + try { + const proxy = String(process.env.YT_EGRESS_PROXY || '').trim(); + if (proxy) args.push('--proxy', proxy); + } catch {} + return args; +} + +export function hasCookiesFile() { + try { + const c = String(process.env.YT_COOKIES_FILE || '').trim(); + return Boolean(c && fs.existsSync(c)); + } catch { return false; } +} + +export function hashSearchKey(parts) { + return crypto.createHash('sha256').update(String(parts)).digest('hex').slice(0, 32); +} + +// Métriques process-wide (remises à zéro au restart, persistées jour par jour en SQLite via db.mjs) +export const ytMetrics = { + scrapeCalls: 0, + scrapeHits: 0, // hit cache (mémoire ou SQLite) + apiCalls: 0, + quotaUnits: 0, // estimation search*100 + videos*1 + fallbacks: 0, + scrapeErrors: 0, + innertubeCalls: 0, + innertubeErrors: 0, +}; + +export function metricsSnapshot() { + return { ...ytMetrics }; +} diff --git a/server/providers/youtube-innertube.mjs b/server/providers/youtube-innertube.mjs new file mode 100644 index 0000000..68c806a --- /dev/null +++ b/server/providers/youtube-innertube.mjs @@ -0,0 +1,331 @@ +// Couche InnerTube directe (façon SmartTube/MediaServiceCore) via youtubei.js. +// WEB client : search + continuations (pagination illimitée), watch-next (related), +// sans clé API ni quota. Le dispatcher bascule sur scrape/API si indisponible. +import { hashSearchKey, ytMetrics } from './youtube-common.mjs'; + +let sessionPromise = null; +let LogSilenced = false; + +async function silenceLibNoise() { + if (LogSilenced) return; + LogSilenced = true; + try { + const { Log } = await import('youtubei.js'); + // youtubei.js loggue en WARN chaque Text sans run assorti (bruit sur les titres) : + // on ne garde que les erreurs. + if (Log?.set_level && Log?.Level) { + const lvl = Log.Level.ERROR ?? Log.Level.WARNING ?? 3; + Log.set_level(lvl); + } + } catch {} +} + +/** Session InnerTube singleton (lazy). Throw code yt_innertube_unavailable si KO. */ +export async function getSession() { + if (!sessionPromise) { + sessionPromise = (async () => { + try { + await silenceLibNoise(); + const { Innertube } = await import('youtubei.js'); + const gl = String(process.env.YT_INNERTUBE_GL || 'FR').trim() || 'FR'; + const hl = String(process.env.YT_INNERTUBE_HL || 'fr').trim() || 'fr'; + return await Innertube.create({ lang: hl, location: gl }); + } catch (e) { + sessionPromise = null; // retry au prochain appel + throw Object.assign( + new Error(`InnerTube indisponible : ${String(e?.message || e).slice(0, 160)}`), + { ytStatus: 502, code: 'yt_innertube_unavailable' }, + ); + } + })(); + } + return sessionPromise; +} + +export function resetSession() { sessionPromise = null; } + +/** "81 973 vues" / "57K" / "1,2 M vues" -> number | undefined. */ +export function parseViewsText(t) { + try { + const s = String(t || '').trim(); + if (!s) return undefined; + const m = s.match(/([\d\s\u00a0.,]+)\s*([KMBkmb]|Mds|M|k|B)?/); + if (!m) return undefined; + const num = Number(m[1].replace(/[\s\u00a0]/g, '').replace(',', '.')); + if (!Number.isFinite(num)) return undefined; + const suffix = (m[2] || '').toLowerCase(); + const mult = suffix === 'b' ? 1e9 : suffix === 'm' || suffix === 'mds' ? 1e6 : suffix === 'k' ? 1e3 : 1; + const v = Math.round(num * mult); + return v > 0 ? v : undefined; + } catch { return undefined; } +} + +/** "2 minutes, 27 seconds" / "1 heure, 5 minutes" (label a11y) -> secondes. */ +export function parseDurationLabel(label) { + try { + const s = String(label || '').toLowerCase(); + if (!s) return undefined; + const get = (re) => { const m = s.match(re); return m ? Number(m[1]) : 0; }; + const h = get(/(\d+)\s*(?:hours?|heures?)/); + const mnt = get(/(\d+)\s*(?:minutes?)/); + const sec = get(/(\d+)\s*(?:seconds?|secondes?)/); + const total = h * 3600 + mnt * 60 + sec; + return total > 0 ? total : undefined; + } catch { return undefined; } +} + +function bestThumb(thumbs) { + try { + const arr = Array.isArray(thumbs) ? thumbs.filter((t) => t?.url) : []; + if (!arr.length) return undefined; + return arr[arr.length - 1].url; + } catch { return undefined; } +} + +/** + * Mappe un node InnerTube (Video, CompactVideo, GridVideo, LockupView, ReelItem, + * ShortsLockupView, PlaylistPanelVideo, WatchCardCompactVideo) vers Suggestion. + * Pur et testable offline. Les LockupView (nouveau renderer YouTube, utilisé + * notamment dans le watch-next façon SmartTube) ont une forme imbriquée propre. + */ +export function mapVideoNode(n) { + if (!n || typeof n !== 'object') return null; + try { + const type = String(n.type || ''); + if (type === 'LockupView') return mapLockupView(n); + if (/playlist|channel|gridchannel|shelf|radio|show|album/i.test(type) + && !/playlistpanelvideo|watchcard/i.test(type)) return null; + const id = n.id ? String(n.id) : null; + if (!id) return null; + const title = n.title?.text ?? (typeof n.title === 'string' ? n.title : ''); + if (!title) return null; + const duration = n.duration?.seconds != null ? Number(n.duration.seconds) : undefined; + const author = n.author || n.uploader || null; + const authorName = author?.name ?? (typeof author === 'string' ? author : undefined); + const authorId = author?.id ? String(author.id) : undefined; + const views = parseViewsText(n.view_count?.text ?? n.view_count ?? n.views?.text); + const isShort = /short|reel/i.test(type) || (Number.isFinite(duration) && duration > 0 && duration <= 70 && /short/i.test(title) === false && /reel|short/i.test(type)); + const badges = Array.isArray(n.badges) ? n.badges.map((b) => b?.label).filter(Boolean) : []; + return { + title: String(title), + id, + url: `https://www.youtube.com/watch?v=${id}`, + thumbnail: bestThumb(n.thumbnails), + uploaderName: authorName, + type: 'video', + ...(Number.isFinite(duration) && duration > 0 ? { duration } : {}), + ...(views !== undefined ? { views } : {}), + ...(n.published?.text ? { publishedAt: String(n.published.text) } : {}), + ...(authorId ? { channelId: authorId, channelExternalId: authorId, channelUrl: `https://www.youtube.com/channel/${authorId}` } : {}), + ...(authorName ? { channelHandle: String(authorName) } : {}), + ...(n.is_live ? { isLive: true } : {}), + ...(isShort ? { isShort: true } : {}), + ...(badges.length ? { badges } : {}), + }; + } catch { return null; } +} + +/** Mappe un LockupView (renderer moderne : watch-next, search, shelves). */ +export function mapLockupView(n) { + try { + const ct = String(n.content_type || '').toUpperCase(); + if (ct && !['VIDEO', 'SHORT', 'MOVIE', 'LIVE'].includes(ct)) return null; + const id = n.content_id ? String(n.content_id) + : (n.renderer_context?.command_context?.on_tap?.payload?.videoId || null); + if (!id) return null; + const meta = n.metadata || {}; + const title = meta.title?.text ?? (typeof meta.title === 'string' ? meta.title : ''); + if (!title) return null; + // Vignette : content_image.image[] (prend la plus large) + let thumbnail; + try { + const imgs = (n.content_image?.image || []).filter((i) => i?.url); + thumbnail = imgs.sort((a, b) => (b.width || 0) - (a.width || 0))[0]?.url; + } catch {} + // Chaîne : 1ère ligne des metadata_rows, sinon label a11y "Go to channel X" + let uploaderName; + try { + const rows = meta.metadata?.metadata_rows || []; + const first = rows[0]?.metadata_parts?.[0]?.text; + uploaderName = first?.text ?? (typeof first === 'string' ? first : undefined); + } catch {} + if (!uploaderName) { + const a11y = String(meta.image?.a11y_label || ''); + const m = a11y.match(/^(?:go to channel|aller sur la cha[îi]ne)\s+(.+)$/i); + if (m) uploaderName = m[1].trim(); + } + // Vues : 2e ligne ("57K", "1,2 M vues"...), durée : label a11y + let views; + try { + const rows = meta.metadata?.metadata_rows || []; + const second = rows[1]?.metadata_parts?.[0]?.text; + views = parseViewsText(second?.text ?? second); + } catch {} + const duration = parseDurationLabel(n.renderer_context?.accessibility_context?.label); + return { + title: String(title), + id, + url: `https://www.youtube.com/watch?v=${id}`, + thumbnail, + uploaderName, + type: 'video', + ...(duration !== undefined ? { duration } : {}), + ...(views !== undefined ? { views } : {}), + ...(uploaderName ? { channelHandle: String(uploaderName) } : {}), + ...(ct === 'SHORT' ? { isShort: true } : {}), + ...(ct === 'LIVE' ? { isLive: true } : {}), + }; + } catch { return null; } +} + +export function mapNodes(nodes) { + const seen = new Set(); + const out = []; + for (const n of nodes || []) { + const m = mapVideoNode(n); + if (!m || seen.has(m.id)) continue; + seen.add(m.id); + out.push(m); + } + return out; +} + +// Chaîne de continuations en mémoire : q_hash -> { feed, pages }. +// (SQLite persiste les pages déjà servies ; la chaîne évite de rejouer les pages 1..N-1.) +const chainCache = new Map(); +const CHAIN_MAX = 50; +function chainGet(k) { const h = chainCache.get(k); if (h) { chainCache.delete(k); chainCache.set(k, h); } return h || null; } +function chainSet(k, v) { + if (chainCache.has(k)) chainCache.delete(k); + chainCache.set(k, v); + while (chainCache.size > CHAIN_MAX) { const o = chainCache.keys().next().value; if (o === undefined) break; chainCache.delete(o); } +} +export function innertubeChainStats() { return { chains: chainCache.size, max: CHAIN_MAX }; } + +function sortParam(sort) { + const s = String(sort || 'relevance').toLowerCase(); + if (s === 'date') return 'upload_date'; + if (s === 'views') return 'view_count'; + return 'relevance'; +} + +/** + * Recherche InnerTube avec vraie pagination (continuations). + * Page 1 ~35-40 vidéos ; pages suivantes via getContinuation() fusionnées (façon SmartTube). + */ +export async function searchViaInnerTube(q, opts = {}) { + const query = String(q || '').trim(); + if (query.length < 2) return []; + const limit = Math.min(50, Math.max(1, Number(opts?.limit || 24))); + const page = Math.min(10, Math.max(1, Number(opts?.page || 1))); + const sort = String(opts?.sort || 'relevance'); + ytMetrics.innertubeCalls = (ytMetrics.innertubeCalls || 0) + 1; + const yt = await getSession(); + const key = `it|${hashSearchKey(`${query.toLowerCase()}|${sort}`)}`; + let entry = chainGet(key); + let feed = entry?.feed || null; + let collected = entry?.items ? [...entry.items] : []; + let pagesDone = entry?.pages || 0; + // Note : une "page" InnerTube fait ~17-20 vidéos quel que soit `limit`. + // On charge donc des continuations jusqu'à couvrir la fenêtre demandée + // [start, start+limit[ (façon SmartTube qui remplit son écran au fil des + // continuations), au lieu d'aligner 1 page API = 1 page UI. + const need = page * limit; + try { + if (!feed) { + feed = await yt.search(query, { sort_by: sortParam(sort) }); + collected = mapNodes(feed.videos); + pagesDone = 1; + } + let guard = 0; + while (collected.length < need && feed?.has_continuation && guard < 12) { + guard++; + feed = await feed.getContinuation(); + const fresh = mapNodes(feed.videos).filter((m) => !collected.some((c) => c.id === m.id)); + collected = collected.concat(fresh); + pagesDone++; + if (!fresh.length) break; + } + } catch (e) { + if (e?.code === 'yt_innertube_unavailable') throw e; + throw Object.assign(new Error(`InnerTube search failed: ${String(e?.message || e).slice(0, 160)}`), { ytStatus: 502, code: 'yt_innertube_failed' }); + } + if (feed) chainSet(key, { feed, items: collected, pages: pagesDone }); + const start = (page - 1) * limit; + return collected.slice(start, start + limit); +} + +/** + * Vidéos connexes façon SmartTube (endpoint watch-next) : ce qui alimente + * la colonne "À suivre" sous le lecteur. + */ +export async function getRelatedViaInnerTube(videoId, limit = 24) { + const id = String(videoId || '').trim(); + if (!id) return []; + const n = Math.min(50, Math.max(1, Number(limit || 24))); + ytMetrics.innertubeCalls = (ytMetrics.innertubeCalls || 0) + 1; + const yt = await getSession(); + try { + const info = await yt.getInfo(id); + const feed = info?.watch_next_feed; + const arr = Array.isArray(feed) ? feed : (feed ? Array.from(feed) : []); + return mapNodes(arr).filter((m) => m.id !== id).slice(0, n); + } catch (e) { + if (e?.code === 'yt_innertube_unavailable') throw e; + throw Object.assign(new Error(`InnerTube related failed: ${String(e?.message || e).slice(0, 160)}`), { ytStatus: 502, code: 'yt_innertube_failed' }); + } +} + +// -------------------- Transcripts via InnerTube -------------------- +// `getInfo().captions.caption_tracks[]` expose les mêmes URLs timedtext que +// yt-dlp découvre via un dump complet (`subtitles`/`automatic_captions`). +// On adapte leur forme vers le format yt-dlp pour réutiliser pickTrack(), +// orderedTracks() et parseTrackText() de transcript.mjs sans les toucher. + +function normCaptionLang(code) { + return String(code || '').trim().toLowerCase().replace(/_/g, '-'); +} + +/** + * Adapte des caption tracks InnerTube brutes vers un pseudo dump yt-dlp + * `{ subtitles, automatic_captions }` (kind 'asr' = auto-généré). + * Pur et testable offline. + */ +export function mapCaptionTracks(captionTracks) { + const subtitles = {}; + const automatic_captions = {}; + for (const t of captionTracks || []) { + const base = String(t?.base_url || ''); + if (!base) continue; + const lang = normCaptionLang(t?.language_code) || 'und'; + const name = t?.name?.text ?? (typeof t?.name === 'string' ? t.name : lang); + const sep = base.includes('?') ? '&' : '?'; + const track = { url: `${base}${sep}fmt=json3`, ext: 'json3', name: String(name) }; + const dict = t?.kind === 'asr' ? automatic_captions : subtitles; + if (!Array.isArray(dict[lang])) dict[lang] = []; + if (!dict[lang].some((x) => x.url === track.url)) dict[lang].push(track); + } + const languages = Array.from(new Set([...Object.keys(subtitles), ...Object.keys(automatic_captions)])); + const trackCount = Object.values(subtitles).concat(Object.values(automatic_captions)) + .reduce((n, arr) => n + arr.length, 0); + return { subtitles, automatic_captions, languages, trackCount }; +} + +/** + * Découvre les pistes de sous-titres d'une vidéo via InnerTube (0 quota, + * pas de spawn yt-dlp). Retourne un pseudo dump yt-dlp prêt pour pickTrack(). + */ +export async function getCaptionTracksViaInnerTube(videoId) { + const id = String(videoId || '').trim(); + if (!id) return { subtitles: {}, automatic_captions: {}, languages: [], trackCount: 0 }; + ytMetrics.innertubeCalls = (ytMetrics.innertubeCalls || 0) + 1; + const yt = await getSession(); + try { + const info = await yt.getInfo(id); + const raw = info?.captions?.caption_tracks || []; + return mapCaptionTracks(Array.isArray(raw) ? raw : Array.from(raw)); + } catch (e) { + if (e?.code === 'yt_innertube_unavailable') throw e; + throw Object.assign(new Error(`InnerTube captions failed: ${String(e?.message || e).slice(0, 160)}`), { ytStatus: 502, code: 'yt_innertube_failed' }); + } +} diff --git a/server/providers/youtube-scrape.mjs b/server/providers/youtube-scrape.mjs new file mode 100644 index 0000000..cf8ea70 --- /dev/null +++ b/server/providers/youtube-scrape.mjs @@ -0,0 +1,197 @@ +// Recherche / channel / trending YouTube SANS clé API via yt-dlp (Step 17 P1). +// Binaire résolu via YT_DLP_PATH > PATH. Jamais de secret en log. +import { execFile as execFileCb } from 'node:child_process'; +import { promisify } from 'node:util'; +import { getYtDlpTimeoutMs, buildYtDlpExtraArgs, resolveYtDlpBin } from './youtube-common.mjs'; + +const execFileAsync = promisify(execFileCb); + +/** + * Normalise une entrée yt-dlp flat-playlist vers Suggestion (même contrat que youtube.mjs). + * @param {any} e entrée yt-dlp + */ +export function mapFlatEntry(e) { + if (!e || typeof e !== 'object') return null; + const id = e.id || e.display_id || null; + if (!id) return null; + const thumbs = Array.isArray(e.thumbnails) ? e.thumbnails : []; + const thumb = thumbs.length + ? (thumbs.find((t) => t?.url && (t.height || 0) >= 360)?.url || thumbs[thumbs.length - 1]?.url) + : (e.thumbnail || undefined); + const duration = e.duration != null ? Number(e.duration) : undefined; + const views = e.view_count != null ? Number(e.view_count) : (e.views != null ? Number(e.views) : undefined); + const channelId = e.channel_id || e.uploader_id || undefined; + const uploaderName = e.channel || e.uploader || undefined; + return { + title: e.title || '', + id: String(id), + url: String(id).startsWith('http') ? String(id) : `https://www.youtube.com/watch?v=${id}`, + thumbnail: thumb, + uploaderName, + type: 'video', + ...(Number.isFinite(duration) && duration > 0 ? { duration } : {}), + ...(Number.isFinite(views) ? { views } : {}), + ...(e.timestamp ? { publishedAt: new Date(Number(e.timestamp) * 1000).toISOString() } : {}), + ...(channelId ? { channelId: String(channelId), channelExternalId: String(channelId), channelUrl: `https://www.youtube.com/channel/${channelId}` } : {}), + ...(uploaderName ? { channelHandle: String(uploaderName) } : {}), + }; +} + +/** Parse le JSON de `yt-dlp --dump-single-json --flat-playlist` (objet ou NDJSON fog). */ +export function parseFlatPlaylistJson(raw) { + try { + const text = String(raw || '').trim(); + if (!text) return []; + // Cas standard : un seul objet JSON avec .entries + try { + const obj = JSON.parse(text); + const entries = Array.isArray(obj?.entries) ? obj.entries : (Array.isArray(obj) ? obj : []); + return entries.map(mapFlatEntry).filter(Boolean); + } catch { + // Fallback NDJSON (une ligne = un JSON) + const out = []; + for (const line of text.split('\n')) { + const t = line.trim(); + if (!t) continue; + try { + const o = JSON.parse(t); + const m = mapFlatEntry(o); + if (m) out.push(m); + } catch {} + } + return out; + } + } catch { return []; } +} + +export function classifyScrapeError(e) { + if (e?.code === 'yt_scrape_no_binary') return e; + const msg = String(e?.message || e || ''); + if (/ENOENT/i.test(msg) || e?.code === 'ENOENT' || e?.errno === 'ENOENT') { + return Object.assign( + new Error('yt-dlp introuvable (ni PATH ni bundled). Installez yt-dlp ou définissez YT_DLP_PATH.'), + { ytStatus: 503, code: 'yt_scrape_no_binary' }, + ); + } + if (/bot|sign in to confirm|challenge|cookies|login/i.test(msg)) { + return Object.assign(new Error('YouTube bot-check (cookies/PO-Token requis)'), { ytStatus: 502, code: 'yt_scrape_bot_check' }); + } + if (/timed out|timeout|ETIMEDOUT|killed|SIGKILL/i.test(msg)) { + return Object.assign(new Error('YouTube scrape timeout'), { ytStatus: 504, code: 'yt_scrape_timeout' }); + } + if (/unable to|unsupported|not available|private|deleted/i.test(msg)) { + return Object.assign(new Error(`YouTube scrape upstream: ${msg.slice(0, 160)}`), { ytStatus: 502, code: 'yt_scrape_upstream' }); + } + return Object.assign(new Error(`YouTube scrape failed: ${msg.slice(0, 200)}`), { ytStatus: 502, code: 'yt_scrape_failed' }); +} + +async function runYtDlpFlat(queryOrUrl, { limit = 10, playlistStart = 1 } = {}) { + const bin = await resolveYtDlpBin(); + const timeout = getYtDlpTimeoutMs(); + const end = playlistStart + Math.max(1, Math.min(50, Number(limit || 10))) - 1; + const args = [ + '--dump-single-json', + '--flat-playlist', + '--no-warnings', + '--no-check-certificates', + '--skip-download', + '--no-playlist', + '--playlist-start', String(playlistStart), + '--playlist-end', String(end), + ...buildYtDlpExtraArgs(), + queryOrUrl, + ]; + try { + const { stdout } = await execFileAsync(bin, args, { timeout, maxBuffer: 16 * 1024 * 1024 }); + return parseFlatPlaylistJson(stdout); + } catch (e) { + // yt-dlp peut écrire du JSON partiel sur stdout même en exit non-zero + const partial = e?.stdout ? parseFlatPlaylistJson(String(e.stdout)) : []; + if (partial.length) return partial; + throw classifyScrapeError(e); + } +} + +/** Recherche sans clé. sort: relevance|date|views */ +export async function searchViaScrape(q, opts = {}) { + const query = String(q || '').trim(); + if (query.length < 2) return []; + const limit = Math.min(50, Math.max(1, Number(opts?.limit || 10))); + const page = Math.max(1, Number(opts?.page || 1)); + const sort = String(opts?.sort || 'relevance').toLowerCase(); + const perPage = limit; + const start = (page - 1) * perPage + 1; + let prefix = 'ytsearch'; + if (sort === 'date') prefix = 'ytsearchdate'; + // views : pas de préfixe natif stable -> ytsearch + tri local + const target = `${prefix}${perPage}:${query}`; + let items = await runYtDlpFlat(target, { limit: perPage, playlistStart: start }); + if (sort === 'views') items = [...items].sort((a, b) => (b.views || 0) - (a.views || 0)); + return items.slice(0, perPage); +} + +const CHANNEL_TABS = { videos: 'videos', shorts: 'shorts', streams: 'streams', live: 'streams', playlists: 'playlists' }; + +/** Contenu chaîne sans clé. type: videos|shorts|playlists|live */ +export async function fetchChannelViaScrape(externalId, { type = 'videos', page = 1, limit = 24 } = {}) { + const raw = String(externalId || '').trim().replace(/^@/, ''); + if (!raw) return { items: [], nextPage: null }; + const tab = CHANNEL_TABS[String(type)] || 'videos'; + const perPage = Math.min(50, Math.max(1, Number(limit || 24))); + const pageNum = Math.max(1, Number(page || 1)); + const start = (pageNum - 1) * perPage + 1; + // UC... -> /channel/UC.../tab, sinon -> /@/handle/tab + const base = /^UC[\w-]{20,}$/.test(raw) + ? `https://www.youtube.com/channel/${raw}/${tab}` + : `https://www.youtube.com/@${raw}/${tab}`; + const items = await runYtDlpFlat(base, { limit: perPage, playlistStart: start }); + // playlists renvoient des ids PL... : on les garde tels quels avec un shape playlist + if (tab === 'playlists') { + const mapped = items.map((s) => ({ id: s.id, title: s.title, thumbnail: s.thumbnail, videoCount: null, updatedAt: s.publishedAt || null })); + return { items: mapped, nextPage: items.length >= perPage ? pageNum + 1 : null }; + } + let filtered = items; + if (type === 'shorts') filtered = items.filter((s) => !s.duration || s.duration <= 70); + return { items: filtered, nextPage: items.length >= perPage ? pageNum + 1 : null }; +} + +/** Trending sans clé. Les onglets /feed/trending ont été retirés côté YouTube + * (redirect home -> erreur tab) ; on tente les tabs puis repli ytsearch trié par vues. */ +export async function getTrendingViaScrape(limit = 24) { + const n = Math.min(50, Math.max(1, Number(limit || 24))); + const tabs = [ + 'https://www.youtube.com/feed/trending', + 'https://www.youtube.com/trending', + ]; + for (const url of tabs) { + try { + const items = await runYtDlpFlat(url, { limit: n, playlistStart: 1 }); + if (items.length) return items; + } catch {} + } + // Repli : recherche générique triée par vues (0 quota, toujours disponible) + return runYtDlpFlat(`ytsearch${n}:top trending videos world`, { limit: n, playlistStart: 1 }); +} + +/** Résout un channel_id via scrape (imprime channel_id sans appel API). */ +export async function resolveChannelIdViaScrape(externalId) { + const raw = String(externalId || '').trim(); + if (/^UC[\w-]{20,}$/.test(raw)) return raw; + const handle = raw.replace(/^@/, ''); + const bin = await resolveYtDlpBin().catch(() => null); + if (!bin) { + throw Object.assign( + new Error('yt-dlp introuvable (ni PATH ni bundled). Installez yt-dlp ou définissez YT_DLP_PATH.'), + { ytStatus: 503, code: 'yt_scrape_no_binary' }, + ); + } + const timeout = Math.min(15000, getYtDlpTimeoutMs()); + try { + const { stdout } = await execFileAsync(bin, + ['--print', '%(channel_id)s', '--no-warnings', '--skip-download', '--playlist-end', '1', ...buildYtDlpExtraArgs(), `https://www.youtube.com/@${handle}/videos`], + { timeout }); + const id = String(stdout || '').trim().split('\n')[0].trim(); + if (/^UC[\w-]{20,}$/.test(id)) return id; + } catch {} + return raw; +} diff --git a/server/providers/youtube.mjs b/server/providers/youtube.mjs index 6a97273..5775a46 100644 --- a/server/providers/youtube.mjs +++ b/server/providers/youtube.mjs @@ -9,6 +9,12 @@ * @property {string=} uploaderName * @property {string=} type */ +import { + getYouTubeKeys, isKeyFailure, getSearchMode, getScrapeTtlMs, + hashSearchKey, ytMetrics, +} from './youtube-common.mjs'; +import { searchViaScrape } from './youtube-scrape.mjs'; +import { searchViaInnerTube } from './youtube-innertube.mjs'; function parseISODurationToSeconds(iso) { if (typeof iso !== 'string' || !iso) return 0; @@ -20,50 +26,6 @@ function parseISODurationToSeconds(iso) { return (hours * 3600) + (minutes * 60) + seconds; } -/** - * Resolve configured YouTube API keys. - * Accepts YOUTUBE_API_KEYS as JSON array ('["k1","k2"]') or CSV ('k1,k2'), - * plus the legacy single YOUTUBE_API_KEY as fallback. De-duplicated. - * @returns {string[]} - */ -function getYouTubeKeys() { - const keys = []; - try { - const raw = process.env.YOUTUBE_API_KEYS; - if (raw && String(raw).trim() && String(raw).trim() !== 'undefined' && String(raw).trim() !== 'null') { - const s = String(raw).trim(); - if (s.startsWith('[')) { - try { - const arr = JSON.parse(s); - if (Array.isArray(arr)) keys.push(...arr.map(v => String(v || '').trim()).filter(Boolean)); - } catch {} - } else { - keys.push(...s.split(',').map(v => String(v || '').trim()).filter(Boolean)); - } - } - } catch {} - try { - const single = process.env.YOUTUBE_API_KEY; - if (single && String(single).trim()) keys.push(String(single).trim()); - } catch {} - return Array.from(new Set(keys.filter(Boolean))); -} - -/** - * Classify a YouTube API failure as retryable with another key. - * - 400 API_KEY_INVALID / "API key expired" -> key is dead, try next. - * - 403 quotaExceeded / rateLimitExceeded / dailyLimitExceeded -> quota, try next. - */ -function isKeyFailure(status, data) { - try { - const reason = data?.error?.errors?.[0]?.reason || ''; - const message = String(data?.error?.message || ''); - if (status === 400 && (reason === 'API_KEY_INVALID' || /api key (expired|invalid)/i.test(message))) return true; - if (status === 403 && /quota|rateLimit|dailyLimit|userRateLimit/i.test(reason + ' ' + message)) return true; - } catch {} - return false; -} - /** GET JSON from YouTube, rotating through keys on key failures. */ async function ytFetchJson(base, paramsWithoutKey) { const keys = getYouTubeKeys(); @@ -76,13 +38,19 @@ async function ytFetchJson(base, paramsWithoutKey) { for (const key of keys) { const params = new URLSearchParams(paramsWithoutKey); params.set('key', key); + ytMetrics.apiCalls++; + try { + const { incYoutubeMetrics } = await import('../db.mjs').catch(() => ({})); + if (String(base).includes('/search')) incYoutubeMetrics?.({ apiCalls: 1, quotaUnits: 100 }); + else incYoutubeMetrics?.({ apiCalls: 1, quotaUnits: 1 }); + ytMetrics.quotaUnits += String(base).includes('/search') ? 100 : 1; + } catch {} const resp = await fetch(`${base}?${params.toString()}`); const data = await resp.json().catch(() => ({})); if (resp.ok) return data; lastStatus = resp.status; lastData = data; lastError = new Error(`YouTube API error: ${resp.status} ${data?.error?.message || ''}`.trim()); - // Only rotate to the next key on key-attributable failures; otherwise fail fast. if (!isKeyFailure(resp.status, data)) break; console.warn(`[YouTube] key ...${String(key).slice(-4)} failed (${resp.status}), trying next key`); } @@ -92,121 +60,278 @@ async function ytFetchJson(base, paramsWithoutKey) { throw err; } +/** + * Parse a `suggestqueries.google.com` (client=youtube) response body into plain strings. + * The endpoint answers either bare JSON (`["q",["s1","s2"],...]`) or wrapped in + * `window.google.ac.h(...)`. Never throws: unparsable bodies yield []. + * @param {string} text + * @returns {string[]} + */ +export function parseYoutubeSuggestResponse(text) { + try { + const raw = String(text || '').trim(); + if (!raw) return []; + let payload = raw; + const start = raw.indexOf('('); + const end = raw.lastIndexOf(')'); + if (start !== -1 && end > start) payload = raw.slice(start + 1, end); + const parsed = JSON.parse(payload); + const candidates = Array.isArray(parsed) && Array.isArray(parsed[1]) ? parsed[1] : []; + return candidates + .map((s) => { + if (Array.isArray(s)) s = s[0]; + return (typeof s === 'string' ? s : String(s ?? '')).trim(); + }) + .filter(Boolean) + .slice(0, 20); + } catch { + return []; + } +} + +// Mémoire LRU process-wide pour le dispatcher (complète SQLite) +const memCache = new Map(); +const MEM_MAX = Number(process.env.YT_SCRAPE_MEM_MAX || 300); +function memGet(k, ttl) { + const hit = memCache.get(k); + if (!hit) return null; + if ((Date.now() - hit.ts) >= ttl) { memCache.delete(k); return null; } + memCache.delete(k); memCache.set(k, hit); + return hit.items; +} +function memSet(k, items) { + if (memCache.has(k)) memCache.delete(k); + memCache.set(k, { ts: Date.now(), items }); + while (memCache.size > MEM_MAX) { const o = memCache.keys().next().value; if (o === undefined) break; memCache.delete(o); } +} +export function ytScrapeCacheStats() { + return { memEntries: memCache.size, memMax: MEM_MAX }; +} + +async function searchViaApi(q, { limit = 10, page = 1, sort = 'relevance' } = {}) { + const keys = getYouTubeKeys(); + if (!keys.length) { + throw Object.assign(new Error('YOUTUBE_API_KEY not configured'), { ytStatus: 503, code: 'youtube_api_key_unavailable' }); + } + let order = 'relevance'; + if (sort === 'date') order = 'date'; + else if (sort === 'views') order = 'viewCount'; + const perPage = Math.min(Math.max(1, Number(limit || 10)), 50); + const targetPage = Math.max(1, Number(page || 1)); + let pageToken = ''; + let currentPage = 1; + let lastItems = []; + while (currentPage <= targetPage) { + const params = { + part: 'snippet', q, type: 'video', maxResults: String(perPage), order, + videoEmbeddable: 'true', safeSearch: 'moderate', + }; + if (pageToken) params.pageToken = pageToken; + const data = await ytFetchJson('https://www.googleapis.com/youtube/v3/search', params); + if (currentPage === targetPage) { lastItems = Array.isArray(data.items) ? data.items : []; break; } + const next = data.nextPageToken; + if (!next) { lastItems = []; break; } + pageToken = String(next); + currentPage++; + } + const videoIds = (lastItems || []).map((item) => item?.id?.videoId).filter(Boolean); + const detailsMap = new Map(); + if (videoIds.length > 0) { + try { + const detailsData = await ytFetchJson('https://www.googleapis.com/youtube/v3/videos', { + part: 'contentDetails,statistics,status', id: videoIds.join(','), + }); + for (const vid of detailsData?.items || []) if (vid?.id) detailsMap.set(vid.id, vid); + } catch (e) { + console.warn('[YouTube] details fetch failed, continuing without durations:', e?.message || e); + } + } + return (lastItems || []).map((item) => { + const videoId = item?.id?.videoId; + const snippet = item?.snippet || {}; + const thumb = snippet.thumbnails?.high?.url || snippet.thumbnails?.medium?.url || snippet.thumbnails?.default?.url || undefined; + const details = videoId ? detailsMap.get(videoId) : null; + const duration = parseISODurationToSeconds(details?.contentDetails?.duration || ''); + const views = details?.statistics?.viewCount != null ? Number(details.statistics.viewCount) : undefined; + const channelId = snippet.channelId || undefined; + const embeddable = details?.status ? details.status.embeddable !== false : undefined; + return { + title: snippet.title || '', id: videoId, + url: videoId ? `https://www.youtube.com/watch?v=${videoId}` : undefined, + thumbnail: thumb, uploaderName: snippet.channelTitle || undefined, type: 'video', + duration: duration > 0 ? duration : undefined, views, + publishedAt: snippet.publishedAt || undefined, channelId, + channelHandle: snippet.channelTitle || undefined, channelExternalId: channelId, + channelUrl: channelId ? `https://www.youtube.com/channel/${channelId}` : undefined, embeddable, + }; + }); +} + /** @type {{ id: 'yt', label: string, search: (q: string, opts: { limit: number, page?: number, sort?: 'relevance'|'date'|'views' }) => Promise }} */ const handler = { id: 'yt', label: 'YouTube', async search(q, opts) { const { limit = 10, page = 1, sort = 'relevance' } = opts || {}; + const mode = getSearchMode(); + const ttl = getScrapeTtlMs(); + const perPage = Math.min(Math.max(1, Number(limit || 10)), 50); + const key = `yt|${hashSearchKey(`${String(q).toLowerCase().trim()}|${perPage}|${page}|${sort}|${mode}`)}`; + // 1) mémoire + const memHit = memGet(key, ttl); + if (memHit) { ytMetrics.scrapeHits++; return memHit; } + // 2) SQLite try { - const keys = getYouTubeKeys(); - if (!keys.length) { - throw Object.assign(new Error('YOUTUBE_API_KEY not configured'), { ytStatus: 503, code: 'youtube_api_key_unavailable' }); + const { getCachedYoutubeSearch } = await import('../db.mjs'); + const cached = getCachedYoutubeSearch(key); + if (cached?.items) { + ytMetrics.scrapeHits++; + memSet(key, cached.items); + return cached.items; } - - let order = 'relevance'; - if (sort === 'date') order = 'date'; - else if (sort === 'views') order = 'viewCount'; - - // Iterate nextPageToken to reach the requested page (1-based) - const perPage = Math.min(Math.max(1, Number(limit || 10)), 50); - const targetPage = Math.max(1, Number(page || 1)); - let pageToken = ''; - let currentPage = 1; - let lastItems = []; - - while (currentPage <= targetPage) { - const params = { - part: 'snippet', - q: q, - type: 'video', - maxResults: String(perPage), - order, - // Ne retourner que des vidéos lisibles en embed (évite l'erreur 153 côté player). - // Les vidéos non-embeddables sont filtrées via le champ status ci-dessous pour /videos. - videoEmbeddable: 'true', - safeSearch: 'moderate', - }; - if (pageToken) params.pageToken = pageToken; - - const data = await ytFetchJson('https://www.googleapis.com/youtube/v3/search', params); - - if (currentPage === targetPage) { - lastItems = Array.isArray(data.items) ? data.items : []; - break; - } - - // Prepare for next iteration - const next = data.nextPageToken; - if (!next) { - // No more pages; requested page beyond available results - lastItems = []; - break; - } - pageToken = String(next); - currentPage++; + } catch {} + const persist = async (items, source) => { + // Ne jamais persister un résultat vide : une page vide transitoire + // (continuation expirée, raté réseau partiel) ne doit pas empoisonner + // le cache et bloquer les requêtes suivantes. + if (!Array.isArray(items) || items.length === 0) return items; + memSet(key, items); + try { + const { setCachedYoutubeSearch, incYoutubeMetrics } = await import('../db.mjs'); + setCachedYoutubeSearch(key, q, items, source, ttl); + if (source === 'scrape') incYoutubeMetrics({ scrapeCalls: 1 }); + } catch {} + return items; + }; + const t0 = Date.now(); + const tryScrape = async () => { + ytMetrics.scrapeCalls++; + try { const { incYoutubeMetrics } = await import('../db.mjs'); incYoutubeMetrics({ scrapeCalls: 1 }); } catch {} + return searchViaScrape(q, { limit: perPage, page, sort }); + }; + const tryApi = () => searchViaApi(q, { limit: perPage, page, sort }); + const tryInnerTube = () => searchViaInnerTube(q, { limit: perPage, page, sort }); + const log = (source, extra = '') => console.log(`[YT search] source=${source} mode=${mode} latency=${Date.now() - t0}ms results=${extra}`); + try { + if (mode === 'api-only') { + const items = await tryApi(); + log('api', items.length); + return persist(items, 'api'); } - - const videoIds = (lastItems || []) - .map(item => item?.id?.videoId) - .filter(Boolean); - - const detailsMap = new Map(); - if (videoIds.length > 0) { + if (mode === 'scrape-only') { + const items = await tryScrape(); + log('scrape', items.length); + return persist(items, 'scrape'); + } + if (mode === 'innertube-only') { + const items = await tryInnerTube(); + log('innertube', items.length); + return persist(items, 'innertube'); + } + if (mode === 'api-first') { try { - const detailsData = await ytFetchJson('https://www.googleapis.com/youtube/v3/videos', { - part: 'contentDetails,statistics,status', - id: videoIds.join(','), - }); - for (const vid of detailsData?.items || []) { - if (vid?.id) detailsMap.set(vid.id, vid); - } - } catch (e) { - console.warn('[YouTube] details fetch failed, continuing without durations:', e?.message || e); + const items = await tryApi(); + log('api', items.length); + return persist(items, 'api'); + } catch (apiErr) { + ytMetrics.fallbacks++; + console.warn(`[YT search] api-first fallback to scrape: ${apiErr?.message || apiErr}`); + const items = await tryScrape(); + log('scrape(fallback)', items.length); + return persist(items, 'scrape'); } } - - return (lastItems || []).map(item => { - const videoId = item?.id?.videoId; - const snippet = item?.snippet || {}; - const thumb = snippet.thumbnails?.high?.url - || snippet.thumbnails?.medium?.url - || snippet.thumbnails?.default?.url - || undefined; - - const details = videoId ? detailsMap.get(videoId) : null; - const isoDuration = details?.contentDetails?.duration || ''; - const duration = parseISODurationToSeconds(isoDuration); - const views = details?.statistics?.viewCount != null ? Number(details.statistics.viewCount) : undefined; - const channelId = snippet.channelId || undefined; - const channelHandle = snippet.channelTitle || undefined; - // status.embeddable === false -> le player renvoie l'erreur 153 ("Video configuration error"). - const embeddable = details?.status ? details.status.embeddable !== false : undefined; - - return { - title: snippet.title || '', - id: videoId, - url: videoId ? `https://www.youtube.com/watch?v=${videoId}` : undefined, - thumbnail: thumb, - uploaderName: snippet.channelTitle || undefined, - type: 'video', - duration: duration > 0 ? duration : undefined, - views, - publishedAt: snippet.publishedAt || undefined, - channelId, - channelHandle, - channelExternalId: channelId, - channelUrl: channelId ? `https://www.youtube.com/channel/${channelId}` : undefined, - embeddable, - }; - }); + if (mode === 'scrape-first') { + // scrape -> API (comportement historique, sans InnerTube) + try { + const items = await tryScrape(); + log('scrape', items.length); + return persist(items, 'scrape'); + } catch (scrapeErr) { + ytMetrics.scrapeErrors++; + ytMetrics.fallbacks++; + console.warn(`[YT search] scrape failed (${scrapeErr?.code || 'unknown'}), fallback to api`); + try { + const items = await tryApi(); + log('api(fallback)', items.length); + return persist(items, 'api'); + } catch (apiErr) { + const noBinary = scrapeErr?.code === 'yt_scrape_no_binary'; + const noKey = apiErr?.code === 'youtube_api_key_unavailable'; + if (noBinary && noKey) { + const err = new Error('YouTube indisponible : yt-dlp introuvable (définissez YT_DLP_PATH) et aucune clé YOUTUBE_API_KEY configurée.'); + err.ytStatus = 503; err.code = 'youtube_no_source'; + throw err; + } + console.error(`[YT search] both sources failed. scrape=${scrapeErr?.message} api=${apiErr?.message}`); + throw scrapeErr; + } + } + } + // innertube-first (défaut) : InnerTube -> scrape yt-dlp -> API officielle. + // Chaque couche ne fait jamais échouer la recherche à elle seule. + const errors = []; + try { + const items = await tryInnerTube(); + log('innertube', items.length); + return persist(items, 'innertube'); + } catch (itErr) { + ytMetrics.innertubeErrors = (ytMetrics.innertubeErrors || 0) + 1; + ytMetrics.fallbacks++; + errors.push(`innertube=${itErr?.message}`); + console.warn(`[YT search] innertube failed (${itErr?.code || 'unknown'}), fallback to scrape`); + } + try { + const items = await tryScrape(); + log('scrape(fallback)', items.length); + return persist(items, 'scrape'); + } catch (scrapeErr) { + ytMetrics.scrapeErrors++; + ytMetrics.fallbacks++; + errors.push(`scrape=${scrapeErr?.message}`); + console.warn(`[YT search] scrape failed (${scrapeErr?.code || 'unknown'}), fallback to api`); + } + try { + const items = await tryApi(); + log('api(fallback)', items.length); + return persist(items, 'api'); + } catch (apiErr) { + errors.push(`api=${apiErr?.message}`); + const noSource = !getYouTubeKeys().length; + if (noSource) { + const err = new Error(`YouTube indisponible (${errors.join(' | ').slice(0, 220)}). Vérifiez le réseau/YT_DLP_PATH ou ajoutez YOUTUBE_API_KEYS.`); + err.ytStatus = 503; err.code = 'youtube_no_source'; + throw err; + } + console.error(`[YT search] all sources failed. ${errors.join(' | ')}`); + const err = new Error(errors[0] || 'search_failed'); + err.ytStatus = 502; err.code = 'youtube_all_sources_failed'; + throw err; + } } catch (error) { - // Log explicite (clé expirée / quota) puis remonte l'erreur pour que - // /api/search la reporte dans `errors.yt` au lieu d'un groupe vide silencieux. const status = error?.ytStatus || 500; console.error(`YouTube search error (status ${status}):`, error?.message || error); throw error; } + }, + async suggest(q, opts) { + const limit = Math.min(20, Math.max(1, Number(opts?.limit || 10))); + const query = String(q || '').trim(); + if (query.length < 2) return []; + try { + const params = new URLSearchParams({ client: 'youtube', ds: 'yt', hl: 'fr', q: query }); + const headers = { 'User-Agent': 'Mozilla/5.0' }; + const proxy = String(process.env.YT_EGRESS_PROXY || '').trim(); + const fetchOpts = { headers, signal: AbortSignal.timeout(6000) }; + // Note : suggest reste sans proxy (endpoint Google public léger) sauf si egress configuré côté infra. + if (proxy) console.debug?.('[YT suggest] egress proxy configured, direct fetch kept for suggest'); + const resp = await fetch(`https://suggestqueries.google.com/complete/search?${params.toString()}`, fetchOpts); + if (!resp.ok) return []; + const text = await resp.text(); + return parseYoutubeSuggestResponse(text).slice(0, limit); + } catch { + return []; + } } }; diff --git a/server/tests/suggest.test.mjs b/server/tests/suggest.test.mjs new file mode 100644 index 0000000..9c99f28 --- /dev/null +++ b/server/tests/suggest.test.mjs @@ -0,0 +1,130 @@ +// Step 15 — Suggest tests: unit parsing/dedup (offline) + API contract (isolated server). +// Run with: npm run test:suggest +// Scenario coverage (todo Step 15): +// - providers=yt returns only yt suggestions +// - unknown provider falls back to the full registry +// Assertions are on fan-out shape, never on external provider content. + +import fs from 'node:fs'; +import path from 'node:path'; +import os from 'node:os'; +import { spawn } from 'node:child_process'; +import net from 'node:net'; + +function expect(cond, msg) { + if (!cond) throw new Error(`Assertion failed: ${msg}`); +} +function logOk(msg) { console.log(`✓ ${msg}`); } + +// ---------- 1) Offline unit tests: parsers + graceful degradation ---------- +const yt = (await import('../providers/youtube.mjs')).default; +const { parseYoutubeSuggestResponse } = await import('../providers/youtube.mjs'); +const dm = (await import('../providers/dailymotion.mjs')).default; + +// Bare JSON payload +expect( + JSON.stringify(parseYoutubeSuggestResponse('["tut",["tutoriel angular","tutoriel android"],[],{}]')) === + JSON.stringify(['tutoriel angular', 'tutoriel android']), + 'bare JSON suggest payload parses', +); +logOk('parseYoutubeSuggestResponse bare JSON'); + +// window.google.ac.h(...) wrapped payload +expect( + JSON.stringify( + parseYoutubeSuggestResponse('window.google.ac.h(["tut",["tutoriel angular"],[],{}])'), + ) === JSON.stringify(['tutoriel angular']), + 'wrapped JSONP suggest payload parses', +); +logOk('parseYoutubeSuggestResponse wrapped payload'); + +// Array-wrapped suggestions (newer format: [text, type, ...]) +expect( + JSON.stringify(parseYoutubeSuggestResponse('["tut",[["tutoriel angular",0],["tutoriel android",0]]]')) === + JSON.stringify(['tutoriel angular', 'tutoriel android']), + 'array-wrapped suggestions keep text only', +); +logOk('parseYoutubeSuggestResponse array-wrapped entries'); + +// Garbage / empty payloads never throw +expect(parseYoutubeSuggestResponse('not json at all').length === 0, 'garbage payload yields []'); +expect(parseYoutubeSuggestResponse('').length === 0, 'empty payload yields []'); +logOk('parseYoutubeSuggestResponse graceful degradation'); + +// min-length guard: no network below 2 chars (would throw offline otherwise) +expect(JSON.stringify(await yt.suggest('a')) === '[]', 'yt.suggest short query returns [] without network'); +expect(JSON.stringify(await dm.suggest('x')) === '[]', 'dm.suggest short query returns [] without network'); +logOk('suggest min-length guard (no network)'); + +// ---------- 2) API contract against a real isolated server ---------- +const PORT = await new Promise((resolve) => { + const srv = net.createServer(); + srv.listen(0, '127.0.0.1', () => { + const p = srv.address().port; + srv.close(() => resolve(p)); + }); +}); +const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'newtube-suggest-test-')); +const server = spawn(process.execPath, ['./server/index.mjs'], { + env: { ...process.env, PORT: String(PORT), NEWTUBE_DB_FILE: path.join(tmpDir, 'suggest.db'), JWT_SECRET: 'suggest-test-secret', NODE_ENV: 'test' }, + stdio: ['ignore', 'pipe', 'pipe'], + cwd: path.resolve(import.meta.dirname, '..', '..'), +}); +let serverLogs = ''; +server.stdout.on('data', (d) => { serverLogs += d.toString(); }); +server.stderr.on('data', (d) => { serverLogs += d.toString(); }); +const baseUrl = `http://127.0.0.1:${PORT}`; +const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); +async function waitForServer(timeoutMs = 20000) { + const start = Date.now(); + while (Date.now() - start < timeoutMs) { + try { + const res = await fetch(`${baseUrl}/`); + if (res.status < 500) return true; + } catch {} + await sleep(300); + } + return false; +} + +let failures = 0; +async function scenario(name, fn) { + try { await fn(); logOk(name); } + catch (e) { failures += 1; console.error(`✗ ${name}`); console.error(e?.stack || String(e)); } +} + +try { + const up = await waitForServer(); + expect(up, 'isolated server starts'); + + await scenario('q too short -> 400', async () => { + const res = await fetch(`${baseUrl}/api/search/suggest?q=a`); + expect(res.status === 400, `expected 400, got ${res.status}`); + }); + + await scenario('providers=yt returns only yt suggestions', async () => { + const res = await fetch(`${baseUrl}/api/search/suggest?q=tutoriel&providers=yt&limit=5`); + expect(res.status === 200, `expected 200, got ${res.status}`); + const body = await res.json(); + expect(body.q === 'tutoriel', 'q echoed'); + expect(JSON.stringify(Object.keys(body.groups)) === JSON.stringify(['yt']), `only yt group (got ${Object.keys(body.groups)})`); + expect(Array.isArray(body.groups.yt), 'yt group is an array'); + expect(body.groups.yt.length <= 5, 'limit respected'); + }); + + await scenario('unknown provider falls back to the full registry', async () => { + const res = await fetch(`${baseUrl}/api/search/suggest?q=tutoriel&providers=xx,yy&limit=3`); + expect(res.status === 200, `expected 200, got ${res.status}`); + const body = await res.json(); + const keys = Object.keys(body.groups).sort(); + expect(JSON.stringify(keys) === JSON.stringify(['dm', 'od', 'pt', 'ru', 'tw', 'yt']), `full registry fallback (got ${keys})`); + }); +} finally { + server.kill(); +} + +if (failures > 0) { + console.error(`\n${failures} suggest scenario(s) failed. Server logs:\n${serverLogs}`); + process.exit(1); +} +console.log('\nAll suggest tests passed.'); diff --git a/server/tests/transcript.test.mjs b/server/tests/transcript.test.mjs new file mode 100644 index 0000000..771f356 --- /dev/null +++ b/server/tests/transcript.test.mjs @@ -0,0 +1,212 @@ +// Step 16 — Transcript tests (offline fixtures for parsers + API contract). +// Run with: npm run test:transcript +import { describe, it } from 'node:test'; +import assert from 'node:assert/strict'; +import fs from 'node:fs'; +import path from 'node:path'; +import os from 'node:os'; +import { spawn } from 'node:child_process'; +import net from 'node:net'; +import { + pickTrack, + parseJson3, + parseVtt, + parseTrackText, + parseXmlCaptions, + orderedTracks, + looksLikeHtmlError, + capLines, + normalizeTranscriptProvider, +} from '../transcript.mjs'; + +const track = (url, ext = 'vtt') => ({ url, ext }); + +describe('pickTrack', () => { + it('prefers manual subtitles over automatic captions', () => { + const json = { + subtitles: { en: [track('https://x/en.vtt')] }, + automatic_captions: { en: [track('https://x/en.auto.vtt')] }, + }; + const { track: t, lang } = pickTrack(json, 'en'); + assert.equal(t.url, 'https://x/en.vtt'); + assert.equal(lang, 'en'); + }); + + it('falls back fr -> fr-* -> auto -> first available', () => { + const json = { + subtitles: { 'fr-CA': [track('https://x/frca.vtt')] }, + automatic_captions: { en: [track('https://x/en.auto.vtt')] }, + }; + assert.equal(pickTrack(json, 'fr').lang, 'fr-CA'); + // 'de' matches nothing: first available (manual preferred) + const fb = pickTrack(json, 'de'); + assert.equal(fb.lang, 'fr-CA'); + // auto-only dict + const autoOnly = pickTrack({ automatic_captions: { en: [track('https://x/e.vtt')] } }, 'en'); + assert.equal(autoOnly.track.url, 'https://x/e.vtt'); + }); + + it('returns null track when no subtitles exist', () => { + assert.deepEqual(pickTrack({}, 'fr'), { track: null, languages: [], lang: null }); + assert.deepEqual(pickTrack({ subtitles: {}, automatic_captions: {} }, 'en'), { + track: null, + languages: [], + lang: null, + }); + }); + + it('exposes the language list for the UI selector', () => { + const { languages } = pickTrack( + { subtitles: { fr: [track('a')], en: [track('b')] }, automatic_captions: { es: [track('c')] } }, + 'fr', + ); + assert.deepEqual([...languages].sort(), ['en', 'es', 'fr']); + }); +}); + +describe('parseJson3', () => { + it('converts events with segs to normalized lines', () => { + const lines = parseJson3({ + events: [ + { tStartMs: 0, dDurationMs: 2500, segs: [{ utf8: 'Bonjour' }] }, + { tStartMs: 2500, dDurationMs: 3100, segs: [{ utf8: 'Bien' }, { utf8: 'venue' }] }, + ], + }); + assert.deepEqual(lines, [ + { t: 0, dur: 2.5, text: 'Bonjour' }, + { t: 2.5, dur: 3.1, text: 'Bienvenue' }, + ]); + }); + + it('drops events without segs or with empty text', () => { + const lines = parseJson3({ events: [{ tStartMs: 0 }, { tStartMs: 1, dDurationMs: 1, segs: [{ utf8: ' ' }] }] }); + assert.deepEqual(lines, []); + assert.deepEqual(parseJson3({}), []); + }); +}); + +describe('parseVtt', () => { + const vtt = `WEBVTT + +00:00:00.000 --> 00:00:02.500 +Bonjour à tous + +12 +00:00:02.500 --> 00:00:05.600 +Bienvenue +dans cette vidéo +`; + it('parses cues with identifiers, tags and multi-line content', () => { + const lines = parseVtt(vtt); + assert.equal(lines.length, 2); + assert.equal(lines[0].t, 0); + assert.equal(lines[0].dur, 2.5); + assert.equal(lines[0].text, 'Bonjour à tous'); + assert.equal(lines[1].text, 'Bienvenue dans cette vidéo'); + assert.ok(Math.abs(lines[1].t - 2.5) < 1e-9); + }); + + it('supports MM:SS.mmm timestamps and decodes entities', () => { + const lines = parseVtt('WEBVTT\n\n01:02.000 --> 01:04.500\nFish & chips\n'); + assert.equal(lines.length, 1); + assert.equal(lines[0].t, 62); + assert.equal(lines[0].text, 'Fish & chips'); + }); + + it('ignores garbage blocks', () => { + assert.deepEqual(parseVtt('WEBVTT\n\nNOTE nothing here\n'), []); + }); +}); + +describe('orderedTracks + parseTrackText + XML captions', () => { + const auto = { + fr: [{ url: 'https://x/fr.json3', ext: 'json3' }], + en: [{ url: 'https://x/en.json3', ext: 'json3' }], + es: [{ url: 'https://x/es.vtt', ext: 'vtt' }], + }; + it('tries requested lang first, then original en', () => { + const langs = orderedTracks({ subtitles: {}, automatic_captions: auto }, 'fr').map((o) => o.lang); + assert.deepEqual(langs, ['fr', 'en', 'es']); + }); + it('dedupes identical track URLs', () => { + const dup = { subtitles: {}, automatic_captions: { en: [{ url: 'https://x/same', ext: 'vtt' }], fr: [{ url: 'https://x/same', ext: 'vtt' }] } }; + assert.equal(orderedTracks(dup, 'fr').length, 1); + }); + it('detects HTML error pages instead of JSON-parsing them', () => { + assert.equal(looksLikeHtmlError('Sorry'), true); + assert.equal(looksLikeHtmlError('WEBVTT\n\n00:00:00.000 --> 1'), false); + assert.deepEqual(parseTrackText(' nope', 'json3'), []); + }); + it('parses srv/ttml XML captions', () => { + const srv = parseXmlCaptions('

Hello

'); + assert.equal(srv.length, 1); + assert.equal(srv[0].text, 'Hello'); + const ttml = parseXmlCaptions('

Bonjour

'); + assert.equal(ttml.length, 1); + assert.equal(ttml[0].t, 1); + }); + it('parseTrackText handles json3 and vtt payloads', () => { + const j = parseTrackText(JSON.stringify({ events: [{ tStartMs: 0, dDurationMs: 1000, segs: [{ utf8: 'Hi' }] }] }), 'json3'); + assert.equal(j.length, 1); + const v = parseTrackText('WEBVTT\n\n00:00:00.000 --> 00:00:01.000\nHello\n', 'vtt'); + assert.equal(v.length, 1); + }); +}); + +describe('capLines + normalizeTranscriptProvider', () => { it('truncates very long transcripts', () => { + const lines = Array.from({ length: 10 }, (_, i) => ({ t: i, dur: 1, text: `l${i}` })); + assert.equal(capLines(lines, 3).length, 3); + }); + it('maps short and long provider ids', () => { + assert.equal(normalizeTranscriptProvider('yt'), 'youtube'); + assert.equal(normalizeTranscriptProvider('youtube'), 'youtube'); + assert.equal(normalizeTranscriptProvider('dm'), 'dailymotion'); + assert.equal(normalizeTranscriptProvider('pt'), 'peertube'); + assert.equal(normalizeTranscriptProvider('xx'), null); + }); +}); + +describe('API contract', () => { + it('rejects unknown providers with 400 { available: false }', async () => { + const port = await new Promise((resolve) => { + const srv = net.createServer(); + srv.listen(0, '127.0.0.1', () => { + const p = srv.address().port; + srv.close(() => resolve(p)); + }); + }); + const tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'newtube-transcript-test-')); + const server = spawn(process.execPath, ['./server/index.mjs'], { + env: { + ...process.env, + PORT: String(port), + NEWTUBE_DB_FILE: path.join(tmpDir, 'transcript.db'), + JWT_SECRET: 'transcript-test-secret', + NODE_ENV: 'test', + }, + stdio: ['ignore', 'pipe', 'pipe'], + cwd: path.resolve(import.meta.dirname, '..', '..'), + }); + const baseUrl = `http://127.0.0.1:${port}`; + const sleep = (ms) => new Promise((r) => setTimeout(r, ms)); + try { + let up = false; + const start = Date.now(); + while (Date.now() - start < 20000) { + try { + const res = await fetch(`${baseUrl}/`); + if (res.status < 500) { up = true; break; } + } catch {} + await sleep(300); + } + assert.ok(up, 'isolated server starts'); + const res = await fetch(`${baseUrl}/api/transcript/xx/abc123?lang=fr`); + assert.equal(res.status, 400); + const body = await res.json(); + assert.equal(body.available, false); + assert.ok(typeof body.error === 'string'); + } finally { + server.kill(); + } + }); +}); diff --git a/server/tests/youtube-innertube.test.mjs b/server/tests/youtube-innertube.test.mjs new file mode 100644 index 0000000..14e84ce --- /dev/null +++ b/server/tests/youtube-innertube.test.mjs @@ -0,0 +1,131 @@ +// Step 18 — tests offline InnerTube (AUCUN réseau : mappers purs + modes). +import assert from 'node:assert/strict'; +import { mapVideoNode, mapLockupView, mapNodes, mapCaptionTracks, parseViewsText, parseDurationLabel } from '../providers/youtube-innertube.mjs'; +import { pickTrack, orderedTracks } from '../transcript.mjs'; +import { getSearchMode, YT_SEARCH_MODES } from '../providers/youtube-common.mjs'; + +function ok(msg) { console.log(`✓ ${msg}`); } + +// 1) parseViewsText FR/EN +{ + assert.equal(parseViewsText('81 973 vues'), 81973); + assert.equal(parseViewsText('1,2 M vues'), 1200000); + assert.equal(parseViewsText('57K'), 57000); + assert.equal(parseViewsText('3 k vues'), 3000); + assert.equal(parseViewsText(''), undefined); + assert.equal(parseViewsText(null), undefined); + ok('parseViewsText FR/EN'); +} + +// 1b) parseDurationLabel (a11y) +{ + assert.equal(parseDurationLabel("What's new in Angular 18 2 minutes, 27 seconds"), 147); + assert.equal(parseDurationLabel('Live 1 heure, 5 minutes'), 3900); + assert.equal(parseDurationLabel('nonsense'), undefined); + ok('parseDurationLabel'); +} + +// 2) mapVideoNode type Video (forme réelle youtubei.js v18) +{ + const m = mapVideoNode({ + type: 'Video', id: 'abc123', + title: { text: 'Demo' }, + duration: { text: '15:22', seconds: 922 }, + thumbnails: [{ url: 'http://t/low.jpg' }, { url: 'http://t/high.jpg' }], + author: { id: 'UC999', name: 'Chaine' }, + published: { text: 'il y a 2 ans' }, + view_count: { text: '81 973 vues' }, + badges: [{ label: 'Sous-titres' }], + is_live: false, + }); + assert.equal(m.id, 'abc123'); + assert.equal(m.url, 'https://www.youtube.com/watch?v=abc123'); + assert.equal(m.thumbnail, 'http://t/high.jpg'); + assert.equal(m.duration, 922); + assert.equal(m.views, 81973); + assert.equal(m.channelId, 'UC999'); + assert.deepEqual(m.badges, ['Sous-titres']); + assert.equal(m.isShort, undefined); + ok('mapVideoNode Video'); +} + +// 3) Shorts/Reel -> isShort ; live -> isLive ; bruit filtré +{ + const s = mapVideoNode({ type: 'ShortsLockupView', id: 'sh1', title: { text: 'Short' }, thumbnails: [{ url: 'u' }] }); + assert.equal(s.isShort, true); + const l = mapVideoNode({ type: 'Video', id: 'lv1', title: { text: 'Live' }, thumbnails: [{ url: 'u' }], is_live: true }); + assert.equal(l.isLive, true); + assert.equal(mapVideoNode({ type: 'Playlist', id: 'PL1', title: { text: 'PL' } }), null); + assert.equal(mapVideoNode({ type: 'Channel', id: 'UC1', title: { text: 'C' } }), null); + assert.equal(mapVideoNode(null), null); + assert.equal(mapVideoNode({ type: 'Video', title: { text: 'no id' } }), null); + ok('mapVideoNode shorts/live/filtrage'); +} + +// 4) mapNodes déduplique +{ + const mk = (id) => ({ type: 'Video', id, title: { text: id }, thumbnails: [{ url: 'u' }] }); + const out = mapNodes([mk('a'), mk('a'), mk('b'), null]); + assert.deepEqual(out.map((x) => x.id), ['a', 'b']); + ok('mapNodes dédup'); +} + +// 4b) mapLockupView (renderer moderne du watch-next façon SmartTube) +{ + const m = mapLockupView({ + type: 'LockupView', content_id: '0wIL6d5TxGc', content_type: 'VIDEO', + content_image: { image: [{ url: 'http://t/small.jpg', width: 168 }, { url: 'http://t/big.jpg', width: 336 }] }, + metadata: { + title: { text: "What's new" }, + metadata: { metadata_rows: [ + { metadata_parts: [{ text: { text: 'Rainer Hahnekamp' } }] }, + { metadata_parts: [{ text: { text: '57K' } }] }, + ] }, + image: { a11y_label: 'Go to channel Rainer Hahnekamp' }, + }, + renderer_context: { accessibility_context: { label: "What's new 2 minutes, 27 seconds" } }, + }); + assert.equal(m.id, '0wIL6d5TxGc'); + assert.equal(m.url, 'https://www.youtube.com/watch?v=0wIL6d5TxGc'); + assert.equal(m.thumbnail, 'http://t/big.jpg'); + assert.equal(m.uploaderName, 'Rainer Hahnekamp'); + assert.equal(m.views, 57000); + assert.equal(m.duration, 147); + assert.equal(mapVideoNode({ type: 'LockupView', content_id: 'x', content_type: 'VIDEO', metadata: { title: { text: 'T' } } }).id, 'x'); + assert.equal(mapLockupView({ type: 'LockupView', content_type: 'PLAYLIST', content_id: 'PL1', metadata: { title: { text: 'PL' } } }), null); + ok('mapLockupView watch-next'); +} + +// 5) modes innertube connus +{ + assert.ok(YT_SEARCH_MODES.includes('innertube-first')); + assert.ok(YT_SEARCH_MODES.includes('innertube-only')); + const prev = process.env.YT_SEARCH_MODE; + process.env.YT_SEARCH_MODE = 'innertube-only'; + assert.equal(getSearchMode(), 'innertube-only'); + if (prev === undefined) delete process.env.YT_SEARCH_MODE; else process.env.YT_SEARCH_MODE = prev; + ok('modes innertube-first/only'); +} + +// 6) mapCaptionTracks : forme InnerTube -> pseudo dump yt-dlp, interop pickTrack +{ + const cap = mapCaptionTracks([ + { base_url: 'https://x/t?caps=asr&x=1', language_code: 'en', kind: 'asr', name: { text: 'English (auto-generated)' } }, + { base_url: 'https://x/t?caps=m&x=2', language_code: 'fr', name: { text: 'Français' } }, + { base_url: '', language_code: 'de', name: 'Deutsch' }, + ]); + assert.deepEqual(cap.languages.sort(), ['en', 'fr']); + assert.equal(cap.trackCount, 2); + assert.match(cap.automatic_captions.en[0].url, /fmt=json3/); + assert.equal(cap.automatic_captions.en[0].ext, 'json3'); + assert.equal(cap.subtitles.fr[0].name, 'Français'); + // Interop : le pseudo dump passe dans pickTrack/orderedTracks sans modification + const picked = pickTrack({ subtitles: cap.subtitles, automatic_captions: cap.automatic_captions }, 'fr'); + assert.equal(picked.lang, 'fr'); + assert.ok(picked.track.url.includes('fmt=json3')); + const ordered = orderedTracks({ subtitles: cap.subtitles, automatic_captions: cap.automatic_captions }, 'de'); + assert.ok(ordered.length >= 2); + ok('mapCaptionTracks + interop pickTrack/orderedTracks'); +} + +console.log('youtube-innertube tests passed'); diff --git a/server/tests/youtube-scrape.test.mjs b/server/tests/youtube-scrape.test.mjs new file mode 100644 index 0000000..1a45b6c --- /dev/null +++ b/server/tests/youtube-scrape.test.mjs @@ -0,0 +1,98 @@ +// Step 17 — tests offline : parsers scrape + dispatcher (AUCUN réseau, AUCUNE clé). +import assert from 'node:assert/strict'; +import { mapFlatEntry, parseFlatPlaylistJson, classifyScrapeError } from '../providers/youtube-scrape.mjs'; +import { getSearchMode, hashSearchKey, buildYtDlpExtraArgs, resolveYtDlpBin, resetYtDlpBinCache } from '../providers/youtube-common.mjs'; + +function ok(msg) { console.log(`✓ ${msg}`); } + +// 1) mapFlatEntry +{ + const s = mapFlatEntry({ + id: 'dQw4w9WgXcQ', title: 'Demo', channel_id: 'UC123', channel: 'Chaine', + duration: 212, view_count: 1000, timestamp: 1700000000, + thumbnails: [{ url: 'http://t/low.jpg', height: 90 }, { url: 'http://t/high.jpg', height: 720 }], + }); + assert.equal(s.id, 'dQw4w9WgXcQ'); + assert.equal(s.url, 'https://www.youtube.com/watch?v=dQw4w9WgXcQ'); + assert.equal(s.thumbnail, 'http://t/high.jpg'); + assert.equal(s.duration, 212); + ok('mapFlatEntry maps flat entry to Suggestion'); +} +assert.equal(mapFlatEntry(null), null); +assert.equal(mapFlatEntry({}), null); +ok('mapFlatEntry null-safe'); + +// 2) parseFlatPlaylistJson : objet + NDJSON + vide +{ + const raw = JSON.stringify({ entries: [{ id: 'a', title: 'A' }, { id: '', title: 'x' }, null] }); + assert.equal(parseFlatPlaylistJson(raw).length, 1); + ok('parseFlatPlaylistJson objet .entries'); +} +{ + const raw = `{"id":"a","title":"A"}\n{"id":"b","title":"B"}\nnot-json\n`; + assert.equal(parseFlatPlaylistJson(raw).length, 2); + ok('parseFlatPlaylistJson NDJSON tolerant'); +} +assert.deepEqual(parseFlatPlaylistJson(''), []); +assert.deepEqual(parseFlatPlaylistJson('{{{'), []); +ok('parseFlatPlaylistJson vide/invalide -> []'); + +// 3) getSearchMode défaut innertube-first + normalisation +{ + const prev = process.env.YT_SEARCH_MODE; + delete process.env.YT_SEARCH_MODE; + assert.equal(getSearchMode(), 'innertube-first'); + process.env.YT_SEARCH_MODE = 'API-FIRST'; + assert.equal(getSearchMode(), 'api-first'); + process.env.YT_SEARCH_MODE = 'scrape-first'; + assert.equal(getSearchMode(), 'scrape-first'); + process.env.YT_SEARCH_MODE = 'nonsense'; + assert.equal(getSearchMode(), 'innertube-first'); + if (prev === undefined) delete process.env.YT_SEARCH_MODE; else process.env.YT_SEARCH_MODE = prev; + ok('getSearchMode défaut + normalisation'); +} + +// 4) hash stable + extra args jamais de secret en clair autre que proxy path +{ + assert.equal(hashSearchKey('a'), hashSearchKey('a')); + assert.notEqual(hashSearchKey('a'), hashSearchKey('b')); + const args = buildYtDlpExtraArgs(); + assert.ok(Array.isArray(args)); + ok('hashSearchKey + buildYtDlpExtraArgs'); +} + +// 5) dispatcher scrape-only sans clé : ne doit PAS exiger YOUTUBE_API_KEY au import, +// et searchViaScrape vide sur query courte +{ + const { searchViaScrape } = await import('../providers/youtube-scrape.mjs'); + assert.deepEqual(await searchViaScrape('a'), []); + ok('searchViaScrape query courte -> [] sans réseau'); +} + +// 6) classifyScrapeError : ENOENT -> yt_scrape_no_binary (message actionnable, jamais de spawn brut en UI) +{ + const e = classifyScrapeError(Object.assign(new Error('spawn yt-dlp ENOENT'), { code: 'ENOENT', errno: 'ENOENT' })); + assert.equal(e.code, 'yt_scrape_no_binary'); + assert.equal(e.ytStatus, 503); + assert.match(e.message, /YT_DLP_PATH/); + const passthrough = classifyScrapeError(Object.assign(new Error('x'), { code: 'yt_scrape_no_binary' })); + assert.equal(passthrough.code, 'yt_scrape_no_binary'); + ok('classifyScrapeError ENOENT -> yt_scrape_no_binary'); +} + +// 7) resolveYtDlpBin : YT_DLP_PATH inexistant -> fallback PATH/bundled (jamais de throw si un binaire existe) +{ + resetYtDlpBinCache(); + const prev = process.env.YT_DLP_PATH; + process.env.YT_DLP_PATH = '/nonexistent/yt-dlp'; + try { + const bin = await resolveYtDlpBin(); + assert.ok(typeof bin === 'string' && bin.length > 0); + ok(`resolveYtDlpBin fallback -> ${bin}`); + } finally { + resetYtDlpBinCache(); + if (prev === undefined) delete process.env.YT_DLP_PATH; else process.env.YT_DLP_PATH = prev; + } +} + +console.log('youtube-scrape tests passed'); diff --git a/server/transcript.mjs b/server/transcript.mjs new file mode 100644 index 0000000..8a6c411 --- /dev/null +++ b/server/transcript.mjs @@ -0,0 +1,309 @@ +// Step 16 — Pure transcript helpers (no network, no DB). +// Data source: `yt-dlp --dump-single-json --skip-download` exposes +// `subtitles` (manual) and `automatic_captions` (auto-generated). +// Track entries are arrays of { url, ext, name } (or single objects). + +/** + * @typedef {{ t: number, dur: number, text: string }} TranscriptLine + */ + +const MAX_LINES_DEFAULT = Number(process.env.TRANSCRIPT_MAX_LINES || 5000); + +/** Normalize a language code for comparison (lowercase, `_` -> `-`). */ +function normLang(code) { + return String(code || '').trim().toLowerCase().replace(/_/g, '-'); +} + +/** Pick the first usable track object from an array-or-single entry. */ +function firstTrack(entry) { + if (!entry) return null; + const list = Array.isArray(entry) ? entry : [entry]; + return list.find((t) => t && typeof t.url === 'string' && t.url) || null; +} + +function trackExtOf(track) { + const ext = String(track?.ext || '').toLowerCase(); + if (ext) return ext; + try { + const u = new URL(String(track?.url || '')); + const m = /\.([a-z0-9]+)(?:[?#]|$)/i.exec(u.pathname || ''); + if (m) return m[1].toLowerCase(); + } catch {} + return ''; +} + +/** Find a language key in `dict` matching `code` exactly or by prefix (`fr` -> `fr-*`). */ +function findLangKey(dict, code) { + const want = normLang(code); + if (!want || !dict) return null; + const keys = Object.keys(dict); + const exact = keys.find((k) => normLang(k) === want); + if (exact) return exact; + // `fr-ca` requested, `fr` available (and vice versa): match on primary subtag + const primary = want.split('-')[0]; + return keys.find((k) => normLang(k).split('-')[0] === primary) || null; +} + +/** + * Select the best subtitle track. + * Priority: manual exact > manual prefix/primary > auto exact > auto prefix/primary > first available. + * @param {any} json yt-dlp dump-single-json + * @param {string} [lang] + * @returns {{ track: any|null, languages: string[], lang: string|null }} + */ +export function pickTrack(json, lang = 'fr') { + const manual = (json && json.subtitles) || {}; + const auto = (json && json.automatic_captions) || {}; + const manualKeys = Object.keys(manual); + const autoKeys = Object.keys(auto); + const languages = Array.from(new Set([...manualKeys, ...autoKeys])); + + if (languages.length === 0) return { track: null, languages: [], lang: null }; + + const manualKey = findLangKey(manual, lang); + if (manualKey) { + const track = firstTrack(manual[manualKey]); + if (track) return { track, languages, lang: manualKey }; + } + const autoKey = findLangKey(auto, lang); + if (autoKey) { + const track = firstTrack(auto[autoKey]); + if (track) return { track, languages, lang: autoKey }; + } + // Fallback: first available track (manual preferred) + for (const key of manualKeys) { + const track = firstTrack(manual[key]); + if (track) return { track, languages, lang: key }; + } + for (const key of autoKeys) { + const track = firstTrack(auto[key]); + if (track) return { track, languages, lang: key }; + } + return { track: null, languages, lang: null }; +} + +/** + * Order candidate tracks to try when the preferred one cannot be fetched. + * YouTube auto-translated tracks (`tlang=xx`) are often rate-limited (HTTP 429 + * or "Sorry" HTML pages) while the original language still works: always try + * the requested language first, then the original (`en`), then everything + * else (manual preferred). De-duplicated by URL. + * @param {any} json yt-dlp dump-single-json + * @param {string} [lang] + * @returns {Array<{ track: any, lang: string|null }>} + */ +export function orderedTracks(json, lang = 'fr') { + const manual = (json && json.subtitles) || {}; + const auto = (json && json.automatic_captions) || {}; + const seen = new Set(); + const out = []; + const push = (key, dict) => { + if (key == null) return; + const t = firstTrack(dict[key]); + if (!t || !t.url || seen.has(String(t.url))) return; + seen.add(String(t.url)); + out.push({ track: t, lang: key }); + }; + const primary = pickTrack(json, lang); + if (primary.track) { + seen.add(String(primary.track.url)); + out.push({ track: primary.track, lang: primary.lang }); + } + // Original language first fallback (avoids translated-track rate limits) + const reqPrimary = String(lang || '').toLowerCase().split('-')[0]; + for (const fallbackLang of ['en', 'fr']) { + if (fallbackLang === reqPrimary) continue; + const k = findLangKey(manual, fallbackLang); + if (k) push(k, manual); + const ka = findLangKey(auto, fallbackLang); + if (ka) push(ka, auto); + } + for (const key of Object.keys(manual)) push(key, manual); + for (const key of Object.keys(auto)) push(key, auto); + return out; +} + +/** + * Parse a YouTube `json3` timedtext payload into normalized lines. + * @param {any} data parsed JSON + * @returns {TranscriptLine[]} + */ +export function parseJson3(data) { + const events = (data && data.events) || []; + if (!Array.isArray(events)) return []; + const out = []; + for (const e of events) { + if (!e || !Array.isArray(e.segs)) continue; + const text = e.segs.map((s) => String(s?.utf8 ?? '')).join('').replace(/\n+/g, ' ').trim(); + if (!text) continue; + out.push({ + t: Number(e.tStartMs || 0) / 1000, + dur: Number(e.dDurationMs || 0) / 1000, + text, + }); + } + return capLines(out); +} + +/** Decode a handful of HTML entities found in VTT payloads. */ +function decodeEntities(s) { + return String(s || '') + .replace(/&/g, '&') + .replace(/</g, '<') + .replace(/>/g, '>') + .replace(/"/g, '"') + .replace(/'/g, "'") + .replace(/ /g, ' '); +} + +/** Strip XML/HTML tags and decode entities to plain cue text. */ +function xmlTextToPlain(s) { + return decodeEntities(String(s || '').replace(/<[^>]*>/g, ' ').replace(/\s+/g, ' ').trim()).trim(); +} + +function parseTimeAttrToSeconds(v) { + if (v == null || v === '') return null; + const s = String(v).trim(); + if (!s) return null; + if (/^\d+(\.\d+)?$/.test(s)) return Number(s); + const m = /(?:(\d+):)?([0-5]?\d):([0-5]\d)(?:\.(\d{1,3}))?/.exec(s); + if (!m) return null; + return Number(m[1] || 0) * 3600 + Number(m[2]) * 60 + Number(m[3]) + Number((m[4] || '0').padEnd(3, '0')) / 1000; +} + +/** + * Parse YouTube XML caption formats (`srv1`/`srv2`/`srv3` `

` cues, + * `ttml` `

` cues) into normalized lines. + * @param {string} text raw XML + * @returns {TranscriptLine[]} + */ +export function parseXmlCaptions(text) { + const out = []; + const src = String(text || ''); + if (!src || !/<(p|text|tt|transcript)\b/i.test(src)) return capLines(out); + // YouTube srv*:

Hello world

+ const srvRe = /]*?(?:t|start)="([^"]*)"[^>]*?(?:d|dur)="([^"]*)"[^>]*>([\s\S]*?)<\/p\s*>/gi; + // Generic ttml:

...

+ const ttmlRe = /]*?begin="([^"]*)"[^>]*?end="([^"]*)"[^>]*>([\s\S]*?)<\/p\s*>/gi; + let m; + while ((m = srvRe.exec(src)) !== null) { + const start = Number(m[1]) / 1000; + const dur = Number(m[2]) / 1000; + const cleaned = xmlTextToPlain(m[3]); + if (!cleaned || !Number.isFinite(start)) continue; + out.push({ t: start, dur: Number.isFinite(dur) && dur > 0 ? dur : 0, text: cleaned }); + } + if (out.length === 0) { + while ((m = ttmlRe.exec(src)) !== null) { + const start = parseTimeAttrToSeconds(m[1]); + const end = parseTimeAttrToSeconds(m[2]); + const cleaned = xmlTextToPlain(m[3]); + if (!cleaned || start == null || end == null) continue; + out.push({ t: start, dur: Math.max(0, end - start), text: cleaned }); + } + } + // Plain Fallback variant + if (out.length === 0) { + const textRe = /]*?start="([^"]*)"[^>]*?(?:dur="([^"]*)")?[^>]*>([\s\S]*?)<\/text\s*>/gi; + while ((m = textRe.exec(src)) !== null) { + const start = Number(m[1]); + const dur = Number(m[2] || 0); + const cleaned = xmlTextToPlain(m[3]); + if (!cleaned || !Number.isFinite(start)) continue; + out.push({ t: start, dur: dur > 0 ? dur : 0, text: cleaned }); + } + } + return capLines(out); +} + +/** Detect YouTube "Sorry / automated queries" style HTML error pages. */ +export function looksLikeHtmlError(text) { + const s = String(text || '').trimStart().slice(0, 512).toLowerCase(); + return s.startsWith('`, ``, timestamps). + * @param {string} text raw VTT + * @returns {TranscriptLine[]} + */ +export function parseVtt(text) { + if (looksLikeHtmlError(text)) return []; + const out = []; + const blocks = String(text || '').replace(/\r\n/g, '\n').split(/\n{2,}/); + const tsRe = /(?:(\d{2,}):)?([0-5]?\d):([0-5]\d)\.(\d{3})\s*-->\s*(?:(\d{2,}):)?([0-5]?\d):([0-5]\d)\.(\d{3})/; + for (const block of blocks) { + const lines = String(block || '').split('\n'); + const timingIdx = lines.findIndex((l) => l.includes('-->')); + if (timingIdx === -1) continue; + const m = tsRe.exec(lines[timingIdx]); + if (!m) continue; + const start = vttTimestampToSeconds(m[1] || '0', m[2], m[3], m[4]); + const end = vttTimestampToSeconds(m[5] || '0', m[6], m[7], m[8]); + const content = lines + .slice(timingIdx + 1) + .join(' ') + // strip inline VTT tags (, , <00:00:01.000>, ...) + .replace(/<[^>]*>/g, ' ') + .replace(/\s+/g, ' ') + .trim(); + const decoded = decodeEntities(content).trim(); + // Skip WEBVTT header leftovers and empty cues + if (!decoded || /^WEBVTT/i.test(decoded)) continue; + out.push({ t: start, dur: Math.max(0, end - start), text: decoded }); + } + return capLines(out); +} + +/** Bound response volume for very long transcripts (Phase 1 decision: truncate). */ +export function capLines(lines, max = MAX_LINES_DEFAULT) { + const list = Array.isArray(lines) ? lines : []; + const n = Math.max(1, Number(max || MAX_LINES_DEFAULT)); + return list.length > n ? list.slice(0, n) : list; +} + +/** + * Parse a downloaded timedtext payload regardless of its format. + * Tries `json3`, then `vtt`, then XML captions (`srv*`/`ttml`). + * HTML error pages always yield `[]`. + */ +export function parseTrackText(text, ext = '') { + if (looksLikeHtmlError(text)) return []; + const e = String(ext || '').toLowerCase(); + const asJson3 = () => { + try { return parseJson3(JSON.parse(String(text))); } catch { return []; } + }; + if (e === 'json3') { + const lines = asJson3(); + if (lines.length) return lines; + const vtt = parseVtt(text); + if (vtt.length) return vtt; + return parseXmlCaptions(text); + } + const vtt = parseVtt(text); + if (vtt.length) return vtt; + const j = asJson3(); + if (j.length) return j; + return parseXmlCaptions(text); +} + +/** Map short (`yt`) and long (`youtube`) provider ids to `providerUrlFrom()` names. */ +export function normalizeTranscriptProvider(provider) { + const p = String(provider || '').trim().toLowerCase(); + const map = { + yt: 'youtube', youtube: 'youtube', + dm: 'dailymotion', dailymotion: 'dailymotion', + tw: 'twitch', twitch: 'twitch', + pt: 'peertube', peertube: 'peertube', + od: 'odysee', odysee: 'odysee', + ru: 'rumble', rumble: 'rumble', + }; + return map[p] || null; +} + +export { trackExtOf as transcriptTrackExt, firstTrack as transcriptFirstTrack }; diff --git a/src/app/search/suggest.service.ts b/src/app/search/suggest.service.ts new file mode 100644 index 0000000..0e08de9 --- /dev/null +++ b/src/app/search/suggest.service.ts @@ -0,0 +1,72 @@ +import { Injectable, inject } from '@angular/core'; +import { HttpClient } from '@angular/common/http'; +import { Observable, of } from 'rxjs'; +import { tap } from 'rxjs/operators'; +import { dedupeSort } from './suggest.util'; + +export interface SuggestResponse { + q: string; + groups: Record; +} + +interface SuggestCacheEntry { + t: number; + data: SuggestResponse; +} + +/** + * Step 15 — Query typeahead service. + * + * Contract: no network while `q.trim().length < 2`. Callers debounce + * (250-300ms) + `switchMap` (in-flight cancellation). Responses are cached + * 5 minutes, deduped and sorted per provider group. + */ +@Injectable({ providedIn: 'root' }) +export class SuggestService { + private http = inject(HttpClient); + private cache = new Map(); + private cacheTtlMs = 5 * 60 * 1000; + + private apiBase(): string { + try { + const port = window?.location?.port || ''; + return port && port !== '4000' ? '/proxy/api' : '/api'; + } catch { + return '/api'; + } + } + + /** Single suggest round-trip (no debounce here — the caller owns it). */ + suggestOnce(q: string, providers?: string[] | 'all', limit = 10): Observable { + const query = String(q ?? '').trim(); + if (query.length < 2) return of({ q: query, groups: {} }); + const prov = Array.isArray(providers) && providers.length > 0 ? [...providers].sort().join(',') : 'all'; + const key = `${prov}|${query.toLowerCase()}|${limit}`; + const now = Date.now(); + const hit = this.cache.get(key); + if (hit && now - hit.t < this.cacheTtlMs) return of(hit.data); + + const params = new URLSearchParams({ q: query, limit: String(limit) }); + if (Array.isArray(providers) && providers.length > 0) params.set('providers', providers.join(',')); + return this.http.get(`${this.apiBase()}/search/suggest?${params.toString()}`).pipe( + tap((res) => { + try { + const groups: Record = {}; + for (const [pid, list] of Object.entries(res?.groups || {})) { + groups[pid] = dedupeSort(Array.isArray(list) ? list : [], query).slice(0, limit); + } + const clean: SuggestResponse = { q: res?.q ?? query, groups }; + if (this.cache.size > 200) this.cache.clear(); + this.cache.set(key, { t: Date.now(), data: clean }); + // Mutate in place so subscribers see the cleaned payload + res.q = clean.q; + res.groups = clean.groups; + } catch {} + }), + ); + } + + clearCache(): void { + this.cache.clear(); + } +} diff --git a/src/app/search/suggest.spec.ts b/src/app/search/suggest.spec.ts new file mode 100644 index 0000000..2783e80 --- /dev/null +++ b/src/app/search/suggest.spec.ts @@ -0,0 +1,103 @@ +import 'zone.js/node'; +import '@angular/compiler'; // JIT needed: decorators are evaluated at import time under ts-node +import { firstValueFrom, of } from 'rxjs'; +import { Injector, runInInjectionContext } from '@angular/core'; +import { HttpClient } from '@angular/common/http'; +import { SuggestService } from './suggest.service'; +import { dedupeSort, mergeGroups, highlightParts } from './suggest.util'; + +/** + * Step 15 — Unit tests for query typeahead (parsing / dedup / min-length guard). + * + * Same harness as Step 11: ts-node isolated transpilation, manual injection + * with a minimal HttpClient fake returning real Observables. + * Run with: npm run test:suggest + */ + +function assertEqual(actual: unknown, expected: unknown, message: string): void { + const a = JSON.stringify(actual); + const b = JSON.stringify(expected); + if (a !== b) { + throw new Error(`${message} (expected ${b}, got ${a})`); + } +} + +function assert(condition: unknown, message: string): void { + if (!condition) { + throw new Error(message); + } +} + +function logOk(msg: string): void { + console.log(`✓ ${msg}`); +} + +(async () => { + // --- Pure helpers: dedupe + sort --- + assertEqual( + dedupeSort(['Tutoriel Angular', 'tutoriel angular ', ' ', 'Tutoriel Android'], 'tutoriel an'), + ['Tutoriel Android', 'Tutoriel Angular'], + 'dedupeSort dedupes case-insensitively and sorts prefix-first then alpha', + ); + logOk('dedupeSort dedup + tri'); + + assertEqual(dedupeSort(['b', 'a', 'c']), ['a', 'b', 'c'], 'dedupeSort sorts alphabetically without query'); + logOk('dedupeSort alpha fallback'); + + // --- Pure helpers: merge groups --- + const merged = mergeGroups({ dm: ['Tutoriel android'], yt: ['Tutoriel Angular', 'tutoriel android'], xx: ['zz'] }); + assertEqual( + merged, + [ + { text: 'Tutoriel Angular', provider: 'yt' }, + { text: 'tutoriel android', provider: 'yt' }, + { text: 'zz', provider: 'xx' }, + ], + 'mergeGroups respects registry order, dedupes across providers, unknowns last', + ); + logOk('mergeGroups order + dedup'); + + // --- Pure helpers: highlight --- + assertEqual( + highlightParts('Tutoriel Angular', 'toriel an'), + { pre: 'Tu', match: 'toriel An', post: 'gular' }, + 'highlightParts splits on first case-insensitive occurrence', + ); + assertEqual( + highlightParts('Bonjour', 'zzz'), + { pre: 'Bonjour', match: '', post: '' }, + 'highlightParts without occurrence returns plain text', + ); + logOk('highlightParts'); + + // --- Service: min length guard (no network below 2 chars) --- + let httpCalls = 0; + const httpFake = { + get: (url: string) => { + httpCalls += 1; + assert(url.includes('/search/suggest'), `suggest hits /api/search/suggest (got ${url})`); + return of({ q: 'angular', groups: { yt: ['angular tutorial', 'Angular'], dm: ['angular tutorial'] } }); + }, + }; + const injector = Injector.create([{ provide: HttpClient, useValue: httpFake }]); + const svc: SuggestService = runInInjectionContext(injector, () => new SuggestService()); + + const short = await firstValueFrom(svc.suggestOnce('a')); + assertEqual(short, { q: 'a', groups: {} }, 'q < 2 chars returns empty without network'); + assertEqual(httpCalls, 0, 'no HTTP call for short queries'); + logOk('suggestOnce min-length guard (no network)'); + + // --- Service: dedup + cache --- + const first = await firstValueFrom(svc.suggestOnce('angular', ['yt', 'dm'])); + assertEqual(httpCalls, 1, 'one HTTP call for a fresh query'); + assertEqual(first.groups['yt'], ['Angular', 'angular tutorial'], 'per-provider groups deduped + sorted'); + const second = await firstValueFrom(svc.suggestOnce('angular', ['dm', 'yt'])); + assertEqual(httpCalls, 1, 'second identical query served from cache (provider order-insensitive)'); + assert(second === first || JSON.stringify(second) === JSON.stringify(first), 'cached payload matches'); + logOk('suggestOnce dedup + cache 5 min'); + + console.log('\nAll suggest tests passed.'); +})().catch((e) => { + console.error(e instanceof Error ? e.stack : String(e)); + process.exit(1); +}); diff --git a/src/app/search/suggest.util.ts b/src/app/search/suggest.util.ts new file mode 100644 index 0000000..ff24ef0 --- /dev/null +++ b/src/app/search/suggest.util.ts @@ -0,0 +1,72 @@ +/** + * Step 15 — Pure helpers for query typeahead (testable without Angular). + */ + +/** Normalize a suggestion string for dedup comparisons. */ +export function normSuggest(s: string): string { + return String(s ?? '').trim().toLowerCase().replace(/\s+/g, ' '); +} + +/** + * Deduplicate (case-insensitive, first occurrence wins) then sort: + * items starting with `q` come first, alphabetical within each group. + */ +export function dedupeSort(items: string[], q = ''): string[] { + const seen = new Set(); + const clean: string[] = []; + for (const raw of items || []) { + const t = String(raw ?? '').trim(); + if (!t) continue; + const k = normSuggest(t); + if (seen.has(k)) continue; + seen.add(k); + clean.push(t); + } + const nq = normSuggest(q); + clean.sort((a, b) => { + const na = normSuggest(a); + const nb = normSuggest(b); + const pa = nq ? (na.startsWith(nq) ? 0 : 1) : 0; + const pb = nq ? (nb.startsWith(nq) ? 0 : 1) : 0; + if (pa !== pb) return pa - pb; + return na < nb ? -1 : na > nb ? 1 : 0; + }); + return clean; +} + +/** + * Flatten provider groups into a single ordered list (registry order), + * deduped case-insensitively. Unknown provider keys go last (insertion order). + */ +export function mergeGroups( + groups: Record | null | undefined, + order: string[] = ['yt', 'dm', 'tw', 'pt', 'od', 'ru'], +): Array<{ text: string; provider: string }> { + const out: Array<{ text: string; provider: string }> = []; + const seen = new Set(); + const push = (pid: string, list: string[] | undefined) => { + for (const raw of list || []) { + const t = String(raw ?? '').trim(); + if (!t) continue; + const k = normSuggest(t); + if (seen.has(k)) continue; + seen.add(k); + out.push({ text: t, provider: pid }); + } + }; + for (const pid of order) push(pid, groups?.[pid]); + for (const pid of Object.keys(groups || {})) { + if (!order.includes(pid)) push(pid, (groups as Record)[pid]); + } + return out; +} + +/** Split `text` on the first case-insensitive occurrence of `q` for highlighting. */ +export function highlightParts(text: string, q: string): { pre: string; match: string; post: string } { + const t = String(text ?? ''); + const needle = String(q ?? '').trim(); + if (!needle) return { pre: t, match: '', post: '' }; + const idx = t.toLowerCase().indexOf(needle.toLowerCase()); + if (idx === -1) return { pre: t, match: '', post: '' }; + return { pre: t.slice(0, idx), match: t.slice(idx, idx + needle.length), post: t.slice(idx + needle.length) }; +} diff --git a/src/components/watch/watch.component.ts b/src/components/watch/watch.component.ts index bcbabde..5c72a97 100644 --- a/src/components/watch/watch.component.ts +++ b/src/components/watch/watch.component.ts @@ -23,6 +23,7 @@ import { formatAbsoluteFr, formatNumberFr } from '../../utils/date.util'; import { ProviderBadgeComponent } from '../../app/shared/components/provider-badge/provider-badge.component'; import { ChannelIdentityComponent } from '../../app/shared/components/channel-identity/channel-identity.component'; import { DurationPipe } from '../../app/shared/pipes/duration.pipe'; +import { UserService } from '../../services/user.service'; @Component({ selector: 'app-watch', @@ -44,6 +45,7 @@ export class WatchComponent implements OnDestroy, AfterViewInit { private auth = inject(AuthService); private http = inject(HttpClient); private iframeProgress = inject(IframeProgressService); + private users = inject(UserService); private channels = inject(ChannelsService); private subs = inject(SubscriptionsService); private routeSubscription!: Subscription; @@ -197,6 +199,8 @@ export class WatchComponent implements OnDestroy, AfterViewInit { // Build a direct Twitch URL as a fallback open-in-new-tab action twitchOpenUrl(): string | null { if (this.provider() !== 'twitch') return null; + const clip = this.twitchClip(); + if (clip) return `https://clips.twitch.tv/${encodeURIComponent(clip)}`; const ch = this.twitchChannel(); const id = this.videoId(); const cur = this.video(); @@ -286,6 +290,20 @@ export class WatchComponent implements OnDestroy, AfterViewInit { liked = signal(false); likeBusy = signal(false); + // --- Transcript state (Step 16, Phase 1: display + language selector, no seek) --- + transcriptOpen = signal(false); + transcriptLoading = signal(false); + transcriptAvailable = signal(false); + transcriptError = signal(null); + // Machine-readable unavailability reason from the API: 'no_subtitles' | + // 'provider_unsupported' | 'temporarily_unavailable' | null. + transcriptReason = signal(null); + // True when retrying may succeed (transient YouTube rate-limit/fetch failure). + transcriptRetryable = signal(false); + transcriptLanguages = signal([]); + transcriptLines = signal>([]); + selectedTranscriptLang = signal('fr'); + // --- Watch history tracking for native player --- watchHistoryId = signal(null); startPositionSeconds = signal(null); @@ -296,6 +314,7 @@ export class WatchComponent implements OnDestroy, AfterViewInit { private providerSel = signal(''); provider = computed(() => this.providerSel() || this.instances.selectedProvider()); private twitchChannel = signal(null); + private twitchClip = signal(null); // Backend-provided embed URL for providers that need it (e.g., Rumble) private rumbleEmbed = signal(null); @@ -303,9 +322,10 @@ export class WatchComponent implements OnDestroy, AfterViewInit { const id = this.videoId(); const p = this.provider(); const ch = this.twitchChannel(); + const clip = this.twitchClip(); const slug = this.odyseeSlug(); const start = Math.max(0, this.startPositionSeconds() || 0); - if (!id && !(p === 'twitch' && ch) && !(p === 'odysee' && slug)) return null; + if (!id && !(p === 'twitch' && (ch || clip)) && !(p === 'odysee' && slug)) return null; try { const host = (location && location.hostname) ? location.hostname : 'localhost'; if (p === 'youtube') { @@ -324,9 +344,15 @@ export class WatchComponent implements OnDestroy, AfterViewInit { return this.sanitizer.bypassSecurityTrustResourceUrl(u); } if (p === 'twitch') { - // Channel or VOD; Twitch requires one or more 'parent' params that match the embedding domain + // Channel, clip ou VOD ; Twitch exige 'parent' = domaine d'embed. const parentsSet = new Set([host, 'localhost', '127.0.0.1']); const parentParams = Array.from(parentsSet).map(h => `parent=${encodeURIComponent(h)}`).join('&'); + // Clips : player dédié clips.twitch.tv (le player VOD rejette les slugs). + const clipId = clip || (/^\d+$/.test(String(id || '')) ? null : ((this.video() as any)?.kind === 'clip' ? id : null)); + if (clipId) { + const u = `https://clips.twitch.tv/embed?clip=${encodeURIComponent(clipId)}&${parentParams}&autoplay=false`; + return this.sanitizer.bypassSecurityTrustResourceUrl(u); + } const base = 'https://player.twitch.tv/'; // ?video= exige un id de VOD numérique ; un login (non numérique) sans // ?channel= doit être traité comme une chaîne (résultats de recherche). @@ -461,11 +487,13 @@ export class WatchComponent implements OnDestroy, AfterViewInit { const slug = this.route.snapshot.queryParamMap.get('slug'); const p = (this.route.snapshot.queryParamMap.get('p') || providerFromPath) as Provider | null; const ch = this.route.snapshot.queryParamMap.get('channel'); + const clip = this.route.snapshot.queryParamMap.get('clip'); if (id) { this.videoId.set(id); this.odyseeSlug.set(slug); this.providerSel.set(p || ''); this.twitchChannel.set(ch); + this.twitchClip.set(clip); // Choisir un provider final (paramètre présent, segment d'URL, ou fallback depuis état courant) let finalProvider = p as Provider | null; @@ -583,6 +611,7 @@ export class WatchComponent implements OnDestroy, AfterViewInit { this.summaryError.set(null); this.selectedQuality.set(null); this.resetDownloadUi(); + this.resetTranscriptUi(); // Reset like state while loading this.liked.set(false); this.likeBusy.set(false); @@ -861,6 +890,189 @@ export class WatchComponent implements OnDestroy, AfterViewInit { this.selectedQuality.set(value || null); } + // --- Transcript logic (Step 16) --- + private resetTranscriptUi() { + this.transcriptOpen.set(false); + this.transcriptLoading.set(false); + this.transcriptAvailable.set(false); + this.transcriptError.set(null); + this.transcriptReason.set(null); + this.transcriptRetryable.set(false); + this.transcriptLanguages.set([]); + this.transcriptLines.set([]); + try { + const prefLang = String(this.users.preferences()?.language || '').trim().toLowerCase().slice(0, 12); + this.selectedTranscriptLang.set(prefLang || 'fr'); + } catch { + this.selectedTranscriptLang.set('fr'); + } + } + + toggleTranscript() { + const next = !this.transcriptOpen(); + this.transcriptOpen.set(next); + if (next && this.transcriptLines().length === 0 && !this.transcriptLoading()) { + this.loadTranscript(); + } + } + + onTranscriptLangChange(value: string) { + const lang = String(value || '').trim().slice(0, 12) || 'fr'; + this.selectedTranscriptLang.set(lang); + this.loadTranscript(); + } + + /** Map a backend transcript state (reason/error/retryable) to the UI: + * user-facing message + whether a "Réessayer" button makes sense. */ + private transcriptStatusFor(data: any): { message: string | null; retryable: boolean; reason: string | null } { + const reason = typeof data?.reason === 'string' ? data.reason : null; + const code = typeof data?.error === 'string' ? data.error : ''; + const retryable = data?.retryable === true + || code === 'transcript_temporarily_unavailable' + || code === 'rate_limited'; + if (reason === 'provider_unsupported') { + return { + message: 'Les transcripts ne sont pas pris en charge pour ce fournisseur (aucun sous-titre exposé).', + retryable: false, + reason, + }; + } + if (reason === 'temporarily_unavailable' || code === 'transcript_temporarily_unavailable') { + return { + message: 'Sous-titres temporairement indisponibles (limite YouTube). Réessayez dans quelques instants.', + retryable: true, + reason: reason || 'temporarily_unavailable', + }; + } + if (reason === 'no_subtitles') { + return { message: null, retryable: false, reason }; + } + // Legacy / unknown error code: sanitize, never leak HTML or parse noise. + const clean = this.transcriptDisplayError(code); + if (clean) return { message: `Transcript indisponible. (${clean})`, retryable, reason }; + if (retryable) { + return { + message: 'Sous-titres temporairement indisponibles. Réessayez dans quelques instants.', + retryable: true, + reason: reason || 'temporarily_unavailable', + }; + } + return { message: null, retryable: false, reason }; + } + + /** Sanitize a backend error code for display: never leak HTML pages or raw + * JSON-parse noise (e.g. proxy / CDN "; + const params = new URLSearchParams({ lang: this.selectedTranscriptLang() }); + if (opts['instance']) params.set('instance', opts['instance']); + if (opts['slug']) params.set('slug', opts['slug']); + const url = `${this.apiBase()}/transcript/${encodeURIComponent(p)}/${encodeURIComponent(id)}?${params.toString()}`; + this.http.get(url).subscribe({ + next: (data) => { + this.transcriptLoading.set(false); + this.transcriptAvailable.set(!!data?.available); + this.transcriptLanguages.set(Array.isArray(data?.languages) ? data.languages : []); + this.transcriptLines.set(Array.isArray(data?.lines) ? data.lines : []); + if (data?.available && typeof data?.lang === 'string' && data.lang) { + this.selectedTranscriptLang.set(data.lang); + } + if (!data?.available) { + const st = this.transcriptStatusFor(data); + this.transcriptReason.set(st.reason); + this.transcriptRetryable.set(st.retryable); + this.transcriptError.set(st.message); + } else { + this.transcriptReason.set(null); + this.transcriptRetryable.set(false); + this.transcriptError.set(null); + } + }, + error: (err) => { + this.transcriptLoading.set(false); + this.transcriptAvailable.set(false); + this.transcriptLines.set([]); + const body = err?.error; + if (body && typeof body === 'object' && Array.isArray(body.languages)) { + this.transcriptLanguages.set(body.languages); + } + const raw = + (body && typeof body === 'object' && (body.error || body.details || body.reason)) || + (typeof body === 'string' ? body : '') || + err?.message || + ''; + const status = err?.status; + if (status === 429 || String(raw || '').trim() === 'rate_limited') { + this.transcriptReason.set('temporarily_unavailable'); + this.transcriptRetryable.set(true); + this.transcriptError.set('Trop de requêtes. Réessayez dans une minute.'); + return; + } + const st = this.transcriptStatusFor( + body && typeof body === 'object' ? body : { error: raw }, + ); + this.transcriptReason.set(st.reason); + this.transcriptRetryable.set(st.retryable); + this.transcriptError.set(st.message); + }, + }); + } catch { + this.transcriptLoading.set(false); + this.transcriptAvailable.set(false); + } + } + // --- Download logic --- openDownloadPanel() { this.downloadOpen.set(true); @@ -1053,6 +1265,10 @@ export class WatchComponent implements OnDestroy, AfterViewInit { if (slug.startsWith('/')) slug = slug.slice(1); qp.slug = slug; } + if (p === 'twitch' && String((v as any).kind || '').toLowerCase() === 'clip') { + qp.clip = v.videoId; + return qp; + } if (p === 'twitch' && (v.type === 'channel' || (v as any).type === 'live')) { // Prefer explicit videoId as channel login; fallback to parsing URL if (v.videoId && !/^\d+$/.test(String(v.videoId))) { @@ -1072,18 +1288,64 @@ export class WatchComponent implements OnDestroy, AfterViewInit { private loadRelatedSuggestions(): void { const cur = this.video(); if (!cur) return; - const title = (cur.title || '').trim(); const maxItems = 12; - if (title) { - this.apiService.searchVideosPage(title).subscribe(res => { - const items = (res.items || []).filter(v => v.videoId !== cur.videoId).slice(0, maxItems); - this.video.update(v => v ? { ...v, relatedStreams: items } : v); - }); - } else { - this.apiService.getTrendingPage().subscribe(res => { - const items = (res.items || []).filter(v => v.videoId !== cur.videoId).slice(0, maxItems); - this.video.update(v => v ? { ...v, relatedStreams: items } : v); - }); - } + const apply = (items: any[]) => { + // Ignore les réponses arrivées après une navigation (race). + try { + if ((this.videoId() || '') !== (cur.videoId || '')) return; + } catch {} + const list = (items || []).filter((v: any) => (v?.videoId || v?.id) !== cur.videoId).slice(0, maxItems); + this.video.update(v => v ? { ...v, relatedStreams: list } : v); + }; + const titleFallback = () => { + const title = (cur.title || '').trim(); + if (title) { + this.apiService.searchVideosPage(title).subscribe(res => { + apply(res.items || []); + }); + } else { + this.apiService.getTrendingPage().subscribe(res => { + apply(res.items || []); + }); + } + }; + // YouTube : vrai watch-next InnerTube via /api/details (0 quota, 0 recherche + // supplémentaire) — façon SmartTube. Autres providers : fallback existant. + try { + if (this.provider() === 'youtube' && cur.videoId) { + const id = cur.videoId; + this.http.get(`/api/details/youtube/${encodeURIComponent(id)}`).subscribe({ + next: (data: any) => { + const rel = Array.isArray(data?.related) ? data.related : []; + if (rel.length > 0) { + apply(rel.map((it: any) => ({ + url: it.url || `https://www.youtube.com/watch?v=${it.id}`, + type: 'video', + title: it.title || '', + thumbnail: it.thumbnail || '', + uploaderName: it.uploaderName || '', + uploaderAvatar: '', + channelId: it.channelId || undefined, + channelHandle: it.channelHandle || undefined, + channelExternalId: it.channelExternalId || it.channelId || undefined, + uploadedDate: it.publishedAt || '', + duration: typeof it.duration === 'number' ? it.duration : 0, + views: typeof it.views === 'number' ? it.views : 0, + uploaded: 0, + videoId: String(it.id || ''), + provider: 'youtube', + isShort: it.isShort === true ? true : undefined, + isLive: it.isLive === true ? true : undefined, + } as Video))); + } else { + titleFallback(); + } + }, + error: () => titleFallback(), + }); + return; + } + } catch {} + titleFallback(); } } diff --git a/todo.md b/todo.md index 5513cc8..3993649 100644 --- a/todo.md +++ b/todo.md @@ -17,6 +17,179 @@ - [x] Step 12: Write basic e2e scenarios for providers filtering and deep-link - [x] Step 13: Add minimal telemetry hooks and events - [x] Step 14: Update README with search UX section, keyboard shortcuts and usage notes +- [x] Step 15: Auto-complétion de la **requête** (typeahead) dans la barre de recherche +- [x] Step 16: **Transcripts** de vidéos sur la page Watch (API + UI, Phase 1 du doc) + +--- + +## Step 15 — Suggestions de requêtes dans la barre de recherche + +### Ce que ça veut dire (et ce que ce n'est pas) + +Quand l'utilisateur tape `tutoriel an`, la barre affiche **avant qu'il valide** une liste de +requêtes complètes probables : `tutoriel angular`, `tutoriel android`… + ses recherches passées. + +> ⚠️ À ne pas confondre avec `SearchSuggestionsComponent` (`src/components/search/search-suggestions.component.ts`), +> qui lui affiche les **résultats de recherche** groupés par provider **après** la validation sur `/search`. +> Le Step 15 ajoute un nouveau panneau de **suggestions de texte** sous l'``. + +### Ce qui existe déjà (à réutiliser) + +| Élément | Fichier | État | +|---|---|---| +| `@Output() searchChange` (query debouncée 300 ms) | `src/components/search/search-box.component.ts:46` | Émis mais **aucun abonné** — le header ne branche que `(submitted)` (`header.component.html:19`) | +| Popover `@provider` + navigation clavier | `search-box.component.ts:16` (`AT_QUERY_RE`), `.html:56-76` | ✅ mais ne couvre que les ids de providers | +| Historique de recherches | `HistoryService.getSearchHistory(n)` | ✅ utilisé dans le Quick Menu | +| Cache in-memory TTL 60 s | `src/app/search/search.service.ts:33` | Pattern à dupliquer | +| Fan-out multi-providers | `GET /api/search` (`server/index.mjs:2116`) | Pattern à dupliquer | + +### Livrables + +1. **Backend** — `GET /api/search/suggest?q=…&providers=yt,dm&limit=10` + → `{ q, groups: { yt: string[], dm: string[], … } }` + - Un `suggest(q, { limit })` optionnel par handler dans `server/providers/*.mjs` + déclaré dans + `server/providers/registry.mjs` (`ProviderAdapter`). + - Providers sans API de suggestions native → `[]` (dégradation propre, jamais d'erreur 500). + - Cache + rate-limit sur le même modèle que `/api/search`. +2. **Front service** — `SuggestService` (ou méthode `suggest()` sur `SearchService`) : + `min length 2`, `debounce 250-300 ms`, `switchMap` (annulation de la requête précédente), + cache 5 min, dédoublonnage + tri. +3. **UI** — panneau sous l'input, sections : + - 🔎 « Recherches récentes » (History, avec icône horloge) + - 📊 Suggestions providers, groupées par provider (badges `YT/DM/…`) ou fusionnées + - Sous-chaîne commune mise en évidence +4. **Interactions** — clic = remplir + soumettre ; ↑/↓/Enter/Tab/Esc ; **priorité au popover `@`** + quand il est ouvert ; `aria-listbox` + `aria-activedescendant` (le `role="combobox"` est déjà sur l'input). +5. **Télémétrie** — `suggest_shown`, `suggest_used` (à ajouter à la whitelist serveur de l'étape 13). +6. **Tests** — `npm run test:suggest` (unitaires parsing/dédup) + scénario e2e : `providers=yt` ne renvoie + que des suggestions `yt`, provider inconnu → fallback registry complet. + +### Critères d'acceptation + +- [x] Aucune requête réseau tant que `q.trim().length < 2` +- [x] frappe rapide = une seule requête en vol (pas d'empilement / de réponse qui écrase la dernière) +- [x] Esc ferme le panneau sans effacer le texte ; Tab complète sans lancer la recherche +- [x] Un provider sans suggestions ou en échec n'affiche ni trou ni erreur dans le panneau +- [x] Focus clavier et lecteurs d'écran cohérents avec l'existant (a11y Step 10) + +--- + +## Step 16 — Transcripts (Phase 1 du `docs/transcript_architecture_dev.md`) + +### Principe directeur + +> **Un endpoint, un module de parsing, une UI.** La disponibilité dépend de la plateforme, mais le +> **code ne change pas** selon le provider (le doc appelle ça « Rung 2 : réutiliser, pas réinventer »). + +Source des données : `yt-dlp --dump-single-json --skip-download` (déjà utilisé côté serveur, ex. +`server/index.mjs:1012`) → on lit `subtitles` / `automatic_captions` → on télécharge la piste `json3` ou `vtt` +→ on normalise en `{ lang, available, languages, lines: [{ t, dur, text }] }`. + +### Périmètre Phase 1 (à faire) + +**Backend** + +- [x] `server/transcript.mjs` : fonctions **pures** `pickTrack(json, lang)`, `parseJson3(data)`, `parseVtt(text)` + (testables sans réseau ni DB) +- [x] Route `GET /api/transcript/:provider/:videoId?lang=&instance=&slug=&sourceUrl=` dans `server/index.mjs` + (`instance/slug/sourceUrl` nécessaires pour PeerTube multi-instances ; réutiliser `providerUrlFrom()`). +- [x] `transcriptCache` : `Map` + TTL long (24 h), clé `transcript:${provider}:${videoId}:${lang}`. +- [x] Rate-limit (~10 req/min/IP) sur le motif `channelsLimiter`. +- [x] Aucun sous-titre → **HTTP 200** `{ available: false }` (pas une erreur). +- [x] Gestion 429 persistant + fallback « temp dir » (`writeAutoSub`) et variables d'env + (`TRANSCRIPT_CACHE_TTL`, `TRANSCRIPT_RATE_LIMIT`, `TRANSCRIPT_FALLBACK_TMP`). + +**Frontend (variante UI #1 du doc)** + +- [x] Bouton « Transcript » à côté du bouton Download (`watch.component.html:159`). +- [x] Panneau copié sur le motif du Download Panel (`watch.component.html:185`) : + `toggleTranscript()` + `loadTranscript()`, chargement/absent/lines, `