feat(perf): chantier B — instantanés SQLite et rafraîchisseur d'arrière-plan
- Table snapshots + écriture traversante du cache client, hydratation au démarrage (SWR : servi + rafraîchi en fond). - Boucle de rafraîchissement (120 s par défaut) : index, étoiles, issues all+open et jalons des projets actifs ; webhook = rafraîchissement immédiat du dépôt ; fraîcheur affichée.
This commit is contained in:
@@ -0,0 +1,116 @@
|
||||
"""Background snapshot refresher — chantier B (0.22.0).
|
||||
|
||||
The expensive part of Kanbante is the fan-out of Gitea list
|
||||
calls behind the portfolio pages (repo index, then issues and
|
||||
milestones for every active project). Since 0.19.0 those calls
|
||||
are parallel and stale-while-revalidate, which fixed repeat
|
||||
loads, but the FIRST page after a restart still paid seconds.
|
||||
|
||||
The refresher pays that cost instead of the user: every
|
||||
KANBANTE_SNAPSHOT_INTERVAL seconds it force-refreshes the hot
|
||||
keys (repo index, starred, issues + milestones of active
|
||||
repos). Every fetch write-throughs to the SQLite `snapshots`
|
||||
table, and the client cache is hydrated from that table at
|
||||
startup — so pages read local data from the very first request
|
||||
and only ever wait on Gitea when no snapshot exists yet.
|
||||
|
||||
Freshness contract, shown in the UI: Gitea-side data may lag
|
||||
by up to one interval; a Gitea webhook refreshes the touched
|
||||
repo immediately (see refresh_repo_snapshots); anything the
|
||||
user does locally (board journal) is always instant.
|
||||
"""
|
||||
|
||||
import asyncio
|
||||
import datetime
|
||||
import logging
|
||||
|
||||
from app import models
|
||||
from app.config import get_settings
|
||||
|
||||
logger = logging.getLogger("kanbante.snapshots")
|
||||
|
||||
_REFRESH_CONCURRENCY = 8
|
||||
|
||||
|
||||
async def refresh_repo_snapshots(gitea, owner: str, repo: str) -> None:
|
||||
"""Refresh one repo's hot keys (webhooks call this).
|
||||
|
||||
Both issue states are refreshed: most pages read "all", but
|
||||
Focus/Triage/Mes issues read "open" — a snapshot set missing
|
||||
either leaves a slow first hit on the table.
|
||||
"""
|
||||
try:
|
||||
await gitea.refresh_repo_issues(owner, repo, "all")
|
||||
await gitea.refresh_repo_issues(owner, repo, "open")
|
||||
await gitea.refresh_milestones(owner, repo)
|
||||
except Exception as exc:
|
||||
logger.debug("repo snapshot refresh failed for %s/%s: %s", owner, repo, exc)
|
||||
|
||||
|
||||
async def refresh_snapshots(gitea, classify_repos) -> int:
|
||||
"""One full refresh cycle. Returns the active-repo count."""
|
||||
repos = await gitea.refresh_search_repos()
|
||||
try:
|
||||
await gitea.refresh_starred()
|
||||
except Exception as exc:
|
||||
logger.debug("starred refresh failed: %s", exc)
|
||||
active, _inactive, _archived = classify_repos(repos)
|
||||
semaphore = asyncio.Semaphore(_REFRESH_CONCURRENCY)
|
||||
|
||||
async def _one(repo: dict) -> None:
|
||||
owner, _, name = repo["full_name"].partition("/")
|
||||
async with semaphore:
|
||||
await refresh_repo_snapshots(gitea, owner, name)
|
||||
|
||||
await asyncio.gather(*(_one(repo) for repo in active))
|
||||
logger.info("snapshot refresh complete (%d active repos)", len(active))
|
||||
return len(active)
|
||||
|
||||
|
||||
async def snapshot_loop(gitea, classify_repos) -> None:
|
||||
"""Periodic refresher, started by the app lifespan.
|
||||
|
||||
Sleeps briefly first: hydration already serves the pages,
|
||||
the loop only has to keep the snapshots warm.
|
||||
"""
|
||||
interval = get_settings().snapshot_interval
|
||||
if interval <= 0:
|
||||
logger.info("snapshot refresher disabled (interval <= 0)")
|
||||
return
|
||||
await asyncio.sleep(10)
|
||||
while True:
|
||||
try:
|
||||
await refresh_snapshots(gitea, classify_repos)
|
||||
except Exception as exc:
|
||||
logger.warning("snapshot refresh cycle failed: %s", exc)
|
||||
await asyncio.sleep(interval)
|
||||
|
||||
|
||||
def freshness_label() -> str | None:
|
||||
"""« à l'instant » / « il y a N min » from the freshest snapshot.
|
||||
|
||||
Honest freshness display (chantier B contract): the sidebar
|
||||
shows how old the Gitea data on screen can be.
|
||||
"""
|
||||
try:
|
||||
latest = models.latest_snapshot_at()
|
||||
except Exception:
|
||||
return None
|
||||
if not latest:
|
||||
return None
|
||||
try:
|
||||
fetched = datetime.datetime.strptime(
|
||||
latest, "%Y-%m-%d %H:%M:%S"
|
||||
).replace(tzinfo=datetime.timezone.utc)
|
||||
except ValueError:
|
||||
return None
|
||||
age = (
|
||||
datetime.datetime.now(datetime.timezone.utc) - fetched
|
||||
).total_seconds()
|
||||
if age < 90:
|
||||
return "à l'instant"
|
||||
if age < 3600:
|
||||
return f"il y a {int(age // 60)} min"
|
||||
if age < 86400:
|
||||
return f"il y a {int(age // 3600)} h"
|
||||
return f"il y a {int(age // 86400)} j"
|
||||
Reference in New Issue
Block a user