Initial commit: 25 scripts Python sauvegardés
Date: 2026-08-28 11:43:34 EDT
This commit is contained in:
@@ -1,3 +1,25 @@
|
|||||||
# hermes-scripts-backup
|
# Hermes Scripts Backup
|
||||||
|
|
||||||
Backup des scripts Python de Hermes skills — auto-commit automatique
|
## Purpose
|
||||||
|
Sauvegarde automatique de tous les scripts Python des compétences Hermes Agent.
|
||||||
|
|
||||||
|
## Structure
|
||||||
|
```
|
||||||
|
skills/
|
||||||
|
productivity/
|
||||||
|
ocr-and-documents/scripts/extract_pymupdf.py
|
||||||
|
google-workspace/scripts/google_api.py
|
||||||
|
linear/scripts/linear_api.py
|
||||||
|
creative/
|
||||||
|
comfyui/tests/test_extract_schema.py
|
||||||
|
bin/
|
||||||
|
restore-skill-scripts.sh — Restaure les scripts depuis ce backup
|
||||||
|
scripts/
|
||||||
|
validate-scripts.py — Valide l'intégrité des scripts
|
||||||
|
```
|
||||||
|
|
||||||
|
## Auto-Commit
|
||||||
|
Les scripts sont sauvegardés automatiquement à chaque modification via un cron job Hermes.
|
||||||
|
|
||||||
|
## Last Updated
|
||||||
|
$(date '+%Y-%m-%d %H:%M:%S %Z')
|
||||||
|
|||||||
@@ -0,0 +1,171 @@
|
|||||||
|
---
|
||||||
|
name: heartmula
|
||||||
|
description: "HeartMuLa: Suno-like song generation from lyrics + tags."
|
||||||
|
version: 1.0.0
|
||||||
|
platforms: [linux, macos, windows]
|
||||||
|
metadata:
|
||||||
|
hermes:
|
||||||
|
tags: [music, audio, generation, ai, heartmula, heartcodec, lyrics, songs]
|
||||||
|
related_skills: [audiocraft]
|
||||||
|
---
|
||||||
|
|
||||||
|
# HeartMuLa - Open-Source Music Generation
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
HeartMuLa is a family of open-source music foundation models (Apache-2.0) that generates music conditioned on lyrics and tags, with multilingual support. Generates full songs from lyrics + tags. Comparable to Suno for open-source. Includes:
|
||||||
|
- **HeartMuLa** - Music language model (3B/7B) for generation from lyrics + tags
|
||||||
|
- **HeartCodec** - 12.5Hz music codec for high-fidelity audio reconstruction
|
||||||
|
- **HeartTranscriptor** - Whisper-based lyrics transcription
|
||||||
|
- **HeartCLAP** - Audio-text alignment model
|
||||||
|
|
||||||
|
## When to Use
|
||||||
|
- User wants to generate music/songs from text descriptions
|
||||||
|
- User wants an open-source Suno alternative
|
||||||
|
- User wants local/offline music generation
|
||||||
|
- User asks about HeartMuLa, heartlib, or AI music generation
|
||||||
|
|
||||||
|
## Hardware Requirements
|
||||||
|
- **Minimum**: 8GB VRAM with `--lazy_load true` (loads/unloads models sequentially)
|
||||||
|
- **Recommended**: 16GB+ VRAM for comfortable single-GPU usage
|
||||||
|
- **Multi-GPU**: Use `--mula_device cuda:0 --codec_device cuda:1` to split across GPUs
|
||||||
|
- 3B model with lazy_load peaks at ~6.2GB VRAM
|
||||||
|
|
||||||
|
## Installation Steps
|
||||||
|
|
||||||
|
### 1. Clone Repository
|
||||||
|
```bash
|
||||||
|
cd ~/ # or desired directory
|
||||||
|
git clone https://github.com/HeartMuLa/heartlib.git
|
||||||
|
cd heartlib
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Create Virtual Environment (Python 3.10 required)
|
||||||
|
```bash
|
||||||
|
uv venv --python 3.10 .venv
|
||||||
|
. .venv/bin/activate
|
||||||
|
uv pip install -e .
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Fix Dependency Compatibility Issues
|
||||||
|
|
||||||
|
**IMPORTANT**: As of Feb 2026, the pinned dependencies have conflicts with newer packages. Apply these fixes:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Upgrade datasets (old version incompatible with current pyarrow)
|
||||||
|
uv pip install --upgrade datasets
|
||||||
|
|
||||||
|
# Upgrade transformers (needed for huggingface-hub 1.x compatibility)
|
||||||
|
uv pip install --upgrade transformers
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Patch Source Code (Required for transformers 5.x)
|
||||||
|
|
||||||
|
**Patch 1 - RoPE cache fix** in `src/heartlib/heartmula/modeling_heartmula.py`:
|
||||||
|
|
||||||
|
In the `setup_caches` method of the `HeartMuLa` class, add RoPE reinitialization after the `reset_caches` try/except block and before the `with device:` block:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Re-initialize RoPE caches that were skipped during meta-device loading
|
||||||
|
from torchtune.models.llama3_1._position_embeddings import Llama3ScaledRoPE
|
||||||
|
for module in self.modules():
|
||||||
|
if isinstance(module, Llama3ScaledRoPE) and not module.is_cache_built:
|
||||||
|
module.rope_init()
|
||||||
|
module.to(device)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why**: `from_pretrained` creates model on meta device first; `Llama3ScaledRoPE.rope_init()` skips cache building on meta tensors, then never rebuilds after weights are loaded to real device.
|
||||||
|
|
||||||
|
**Patch 2 - HeartCodec loading fix** in `src/heartlib/pipelines/music_generation.py`:
|
||||||
|
|
||||||
|
Add `ignore_mismatched_sizes=True` to ALL `HeartCodec.from_pretrained()` calls (there are 2: the eager load in `__init__` and the lazy load in the `codec` property).
|
||||||
|
|
||||||
|
**Why**: VQ codebook `initted` buffers have shape `[1]` in checkpoint vs `[]` in model. Same data, just scalar vs 0-d tensor. Safe to ignore.
|
||||||
|
|
||||||
|
### 5. Download Model Checkpoints
|
||||||
|
```bash
|
||||||
|
cd heartlib # project root
|
||||||
|
hf download --local-dir './ckpt' 'HeartMuLa/HeartMuLaGen'
|
||||||
|
hf download --local-dir './ckpt/HeartMuLa-oss-3B' 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year'
|
||||||
|
hf download --local-dir './ckpt/HeartCodec-oss' 'HeartMuLa/HeartCodec-oss-20260123'
|
||||||
|
```
|
||||||
|
|
||||||
|
All 3 can be downloaded in parallel. Total size is several GB.
|
||||||
|
|
||||||
|
## GPU / CUDA
|
||||||
|
|
||||||
|
HeartMuLa uses CUDA by default (`--mula_device cuda --codec_device cuda`). No extra setup needed if the user has an NVIDIA GPU with PyTorch CUDA support installed.
|
||||||
|
|
||||||
|
- The installed `torch==2.4.1` includes CUDA 12.1 support out of the box
|
||||||
|
- `torchtune` may report version `0.4.0+cpu` — this is just package metadata, it still uses CUDA via PyTorch
|
||||||
|
- To verify GPU is being used, look for "CUDA memory" lines in the output (e.g. "CUDA memory before unloading: 6.20 GB")
|
||||||
|
- **No GPU?** You can run on CPU with `--mula_device cpu --codec_device cpu`, but expect generation to be **extremely slow** (potentially 30-60+ minutes for a single song vs ~4 minutes on GPU). CPU mode also requires significant RAM (~12GB+ free). If the user has no NVIDIA GPU, recommend using a cloud GPU service (Google Colab free tier with T4, Lambda Labs, etc.) or the online demo at https://heartmula.github.io/ instead.
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
### Basic Generation
|
||||||
|
```bash
|
||||||
|
cd heartlib
|
||||||
|
. .venv/bin/activate
|
||||||
|
python ./examples/run_music_generation.py \
|
||||||
|
--model_path=./ckpt \
|
||||||
|
--version="3B" \
|
||||||
|
--lyrics="./assets/lyrics.txt" \
|
||||||
|
--tags="./assets/tags.txt" \
|
||||||
|
--save_path="./assets/output.mp3" \
|
||||||
|
--lazy_load true
|
||||||
|
```
|
||||||
|
|
||||||
|
### Input Formatting
|
||||||
|
|
||||||
|
**Tags** (comma-separated, no spaces):
|
||||||
|
```
|
||||||
|
piano,happy,wedding,synthesizer,romantic
|
||||||
|
```
|
||||||
|
or
|
||||||
|
```
|
||||||
|
rock,energetic,guitar,drums,male-vocal
|
||||||
|
```
|
||||||
|
|
||||||
|
**Lyrics** (use bracketed structural tags):
|
||||||
|
```
|
||||||
|
[Intro]
|
||||||
|
|
||||||
|
[Verse]
|
||||||
|
Your lyrics here...
|
||||||
|
|
||||||
|
[Chorus]
|
||||||
|
Chorus lyrics...
|
||||||
|
|
||||||
|
[Bridge]
|
||||||
|
Bridge lyrics...
|
||||||
|
|
||||||
|
[Outro]
|
||||||
|
```
|
||||||
|
|
||||||
|
### Key Parameters
|
||||||
|
| Parameter | Default | Description |
|
||||||
|
|-----------|---------|-------------|
|
||||||
|
| `--max_audio_length_ms` | 240000 | Max length in ms (240s = 4 min) |
|
||||||
|
| `--topk` | 50 | Top-k sampling |
|
||||||
|
| `--temperature` | 1.0 | Sampling temperature |
|
||||||
|
| `--cfg_scale` | 1.5 | Classifier-free guidance scale |
|
||||||
|
| `--lazy_load` | false | Load/unload models on demand (saves VRAM) |
|
||||||
|
| `--mula_dtype` | bfloat16 | Dtype for HeartMuLa (bf16 recommended) |
|
||||||
|
| `--codec_dtype` | float32 | Dtype for HeartCodec (fp32 recommended for quality) |
|
||||||
|
|
||||||
|
### Performance
|
||||||
|
- RTF (Real-Time Factor) ≈ 1.0 — a 4-minute song takes ~4 minutes to generate
|
||||||
|
- Output: MP3, 48kHz stereo, 128kbps
|
||||||
|
|
||||||
|
## Pitfalls
|
||||||
|
1. **Do NOT use bf16 for HeartCodec** — degrades audio quality. Use fp32 (default).
|
||||||
|
2. **Tags may be ignored** — known issue (#90). Lyrics tend to dominate; experiment with tag ordering.
|
||||||
|
3. **Triton not available on macOS** — Linux/CUDA only for GPU acceleration.
|
||||||
|
4. **RTX 5080 incompatibility** reported in upstream issues.
|
||||||
|
5. The dependency pin conflicts require the manual upgrades and patches described above.
|
||||||
|
|
||||||
|
## Links
|
||||||
|
- Repo: https://github.com/HeartMuLa/heartlib
|
||||||
|
- Models: https://huggingface.co/HeartMuLa
|
||||||
|
- Paper: https://arxiv.org/abs/2601.10547
|
||||||
|
- License: Apache-2.0
|
||||||
@@ -0,0 +1,193 @@
|
|||||||
|
---
|
||||||
|
name: kanban-orchestrator
|
||||||
|
description: Decomposition playbook + anti-temptation rules for an orchestrator profile routing work through Kanban. The "don't do the work yourself" rule and the basic lifecycle are auto-injected into every kanban worker's system prompt; this skill is the deeper playbook when you're specifically playing the orchestrator role.
|
||||||
|
version: 3.0.0
|
||||||
|
platforms: [linux, macos, windows]
|
||||||
|
metadata:
|
||||||
|
hermes:
|
||||||
|
tags: [kanban, multi-agent, orchestration, routing]
|
||||||
|
related_skills: [kanban-worker]
|
||||||
|
---
|
||||||
|
|
||||||
|
# Kanban Orchestrator — Decomposition Playbook
|
||||||
|
|
||||||
|
> The **core worker lifecycle** (including the `kanban_create` fan-out pattern and the "decompose, don't execute" rule) is auto-injected into every kanban process via the `KANBAN_GUIDANCE` system-prompt block. This skill is the deeper playbook when you're an orchestrator profile whose whole job is routing.
|
||||||
|
|
||||||
|
## Profiles are user-configured — not a fixed roster
|
||||||
|
|
||||||
|
Hermes setups vary widely. Some users run a single profile that does everything; some run a small fleet (`docker-worker`, `cron-worker`); some run a curated specialist team they've named themselves. There is **no default specialist roster** — the orchestrator skill does not know what profiles exist on this machine.
|
||||||
|
|
||||||
|
Before fanning out, you must ground the decomposition in the profiles that actually exist. The dispatcher silently fails to spawn unknown assignee names — it doesn't autocorrect, doesn't suggest, doesn't fall back. So a card assigned to `researcher` on a setup that only has `docker-worker` just sits in `ready` forever.
|
||||||
|
|
||||||
|
**Step 0: discover available profiles before planning.**
|
||||||
|
|
||||||
|
Use one of these:
|
||||||
|
|
||||||
|
- `hermes profile list` — prints the table of profiles configured on this machine. Run it through your terminal tool if you have one; otherwise ask the user.
|
||||||
|
- `kanban_list(assignee="<some-name>")` — sanity-check a single name. Returns an empty list (rather than an error) for an unknown assignee, so this only confirms a name you're already considering.
|
||||||
|
- **Just ask the user.** "What profiles do you have set up?" is a fine first turn when the goal needs more than one specialist.
|
||||||
|
|
||||||
|
Cache the result in your working memory for the rest of the conversation. Re-asking every turn wastes a tool call.
|
||||||
|
|
||||||
|
## When to use the board (vs. just doing the work)
|
||||||
|
|
||||||
|
Create Kanban tasks when any of these are true:
|
||||||
|
|
||||||
|
1. **Multiple specialists are needed.** Research + analysis + writing is three profiles.
|
||||||
|
2. **The work should survive a crash or restart.** Long-running, recurring, or important.
|
||||||
|
3. **The user might want to interject.** Human-in-the-loop at any step.
|
||||||
|
4. **Multiple subtasks can run in parallel.** Fan-out for speed.
|
||||||
|
5. **Review / iteration is expected.** A reviewer profile loops on drafter output.
|
||||||
|
6. **The audit trail matters.** Board rows persist in SQLite forever.
|
||||||
|
|
||||||
|
If *none* of those apply — it's a small one-shot reasoning task — use `delegate_task` instead or answer the user directly.
|
||||||
|
|
||||||
|
## The anti-temptation rules
|
||||||
|
|
||||||
|
Your job description says "route, don't execute." The rules that enforce that:
|
||||||
|
|
||||||
|
- **Do not execute the work yourself.** Your restricted toolset usually doesn't even include terminal/file/code/web for implementation. If you find yourself "just fixing this quickly" — stop and create a task for the right specialist.
|
||||||
|
- **For any concrete task, create a Kanban task and assign it.** Every single time.
|
||||||
|
- **Split multi-lane requests before creating cards.** A user prompt can contain several independent workstreams. Extract those lanes first, then create one card per lane instead of bundling unrelated work into a single implementer card.
|
||||||
|
- **Run independent lanes in parallel.** If two cards do not need each other's output, leave them unlinked so the dispatcher can fan them out. Link only true data dependencies.
|
||||||
|
- **Never create dependent work as independent ready cards.** If a card must wait for another card, pass `parents=[...]` in the original `kanban_create` call. Do not create it first and link it later, and do not rely on prose like "wait for T1" inside the body.
|
||||||
|
- **If no specialist fits the available profiles, ask the user which profile to create or which existing profile to use.** Do not invent profile names; the dispatcher will silently drop unknown assignees.
|
||||||
|
- **Decompose, route, and summarize — that's the whole job.**
|
||||||
|
|
||||||
|
## Decomposition playbook
|
||||||
|
|
||||||
|
### Step 1 — Understand the goal
|
||||||
|
|
||||||
|
Ask clarifying questions if the goal is ambiguous. Cheap to ask; expensive to spawn the wrong fleet.
|
||||||
|
|
||||||
|
### Step 2 — Sketch the task graph
|
||||||
|
|
||||||
|
Before creating anything, draft the graph out loud (in your response to the user). Treat every concrete workstream as a candidate card:
|
||||||
|
|
||||||
|
1. Extract the lanes from the request.
|
||||||
|
2. Map each lane to one of the profiles you discovered in Step 0. If a lane doesn't fit any existing profile, ask the user which to use or create.
|
||||||
|
3. Decide whether each lane is independent or gated by another lane.
|
||||||
|
4. Create independent lanes as parallel cards with no parent links.
|
||||||
|
5. Create synthesis/review/integration cards with parent links to the lanes they depend on. A child created with unfinished parents starts in `todo`; the dispatcher promotes it to `ready` only after every parent is done.
|
||||||
|
|
||||||
|
Examples of prompts that should fan out (using placeholder profile names — substitute whatever exists on the user's setup):
|
||||||
|
|
||||||
|
- "Build an app" → one card to a design-oriented profile for product/UI direction, one or two cards to engineering profiles for implementation, plus a later integration/review card if the user has a reviewer profile.
|
||||||
|
- "Fix blockers and check model variants" → one implementation card for the blocker fixes plus one discovery/research card for config/source verification. A final reviewer card can depend on both.
|
||||||
|
- "Research docs and implement" → a docs-research card can run in parallel with a codebase-discovery card; implementation waits only if it truly needs those findings.
|
||||||
|
- "Analyze this screenshot and find the related code" → one card to a vision-capable profile for the visual analysis while another searches the codebase.
|
||||||
|
|
||||||
|
Words like "also," "finally," or "and" do not automatically imply a dependency. They often mean "make sure this is covered before reporting back." Only link tasks when one card cannot start until another card's output exists.
|
||||||
|
|
||||||
|
Show the graph to the user before creating cards. Let them correct it — including which actual profile name should own each lane.
|
||||||
|
|
||||||
|
### Step 3 — Create tasks and link
|
||||||
|
|
||||||
|
Use the profile names from Step 0. The example below uses placeholders `<profile-A>`, `<profile-B>`, `<profile-C>` — replace them with what the user actually has.
|
||||||
|
|
||||||
|
```python
|
||||||
|
t1 = kanban_create(
|
||||||
|
title="research: Postgres cost vs current",
|
||||||
|
assignee="<profile-A>", # whichever profile handles research on this setup
|
||||||
|
body="Compare estimated infrastructure costs, migration costs, and ongoing ops costs over a 3-year window. Sources: AWS/GCP pricing, team time estimates, current Postgres bills from peers.",
|
||||||
|
tenant=os.environ.get("HERMES_TENANT"),
|
||||||
|
)["task_id"]
|
||||||
|
|
||||||
|
t2 = kanban_create(
|
||||||
|
title="research: Postgres performance vs current",
|
||||||
|
assignee="<profile-A>", # same profile, run in parallel
|
||||||
|
body="Compare query latency, throughput, and scaling characteristics at our expected data volume (~500GB, 10k QPS peak). Sources: benchmark papers, public case studies, pgbench results if easy.",
|
||||||
|
)["task_id"]
|
||||||
|
|
||||||
|
t3 = kanban_create(
|
||||||
|
title="synthesize migration recommendation",
|
||||||
|
assignee="<profile-B>", # whichever profile does synthesis/analysis
|
||||||
|
body="Read the findings from T1 (cost) and T2 (performance). Produce a 1-page recommendation with explicit trade-offs and a go/no-go call.",
|
||||||
|
parents=[t1, t2],
|
||||||
|
)["task_id"]
|
||||||
|
|
||||||
|
t4 = kanban_create(
|
||||||
|
title="draft decision memo",
|
||||||
|
assignee="<profile-C>", # whichever profile drafts user-facing prose
|
||||||
|
body="Turn the analyst's recommendation into a 2-page memo for the CTO. Match the tone of previous decision memos in the team's knowledge base.",
|
||||||
|
parents=[t3],
|
||||||
|
)["task_id"]
|
||||||
|
```
|
||||||
|
|
||||||
|
`parents=[...]` gates promotion — children stay in `todo` until every parent reaches `done`, then auto-promote to `ready`. No manual coordination needed; the dispatcher and dependency engine handle it.
|
||||||
|
|
||||||
|
If the task graph has dependencies, create the parent cards first, capture their returned ids, and include those ids in the child card's `parents` list during the child `kanban_create` call. Avoid creating all cards in parallel and linking them afterward; that creates a window where the dispatcher can claim a child before its inputs exist.
|
||||||
|
|
||||||
|
### Step 4 — Complete your own task
|
||||||
|
|
||||||
|
If you were spawned as a task yourself (e.g. a planner profile was assigned `T0: "investigate Postgres migration"`), mark it done with a summary of what you created:
|
||||||
|
|
||||||
|
```python
|
||||||
|
kanban_complete(
|
||||||
|
summary="decomposed into T1-T4: 2 research lanes in parallel, 1 synthesis on their outputs, 1 prose draft on the recommendation",
|
||||||
|
metadata={
|
||||||
|
"task_graph": {
|
||||||
|
"T1": {"assignee": "<profile-A>", "parents": []},
|
||||||
|
"T2": {"assignee": "<profile-A>", "parents": []},
|
||||||
|
"T3": {"assignee": "<profile-B>", "parents": ["T1", "T2"]},
|
||||||
|
"T4": {"assignee": "<profile-C>", "parents": ["T3"]},
|
||||||
|
},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 5 — Report back to the user
|
||||||
|
|
||||||
|
Tell them what you created in plain prose, naming the actual profiles you used:
|
||||||
|
|
||||||
|
> I've queued 4 tasks:
|
||||||
|
> - **T1** (`<profile-A>`): cost comparison
|
||||||
|
> - **T2** (`<profile-A>`): performance comparison, in parallel with T1
|
||||||
|
> - **T3** (`<profile-B>`): synthesizes T1 + T2 into a recommendation
|
||||||
|
> - **T4** (`<profile-C>`): turns T3 into a CTO memo
|
||||||
|
>
|
||||||
|
> The dispatcher will pick up T1 and T2 now. T3 starts when both finish. You'll get a gateway ping when T4 completes. Use the dashboard or `hermes kanban tail <id>` to follow along.
|
||||||
|
|
||||||
|
## Common patterns
|
||||||
|
|
||||||
|
**Fan-out + fan-in (research → synthesize):** N research-style cards with no parents, one synthesis card with all of them as parents.
|
||||||
|
|
||||||
|
**Parallel implementation + validation:** one implementer card makes the change while one explorer/researcher card verifies config, docs, or source mapping. A reviewer card can depend on both. Do not make the implementer own unrelated verification just because the user mentioned both in one sentence.
|
||||||
|
|
||||||
|
**Pipeline with gates:** `planner → implementer → reviewer`. Each stage's `parents=[previous_task]`. Reviewer blocks or completes; if reviewer blocks, the operator unblocks with feedback and respawns.
|
||||||
|
|
||||||
|
**Same-profile queue:** N tasks, all assigned to the same profile, no dependencies between them. Dispatcher serializes — that profile processes them in priority order, accumulating experience in its own memory.
|
||||||
|
|
||||||
|
**Human-in-the-loop:** Any task can `kanban_block()` to wait for input. Dispatcher respawns after `/unblock`. The comment thread carries the full context.
|
||||||
|
|
||||||
|
## Pitfalls
|
||||||
|
|
||||||
|
**Inventing profile names that don't exist.** The dispatcher silently fails to spawn unknown assignees — the card just sits in `ready` forever. Always assign to a profile from your Step 0 discovery; ask the user if you're unsure.
|
||||||
|
|
||||||
|
**Unassigned tasks (assignee is None).** Same symptom as above — the dispatcher only picks up tasks with a valid assignee matching an active profile. A task created without `assignee=`, or one whose assignee was cleared to `None`, will never leave `ready`. Always verify with `kanban_show` or a direct DB query (`SELECT assignee FROM tasks WHERE id=?`) that the task actually has a non-null assignee. See `references/cli-recovery.md` for the fix recipe.
|
||||||
|
|
||||||
|
**CLI `kanban assign` can fail silently.** Outside the hermes-agent venv, `python3 -m hermes_cli.kanban assign <id> <profile>` may return exit 0 with no output and no effect — the module import fails but the error is swallowed. Prefer using `execute_code` with a direct `kanban_db.assign_task()` call, setting `HERMES_KANBAN_DB` explicitly. See `references/cli-recovery.md` for the canonical snippet.
|
||||||
|
|
||||||
|
**Bundling independent lanes into one card.** If the user asks for two independent outcomes, create two cards. Example: "fix blockers and check model variants" is not one fixer task; create a fixer/engineer card for the fixes and an explorer/researcher card for the variant check, then optionally gate review on both.
|
||||||
|
|
||||||
|
**Over-linking because of wording.** "Finally check X" may still be parallel with implementation if X is static config, docs, or source discovery. Link it after implementation only when the check depends on the implementation result.
|
||||||
|
|
||||||
|
**Forgetting dependency links.** If the task graph says `research -> implement -> review`, do not create all tasks as independent ready cards. Use parent links so implement/review cannot run before their inputs exist.
|
||||||
|
|
||||||
|
**Reassignment vs. new task.** If a reviewer blocks with "needs changes," create a NEW task linked from the reviewer's task — don't re-run the same task with a stern look. The new task is assigned to the original implementer profile.
|
||||||
|
|
||||||
|
**Argument order for links.** `kanban_link(parent_id=..., child_id=...)` — parent first. Mixing them up demotes the wrong task to `todo`.
|
||||||
|
|
||||||
|
**Don't pre-create the whole graph if the shape depends on intermediate findings.** If T3's structure depends on what T1 and T2 find, let T3 exist as a "synthesize findings" task whose own first step is to read parent handoffs and plan the rest. Orchestrators can spawn orchestrators.
|
||||||
|
|
||||||
|
**Tenant inheritance.** If `HERMES_TENANT` is set in your env, pass `tenant=os.environ.get("HERMES_TENANT")` on every `kanban_create` call so child tasks stay in the same namespace.
|
||||||
|
|
||||||
|
## Recovering stuck workers
|
||||||
|
|
||||||
|
When a worker profile keeps crashing, hallucinating, or getting blocked by its own mistakes (usually: wrong model, missing skill, broken credential), the kanban dashboard flags the task with a ⚠ badge and opens a **Recovery** section in the drawer. Three primary actions:
|
||||||
|
|
||||||
|
1. **Reclaim** (or `hermes kanban reclaim <task_id>`) — abort the running worker immediately and reset the task to `ready`. The existing claim TTL is ~15 min; this is the fast path out.
|
||||||
|
2. **Reassign** (or `hermes kanban reassign <task_id> <new-profile> --reclaim`) — switch the task to a different profile (one that exists on this setup) and let the dispatcher pick it up with a fresh worker.
|
||||||
|
3. **Change profile model** — the dashboard prints a copy-paste hint for `hermes -p <profile> model` since profile config lives on disk; edit it in a terminal, then Reclaim to retry with the new model.
|
||||||
|
|
||||||
|
Hallucination warnings appear on tasks where a worker's `kanban_complete(created_cards=[...])` claim included card ids that don't exist or weren't created by the worker's profile (the gate blocks the completion), or where the free-form summary references `t_<hex>` ids that don't resolve (advisory prose scan, non-blocking). Both produce audit events that persist even after recovery actions — the trail stays for debugging.
|
||||||
@@ -0,0 +1,326 @@
|
|||||||
|
# Instructions for *ACL Proceedings
|
||||||
|
|
||||||
|
The following instructions are for authors of papers submitted for review to ACL conferences (hereafter, "review version") or paper accepted for publication in its proceedings (hereafter, "final version").
|
||||||
|
All authors are required to adhere to these specifications.
|
||||||
|
|
||||||
|
## Style Files
|
||||||
|
|
||||||
|
*ACL provides style files for LaTeX and Microsoft Word that meet these requirements. They can be found at:
|
||||||
|
|
||||||
|
> https://acl-org.github.io/ACLPUB/
|
||||||
|
|
||||||
|
We strongly recommend the use of these style files, which have been appropriately tailored for the *ACL proceedings.
|
||||||
|
|
||||||
|
## Paper Length
|
||||||
|
|
||||||
|
The conference accepts submissions of long papers and short papers.
|
||||||
|
Review versions of long papers may have up to eight (8) pages of content plus unlimited pages for references.
|
||||||
|
Upon acceptance, final versions of long papers will be given one additional page -- up to nine (9) pages of content plus unlimited pages for acknowledgements and references -- so that reviewers' comments can be taken into account.
|
||||||
|
Review versions of short papers may have up to four (4) pages of content, plus unlimited pages for references.
|
||||||
|
Final versions of short papers may have up to five (5) pages, plus unlimited pages for acknowledgements and references.
|
||||||
|
For both long and short papers, all figures and tables that are part of the main text must fit within these page limits.
|
||||||
|
|
||||||
|
The conference encourages submission of appendices and supplementary material, which are not required to fit within these page limits. However, review versions of papers must be self-contained: it is optional for reviewers to look at appendices or supplementary material. Please see [Appendices](#Appendices) and [Supplementary](#Supplementary Material) for more information.
|
||||||
|
|
||||||
|
Review versions should not refer, for further detail, to documents, code or data resources that are not available to the reviewers.
|
||||||
|
|
||||||
|
Papers that do not conform to these requirements may be rejected without review.
|
||||||
|
|
||||||
|
Workshop chairs may have different rules for allowed length and whether appendices or supplementary materials are welcome.
|
||||||
|
As always, the respective call for papers is the authoritative source.
|
||||||
|
|
||||||
|
## Anonymity
|
||||||
|
|
||||||
|
As reviewing will be double-blind, review versions must not include any identifying information about the authors (such as names, affiliations, or URLs).
|
||||||
|
Self-references that reveal the author's identity, e.g.,
|
||||||
|
|
||||||
|
> We previously showed (Gusfield, 1997)...
|
||||||
|
|
||||||
|
must be avoided, and anonymous citations, e.g.,
|
||||||
|
|
||||||
|
> We previously showed (Anonymous, 1997)...
|
||||||
|
|
||||||
|
should also be avoided. Instead, use citations such as
|
||||||
|
|
||||||
|
> Gusfield (1997) previously showed...
|
||||||
|
|
||||||
|
Review versions must not include acknowledgements.
|
||||||
|
|
||||||
|
**Papers that do not conform to these requirements may be rejected without review.**
|
||||||
|
|
||||||
|
Any preliminary non-archival versions of submitted papers should be listed in the submission form but not in the review version of the paper.
|
||||||
|
Reviewers are generally aware that authors may present preliminary versions of their work in other venues, but will not be provided the list of previous presentations from the submission form.
|
||||||
|
|
||||||
|
Once a paper has been accepted to the conference, the final version should include the author's names and affiliations, and is allowed to use self-references.
|
||||||
|
|
||||||
|
## Multiple Submission
|
||||||
|
|
||||||
|
Papers that have been or will be submitted to other meetings or publications must indicate this at submission time in the START submission form, and must be withdrawn from the other venues if accepted by *ACL.
|
||||||
|
Authors of papers accepted for presentation at *ACL must notify the program chairs by the deadline for final versions ("camera-ready deadline") whether the paper will be presented.
|
||||||
|
We will not accept for publication or presentation any papers that overlap significantly in content or results with papers that will be (or have been) published elsewhere.
|
||||||
|
|
||||||
|
Authors submitting more than one paper to *ACL must ensure that submissions do not overlap significantly (>25%) with each other in content or results.
|
||||||
|
|
||||||
|
## Formatting Instructions
|
||||||
|
|
||||||
|
### File Format
|
||||||
|
|
||||||
|
Papers must be in Adobe Portable Document Format (PDF).
|
||||||
|
Please make sure that your PDF file embeds all necessary fonts (especially for tree diagrams, symbols, and Asian languages).
|
||||||
|
When you print or create the PDF file, there is usually an option in your printer setup to include none, all or just non-standard fonts.
|
||||||
|
Please make sure that you select the option of including *all* the fonts.
|
||||||
|
**Before sending it, test your PDF by printing it from a computer different from the one where it was created.**
|
||||||
|
|
||||||
|
Some word processors may generate very large PDF files, where each page is rendered as an image.
|
||||||
|
Such images may reproduce poorly.
|
||||||
|
In this case, try alternative ways to obtain the PDF.
|
||||||
|
|
||||||
|
All papers must use **A4 paper format** (21 cm x 29.7 cm).
|
||||||
|
Papers must not be submitted with any other paper size.
|
||||||
|
|
||||||
|
If you cannot meet the above requirements, please contact the publication chairs as soon as possible.
|
||||||
|
|
||||||
|
### Layout
|
||||||
|
|
||||||
|
All text except for page numbers must fit within the margins.
|
||||||
|
|
||||||
|
Review versions should have page numbers, centered in the bottom margin, but **pages should not be numbered in the final version.**
|
||||||
|
|
||||||
|
Manuscripts must be set in two columns.
|
||||||
|
Exceptions to the two-column format include the title, authors' names and complete addresses, which must be centered at the top of the first page, and any full-width figures or tables.
|
||||||
|
|
||||||
|
The exact dimensions for a page on A4 paper are:
|
||||||
|
|
||||||
|
* Left margin: 2.5 cm
|
||||||
|
* Right margin: 2.5 cm
|
||||||
|
* Top margin: 2.5 cm
|
||||||
|
* Bottom margin: 2.5 cm
|
||||||
|
* Column width: 7.7 cm
|
||||||
|
* Column height: 24.7 cm
|
||||||
|
* Gap between columns: 0.6 cm
|
||||||
|
|
||||||
|
In the review version, a ruler (line numbers in the left and right margins of the article) should be printed, so that reviewers may comment on particular lines in the paper.
|
||||||
|
The ruler should not change the appearance of any other content on the page.
|
||||||
|
The final version should not contain a ruler.
|
||||||
|
|
||||||
|
### Fonts
|
||||||
|
|
||||||
|
All text (except non-Latin scripts and mathematical formulas) should be set in **Times Roman**.
|
||||||
|
If Times Roman is unavailable, you may use **Times New Roman** or **Computer Modern Roman.**
|
||||||
|
|
||||||
|
The following table specifies what font sizes and styles must be used for each type of text in the manuscript.
|
||||||
|
|
||||||
|
| Type of Text | Font Size | Style |
|
||||||
|
| --------------------- | --------- | ----- |
|
||||||
|
| paper title | 15 pt | bold |
|
||||||
|
| author names | 12 pt | bold |
|
||||||
|
| author affiliation | 12 pt | |
|
||||||
|
| the word ``Abstract'' | 12 pt | bold |
|
||||||
|
| section titles | 12 pt | bold |
|
||||||
|
| subsection titles | 11 pt | bold |
|
||||||
|
| document text | 11 pt | |
|
||||||
|
| captions | 10 pt | |
|
||||||
|
| abstract text | 10 pt | |
|
||||||
|
| bibliography | 10 pt | |
|
||||||
|
| footnotes | 9 pt | |
|
||||||
|
|
||||||
|
### Title and Authors
|
||||||
|
|
||||||
|
Center the title, author's name(s) and affiliation(s) across both columns.
|
||||||
|
|
||||||
|
Place the title centered at the top of the first page, in 15-point bold.
|
||||||
|
Long titles should be typed on two lines without a blank line intervening.
|
||||||
|
Put the title 2.5 cm from the top of the page.
|
||||||
|
Write the title in [title case](https://apastyle.apa.org/style-grammar-guidelines/capitalization/title-case); do not write the title in all capital letters, except for acronyms (e.g., "BLEU") or proper nouns ("English") that are normally uppercased or capitalized.
|
||||||
|
|
||||||
|
Place the author name(s) and affiliation(s) under the title.
|
||||||
|
Write authors' full names; do not abbreviate given names to initials, unless they are normally written as initials ("Margaret Mitchell", not "M. Mitchell").
|
||||||
|
Do not format surnames in all capitals ("Mitchell", not "MITCHELL").
|
||||||
|
|
||||||
|
Do not use footnotes for affiliations.
|
||||||
|
The affiliation should contain the author's complete address, and if possible, an electronic mail address.
|
||||||
|
|
||||||
|
The title, author names and addresses should be completely identical to those entered to the paper submission website in order to maintain the consistency of author information among all publications of the conference.
|
||||||
|
If they are different, the publication chairs may resolve the difference without consulting with you; so it is in your own interest to double-check that the information is consistent.
|
||||||
|
|
||||||
|
Start the body of the first page 7.5 cm from the top of the page.
|
||||||
|
**Even in the review version of the paper, you should maintain space for names and addresses so that they will fit in the final version.**
|
||||||
|
|
||||||
|
### Abstract
|
||||||
|
|
||||||
|
Type the abstract at the beginning of the first column.
|
||||||
|
Center the word **Abstract** in 12 point bold above the body of the abstract.
|
||||||
|
The width of the abstract should be smaller than the
|
||||||
|
normal column width by 0.6 cm on each side.
|
||||||
|
The abstract text should be 10 point roman, single-spaced.
|
||||||
|
|
||||||
|
The abstract should be a concise summary of the general thesis and conclusions of the paper.
|
||||||
|
It should be no longer than 200 words.
|
||||||
|
|
||||||
|
### Text
|
||||||
|
|
||||||
|
Begin typing the main body of the text immediately after the abstract, continuing in two columns.
|
||||||
|
The text should be 11 point roman, single-spaced.
|
||||||
|
|
||||||
|
Indent 0.4 cm when starting a new paragraph, except for the first paragraph in a section.
|
||||||
|
|
||||||
|
### Sections
|
||||||
|
|
||||||
|
Use numbered sections (Arabic numerals) to facilitate cross references.
|
||||||
|
Number subsections with the section number and the subsection number separated by a dot, in Arabic numerals, e.g.,
|
||||||
|
|
||||||
|
> 1 Introduction
|
||||||
|
|
||||||
|
or
|
||||||
|
|
||||||
|
> 6.1 File Format
|
||||||
|
|
||||||
|
### Footnotes
|
||||||
|
Put footnotes at the bottom of the page and use 9 point font.
|
||||||
|
They may be numbered or referred to by asterisks or other symbols.
|
||||||
|
Footnotes should be separated from the text by a line.
|
||||||
|
|
||||||
|
### Figures and tables
|
||||||
|
|
||||||
|
Place figures and tables in the paper near where they are first discussed, rather than at the end, if possible.
|
||||||
|
Wide figures/tables may run across both columns.
|
||||||
|
|
||||||
|
To accommodate people who are color-blind (as well as those printing with black-and-white printers), grayscale readability is strongly encouraged.
|
||||||
|
Color is not forbidden, but authors should ensure that tables and figures do not rely solely on color to convey critical distinctions.
|
||||||
|
|
||||||
|
**Captions:**
|
||||||
|
Provide a caption for every figure/table; number each one sequentially in the form:
|
||||||
|
|
||||||
|
> Figure 1: Caption of the Figure.
|
||||||
|
|
||||||
|
and
|
||||||
|
|
||||||
|
> Table 1: Caption of the Table.
|
||||||
|
|
||||||
|
Captions should be placed below figures/tables, in 10 point roman type.
|
||||||
|
Captions that are one line are centered.
|
||||||
|
Captions longer than one line are left-aligned.
|
||||||
|
|
||||||
|
### Hyperlinks
|
||||||
|
|
||||||
|
Within-document and external hyperlinks should be dark blue (hex #000099), not underlined or boxed.
|
||||||
|
|
||||||
|
### Non-English Text
|
||||||
|
|
||||||
|
Text in languages other than English should be accompanied by translations into English, and text in scripts other than Latin should \emph{also} be accompanied by transliterations into Latin script, since not all readers can recognize non-Latin characters easily.
|
||||||
|
|
||||||
|
For example, παράδειγμα *paradeigma* ‘example’ is a Greek word, and this is a Greek sentence:
|
||||||
|
|
||||||
|
> Αυτό είναι ένα παράδειγμα.
|
||||||
|
> auto einai ena paradeigma.
|
||||||
|
> ‘This is an example.’
|
||||||
|
|
||||||
|
### Citations
|
||||||
|
|
||||||
|
Citations within the text appear in parentheses (Gusfield, 1997), or, if the author's name appears in the text itself: Gusfield (1997).
|
||||||
|
Append lowercase letters to the year in cases of ambiguities.
|
||||||
|
Cite papers with two authors using both authors' names (Aho and Ullman, 1972), but cite papers with more than two authors by the first author's name and ``et al.'' (Chandra et al., 1981).
|
||||||
|
Collapse multiple citations into a single pair of parentheses (Gusfield, 1997; Aho and Ullman, 1972).
|
||||||
|
|
||||||
|
Refrain from using full citations as sentence constituents.
|
||||||
|
Instead of
|
||||||
|
|
||||||
|
> (Gusfield, 1997) showed that ...
|
||||||
|
> In (Gusfield, 1997), ...''
|
||||||
|
|
||||||
|
write
|
||||||
|
|
||||||
|
> Gusfield (1997) showed that ...
|
||||||
|
> In Gusfield (1997), ...
|
||||||
|
|
||||||
|
Submissions should accurately reference prior and related work, including code and data.
|
||||||
|
If a piece of prior work appeared in multiple venues, the version that appeared in a refereed, archival venue should be referenced.
|
||||||
|
If multiple versions of a piece of prior work exist, the one used by the authors should be referenced.
|
||||||
|
|
||||||
|
### Acknowledgments
|
||||||
|
|
||||||
|
The acknowledgments should go immediately before the references.
|
||||||
|
Do not number the acknowledgments section.
|
||||||
|
Do not include this section in the review version.
|
||||||
|
|
||||||
|
### References
|
||||||
|
|
||||||
|
Gather the full set of references together under the unnumbered section heading **References**.
|
||||||
|
Place the References section before any Appendices.
|
||||||
|
Arrange the references alphabetically by first author, rather than by order of occurrence in the text.
|
||||||
|
|
||||||
|
Provide as complete a citation as possible, using a consistent format, such as the [one for Computational Linguistics](http://cljournal.org/style_guide_refs.html) or the one in the [Publication Manual of the American Psychological Association](https://apastyle.apa.org/products/publication-manual-7th-edition).
|
||||||
|
Use full names for authors, not just initials.
|
||||||
|
Authors should not rely on automated citation indices to provide accurate references for prior and related work.
|
||||||
|
|
||||||
|
As part of our work to make ACL materials more widely used and cited outside of our discipline, ACL has registered as a CrossRef member, as a registrant of Digital Object Identifiers (DOIs), the standard for registering permanent URNs for referencing scholarly materials.
|
||||||
|
|
||||||
|
All references are required to contain DOIs of all cited works when possible, or, as a second resort, links to ACL Anthology pages.
|
||||||
|
Appropriate records should be found for most materials in the current [ACL Anthology](https://aclweb.org/anthology/).
|
||||||
|
|
||||||
|
Example article in a journal:
|
||||||
|
|
||||||
|
> Rie Kubota Ando and Tong Zhang. 2005. [A framework for learning predictive structures from multiple tasks and unlabeled data](https://www.jmlr.org/papers/v6/ando05a.html). *Journal of Machine Learning Research*, 6:1817–1853.
|
||||||
|
|
||||||
|
Example paper in non-ACL proceedings, with DOI:
|
||||||
|
|
||||||
|
> Galen Andrew and Jianfeng Gao. 2007. [Scalable training of L1-regularized log-linear models](https://doi.org/10.1145/1273496.1273501). In *Proceedings of the 24th International Conference on Machine Learning*, pages 33–40.
|
||||||
|
|
||||||
|
Example ACL Anthology paper with DOI:
|
||||||
|
|
||||||
|
> James Goodman, Andreas Vlachos, and Jason Naradowsky. 2016. [Noise reduction and targeted exploration in imitation learning for Abstract Meaning Representation parsing](http://dx.doi.org/10.18653/v1/P16-1001). In *Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1–45711, Berlin, Germany. Association for Computational Linguistics.
|
||||||
|
|
||||||
|
Example ACL Anthology paper without DOI:
|
||||||
|
|
||||||
|
> Benjamin Börschinger and Mark Johnson. 2011. [A particle filter algorithm for Bayesian word segmentation](https://www.aclweb.org/anthology/U11-1004/). In *Proceedings of the Australasian Language Technology Association Workshop 2011*, pages 10–44718, Canberra, Australia.
|
||||||
|
|
||||||
|
Example arXiv paper:
|
||||||
|
|
||||||
|
> Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. [Yara parser: A fast and accurate dependency parser](http://arxiv.org/abs/1503.06733). *Computing Research Repository*, arXiv:1503.06733. Version 2.
|
||||||
|
|
||||||
|
## Appendices
|
||||||
|
|
||||||
|
Appendices are material that can be read, and include lemmas, formulas, proofs, and tables that are not critical to the reading and understanding of the paper.
|
||||||
|
Letter them in sequence and provide an informative title:
|
||||||
|
|
||||||
|
> Appendix A. Title of Appendix
|
||||||
|
|
||||||
|
The appendices come after the references.
|
||||||
|
|
||||||
|
Review versions of appendices must follow the same anonymity guidelines as the main paper.
|
||||||
|
|
||||||
|
## Supplementary Material
|
||||||
|
|
||||||
|
Submissions may include non-readable supplementary material used in the work and described in the paper.
|
||||||
|
Any accompanying software and/or data should include licenses and documentation of research review as appropriate.
|
||||||
|
Supplementary material may report preprocessing decisions, model parameters, and other details necessary for the replication of the experiments reported in the paper.
|
||||||
|
Seemingly small preprocessing decisions can sometimes make a large difference in performance, so it is crucial to record such decisions to precisely characterize state-of-the-art methods.
|
||||||
|
|
||||||
|
Nonetheless, supplementary material should be supplementary (rather than central) to the paper.
|
||||||
|
**Submissions that misuse the supplementary material may be rejected without review.**
|
||||||
|
Supplementary material may include explanations or details of proofs or derivations that do not fit into the paper, lists of features or feature templates, sample inputs and outputs for a system, pseudo-code or source code, and data.
|
||||||
|
(Source code and data should be separate uploads, rather than part of the paper).
|
||||||
|
|
||||||
|
The paper should not rely on the supplementary material: while the paper may refer to and cite the supplementary material and the supplementary material will be available to the reviewers, they will not be asked to review the supplementary material.
|
||||||
|
|
||||||
|
Review versions of supplementary material must follow the same anonymity guidelines as the main paper.
|
||||||
|
|
||||||
|
## Credits
|
||||||
|
|
||||||
|
This document has been adapted from the instructions for earlier ACL and NAACL proceedings, including those for
|
||||||
|
ACL 2020 by Steven Bethard, Ryan Cotterell and Rui Yan,
|
||||||
|
ACL 2019 by Douwe Kiela and Ivan Ivan Vulić,
|
||||||
|
NAACL 2019 by Stephanie Lukin and Alla Roskovskaya,
|
||||||
|
ACL 2018 by Shay Cohen, Kevin Gimpel, and Wei Lu,
|
||||||
|
NAACL 2018 by Margaret Mitchell and Stephanie Lukin,
|
||||||
|
BibTeX suggestions for (NA)ACL 2017/2018 from Jason Eisner,
|
||||||
|
ACL 2017 by Dan Gildea and Min-Yen Kan,
|
||||||
|
NAACL 2017 by Margaret Mitchell,
|
||||||
|
ACL 2012 by Maggie Li and Michael White,
|
||||||
|
ACL 2010 by Jing-Shin Chang and Philipp Koehn,
|
||||||
|
ACL 2008 by Johanna D. Moore, Simone Teufel, James Allan, and Sadaoki Furui,
|
||||||
|
ACL 2005 by Hwee Tou Ng and Kemal Oflazer,
|
||||||
|
ACL 2002 by Eugene Charniak and Dekang Lin,
|
||||||
|
and earlier ACL and EACL formats written by several people, including
|
||||||
|
John Chen, Henry S. Thompson and Donald Walker.
|
||||||
|
Additional elements were taken from the formatting instructions of the *International Joint Conference on Artificial Intelligence* and the *Conference on Computer Vision and Pattern Recognition*.
|
||||||
+437
@@ -0,0 +1,437 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
check_deps.py — Verify a ComfyUI workflow's dependencies (custom nodes, models,
|
||||||
|
embeddings) against a running server.
|
||||||
|
|
||||||
|
Improvements over v1:
|
||||||
|
- Cloud-aware endpoint mapping (handles `/api/experiment/models/{folder}` and
|
||||||
|
`/api/object_info` variants verified against live cloud API)
|
||||||
|
- Distinguishes 200-empty (genuinely no models in folder) vs 404
|
||||||
|
(folder doesn't exist) vs 403 (auth/tier issue) — no silent passes
|
||||||
|
- Outputs concrete remediation commands (e.g. `comfy node install <name>`)
|
||||||
|
when nodes are missing
|
||||||
|
- Detects embedding references inside prompt strings as model deps
|
||||||
|
- Skips check on cloud free tier `/api/object_info` (403) without false alarm
|
||||||
|
- Accepts API key from CLI flag OR $COMFY_CLOUD_API_KEY env var
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 check_deps.py workflow_api.json
|
||||||
|
python3 check_deps.py workflow_api.json --host 127.0.0.1 --port 8188
|
||||||
|
python3 check_deps.py workflow_api.json --host https://cloud.comfy.org
|
||||||
|
|
||||||
|
Stdlib-only. Python 3.10+.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||||
|
from _common import ( # noqa: E402
|
||||||
|
DEFAULT_LOCAL_HOST, ENV_API_KEY,
|
||||||
|
emit_json, folder_aliases_for, http_get, is_cloud_host,
|
||||||
|
iter_embedding_refs, iter_model_deps, iter_nodes, parse_model_list,
|
||||||
|
resolve_api_key, resolve_url, unwrap_workflow,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# Known node → custom-node-package map. When a workflow needs a node we don't
|
||||||
|
# recognize, suggesting the right `comfy node install ...` makes the difference
|
||||||
|
# between a working agent and a stuck one.
|
||||||
|
NODE_TO_PACKAGE: dict[str, str] = {
|
||||||
|
# rgthree (Reroute is JS-only and doesn't appear in /object_info)
|
||||||
|
"Power Lora Loader (rgthree)": "rgthree-comfy",
|
||||||
|
"Image Comparer (rgthree)": "rgthree-comfy",
|
||||||
|
"Seed (rgthree)": "rgthree-comfy",
|
||||||
|
"Display Any (rgthree)": "rgthree-comfy",
|
||||||
|
"Display Int (rgthree)": "rgthree-comfy",
|
||||||
|
# Impact pack
|
||||||
|
"FaceDetailer": "comfyui-impact-pack",
|
||||||
|
"DetailerForEach": "comfyui-impact-pack",
|
||||||
|
"BboxDetectorSEGS": "comfyui-impact-pack",
|
||||||
|
"SAMLoader": "comfyui-impact-pack",
|
||||||
|
"ImpactWildcardProcessor": "comfyui-impact-pack",
|
||||||
|
# Impact subpack (separate package)
|
||||||
|
"UltralyticsDetectorProvider": "comfyui-impact-subpack",
|
||||||
|
# Was Node Suite
|
||||||
|
"Image Save": "was-node-suite-comfyui",
|
||||||
|
"Number Counter": "was-node-suite-comfyui",
|
||||||
|
"Text String": "was-node-suite-comfyui",
|
||||||
|
# easy-use
|
||||||
|
"easy fullLoader": "comfyui-easy-use",
|
||||||
|
"easy positive": "comfyui-easy-use",
|
||||||
|
"easy negative": "comfyui-easy-use",
|
||||||
|
"easy seed": "comfyui-easy-use",
|
||||||
|
"easy imageSave": "comfyui-easy-use",
|
||||||
|
# Video Helper Suite
|
||||||
|
"VHS_VideoCombine": "comfyui-videohelpersuite",
|
||||||
|
"VHS_LoadVideo": "comfyui-videohelpersuite",
|
||||||
|
"VHS_LoadAudio": "comfyui-videohelpersuite",
|
||||||
|
# AnimateDiff
|
||||||
|
"ADE_AnimateDiffLoaderWithContext": "comfyui-animatediff-evolved",
|
||||||
|
"ADE_AnimateDiffLoaderGen1": "comfyui-animatediff-evolved",
|
||||||
|
"ADE_LoadAnimateDiffModel": "comfyui-animatediff-evolved",
|
||||||
|
# ControlNet aux preprocessors (full class names)
|
||||||
|
"CannyEdgePreprocessor": "comfyui_controlnet_aux",
|
||||||
|
"DWPreprocessor": "comfyui_controlnet_aux",
|
||||||
|
"OpenposePreprocessor": "comfyui_controlnet_aux",
|
||||||
|
"DepthAnythingPreprocessor": "comfyui_controlnet_aux",
|
||||||
|
"Zoe_DepthAnythingPreprocessor": "comfyui_controlnet_aux",
|
||||||
|
"AnimalPosePreprocessor": "comfyui_controlnet_aux",
|
||||||
|
# IPAdapter Plus
|
||||||
|
"IPAdapterAdvanced": "comfyui_ipadapter_plus",
|
||||||
|
"IPAdapterUnifiedLoader": "comfyui_ipadapter_plus",
|
||||||
|
"IPAdapterModelLoader": "comfyui_ipadapter_plus",
|
||||||
|
"IPAdapterInsightFaceLoader": "comfyui_ipadapter_plus",
|
||||||
|
# InstantID
|
||||||
|
"InstantIDModelLoader": "comfyui_instantid",
|
||||||
|
"ApplyInstantID": "comfyui_instantid",
|
||||||
|
# Comfy essentials (note: registry slug uses underscore, not hyphen)
|
||||||
|
"GetImageSize+": "comfyui_essentials",
|
||||||
|
"ImageBatchMultiple+": "comfyui_essentials",
|
||||||
|
# pysssss
|
||||||
|
"ShowText|pysssss": "comfyui-custom-scripts",
|
||||||
|
"PreviewImage|pysssss": "comfyui-custom-scripts",
|
||||||
|
# SUPIR
|
||||||
|
"SUPIR_Upscale": "comfyui-supir",
|
||||||
|
"SUPIR_first_stage": "comfyui-supir",
|
||||||
|
# GGUF (case-sensitive registry slug)
|
||||||
|
"UNETLoaderGGUF": "ComfyUI-GGUF",
|
||||||
|
"DualCLIPLoaderGGUF": "ComfyUI-GGUF",
|
||||||
|
# Florence2
|
||||||
|
"Florence2Run": "comfyui-florence2",
|
||||||
|
# WAS
|
||||||
|
"Image Filter Adjustments": "was-node-suite-comfyui",
|
||||||
|
# Photomaker (case-sensitive)
|
||||||
|
"PhotoMakerLoader": "ComfyUI-PhotoMaker-Plus",
|
||||||
|
# Wan video (case-sensitive)
|
||||||
|
"WanVideoSampler": "ComfyUI-WanVideoWrapper",
|
||||||
|
"WanVideoModelLoader": "ComfyUI-WanVideoWrapper",
|
||||||
|
}
|
||||||
|
|
||||||
|
# Nodes whose package isn't on the comfy registry — need git-URL install via
|
||||||
|
# ComfyUI-Manager. We surface a helpful hint instead of an unrunnable command.
|
||||||
|
NODE_TO_GIT_URL: dict[str, str] = {
|
||||||
|
"HunyuanVideoSampler": "https://github.com/kijai/ComfyUI-HunyuanVideoWrapper",
|
||||||
|
"HunyuanVideoModelLoader": "https://github.com/kijai/ComfyUI-HunyuanVideoWrapper",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_object_info(url: str, headers: dict) -> tuple[set[str] | None, dict | None]:
|
||||||
|
"""Returns (installed_node_set, error_info). Error info is a dict if we
|
||||||
|
couldn't query (e.g. cloud free tier), else None.
|
||||||
|
"""
|
||||||
|
r = http_get(url, headers=headers, retries=2, timeout=30)
|
||||||
|
if r.status == 200:
|
||||||
|
try:
|
||||||
|
data = r.json()
|
||||||
|
if isinstance(data, dict):
|
||||||
|
return set(data.keys()), None
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return None, {"http_status": 200, "reason": "non-dict response"}
|
||||||
|
if r.status == 403:
|
||||||
|
try:
|
||||||
|
body = r.json()
|
||||||
|
except Exception:
|
||||||
|
body = {"raw": r.text()[:200]}
|
||||||
|
return None, {"http_status": 403, "reason": "forbidden", "body": body}
|
||||||
|
if r.status == 404:
|
||||||
|
return None, {"http_status": 404, "reason": "endpoint not found"}
|
||||||
|
return None, {"http_status": r.status, "reason": "unexpected", "body": r.text()[:200]}
|
||||||
|
|
||||||
|
|
||||||
|
def _fetch_one_folder(
|
||||||
|
base: str, folder: str, headers: dict, *, is_cloud: bool,
|
||||||
|
) -> tuple[set[str] | None, dict | None]:
|
||||||
|
"""Single-folder fetch, no aliasing. Returns (installed_set, error_info)."""
|
||||||
|
url = resolve_url(base, f"/models/{folder}", is_cloud=is_cloud)
|
||||||
|
r = http_get(url, headers=headers, retries=2, timeout=30)
|
||||||
|
if r.status == 200:
|
||||||
|
try:
|
||||||
|
return parse_model_list(r.json()), None
|
||||||
|
except Exception:
|
||||||
|
return set(), {"http_status": 200, "reason": "non-list response"}
|
||||||
|
if r.status == 404:
|
||||||
|
body_text = r.text()
|
||||||
|
try:
|
||||||
|
body = r.json()
|
||||||
|
except Exception:
|
||||||
|
body = {"raw": body_text[:200]}
|
||||||
|
code = body.get("code") if isinstance(body, dict) else None
|
||||||
|
if code == "folder_not_found":
|
||||||
|
# Folder is genuinely empty/missing on server — not the same as
|
||||||
|
# "endpoint missing". Return empty set with informational error.
|
||||||
|
return set(), {"http_status": 404, "reason": "folder_empty_or_unknown", "body": body}
|
||||||
|
return None, {"http_status": 404, "reason": "endpoint not found", "body": body}
|
||||||
|
if r.status == 403:
|
||||||
|
try:
|
||||||
|
body = r.json()
|
||||||
|
except Exception:
|
||||||
|
body = {}
|
||||||
|
return None, {"http_status": 403, "reason": "forbidden", "body": body}
|
||||||
|
return None, {"http_status": r.status, "reason": "unexpected"}
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_models_for_folder(
|
||||||
|
base: str, folder: str, headers: dict, *, is_cloud: bool,
|
||||||
|
) -> tuple[set[str] | None, dict | None]:
|
||||||
|
"""Fetch installed models for a folder, trying aliases.
|
||||||
|
|
||||||
|
Folder renames over time (e.g. unet → diffusion_models, clip → text_encoders)
|
||||||
|
mean a workflow asking for a model in `unet` may need to look in
|
||||||
|
`diffusion_models`. We union models from every reachable alias.
|
||||||
|
|
||||||
|
Returns (combined_set | None, last_error | None).
|
||||||
|
"""
|
||||||
|
aliases = folder_aliases_for(folder)
|
||||||
|
combined: set[str] = set()
|
||||||
|
any_success = False
|
||||||
|
last_err: dict | None = None
|
||||||
|
for alias in aliases:
|
||||||
|
models, err = _fetch_one_folder(base, alias, headers, is_cloud=is_cloud)
|
||||||
|
if models is not None:
|
||||||
|
combined.update(models)
|
||||||
|
any_success = True
|
||||||
|
last_err = None
|
||||||
|
else:
|
||||||
|
last_err = err
|
||||||
|
if not any_success:
|
||||||
|
return None, last_err
|
||||||
|
return combined, None
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_embeddings(base: str, headers: dict, *, is_cloud: bool) -> tuple[set[str] | None, dict | None]:
|
||||||
|
"""Local ComfyUI exposes /embeddings; cloud uses /experiment/models/embeddings."""
|
||||||
|
if is_cloud:
|
||||||
|
return fetch_models_for_folder(base, "embeddings", headers, is_cloud=True)
|
||||||
|
# Local: dedicated /embeddings returns a flat list of names
|
||||||
|
r = http_get(resolve_url(base, "/embeddings", is_cloud=False), headers=headers, retries=2)
|
||||||
|
if r.status == 200:
|
||||||
|
try:
|
||||||
|
data = r.json()
|
||||||
|
if isinstance(data, list):
|
||||||
|
# Strip extensions from the registered names since prompt syntax
|
||||||
|
# usually omits them ("embedding:goodvibes" vs "goodvibes.pt")
|
||||||
|
names = set()
|
||||||
|
for n in data:
|
||||||
|
if isinstance(n, str):
|
||||||
|
names.add(n)
|
||||||
|
# Also store stem for fuzzy matching
|
||||||
|
names.add(Path(n).stem)
|
||||||
|
return names, None
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return None, {"http_status": r.status, "reason": "unexpected"}
|
||||||
|
|
||||||
|
|
||||||
|
def normalize_for_match(name: str) -> set[str]:
|
||||||
|
"""Generate matching variants of a model name (with/without extension, slashes, etc.)"""
|
||||||
|
s = {name}
|
||||||
|
s.add(Path(name).stem)
|
||||||
|
s.add(Path(name).name)
|
||||||
|
# ComfyUI sometimes strips/keeps the leading folder
|
||||||
|
if "/" in name or "\\" in name:
|
||||||
|
flat = name.replace("\\", "/").split("/")[-1]
|
||||||
|
s.add(flat)
|
||||||
|
s.add(Path(flat).stem)
|
||||||
|
return {x for x in s if x}
|
||||||
|
|
||||||
|
|
||||||
|
def model_present(needed: str, installed: set[str]) -> bool:
|
||||||
|
if not installed:
|
||||||
|
return False
|
||||||
|
needed_variants = normalize_for_match(needed)
|
||||||
|
installed_norm: set[str] = set()
|
||||||
|
for inst in installed:
|
||||||
|
installed_norm.update(normalize_for_match(inst))
|
||||||
|
return bool(needed_variants & installed_norm)
|
||||||
|
|
||||||
|
|
||||||
|
def suggest_install_command(node_class: str) -> str | None:
|
||||||
|
pkg = NODE_TO_PACKAGE.get(node_class)
|
||||||
|
if pkg:
|
||||||
|
return f"comfy node install {pkg}"
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def suggest_git_url(node_class: str) -> str | None:
|
||||||
|
"""For nodes not on the registry, return a git URL the user can hand to
|
||||||
|
ComfyUI-Manager's `/manager/queue/install` endpoint."""
|
||||||
|
return NODE_TO_GIT_URL.get(node_class)
|
||||||
|
|
||||||
|
|
||||||
|
def check_deps(
|
||||||
|
workflow: dict, host: str, *, api_key: str | None = None,
|
||||||
|
) -> dict:
|
||||||
|
headers: dict[str, str] = {}
|
||||||
|
if api_key:
|
||||||
|
headers["X-API-Key"] = api_key
|
||||||
|
|
||||||
|
is_cloud = is_cloud_host(host)
|
||||||
|
base = host.rstrip("/")
|
||||||
|
|
||||||
|
# ---- 1. Required nodes ----
|
||||||
|
required_nodes: set[str] = set()
|
||||||
|
for _, node in iter_nodes(workflow):
|
||||||
|
required_nodes.add(node["class_type"])
|
||||||
|
|
||||||
|
object_info_url = resolve_url(base, "/object_info", is_cloud=is_cloud)
|
||||||
|
installed_nodes, obj_err = fetch_object_info(object_info_url, headers)
|
||||||
|
|
||||||
|
missing_nodes: list[dict] = []
|
||||||
|
node_check_skipped = False
|
||||||
|
if installed_nodes is None:
|
||||||
|
# Couldn't query (e.g. cloud free tier). Don't false-alarm; mark skipped.
|
||||||
|
node_check_skipped = True
|
||||||
|
else:
|
||||||
|
for cls in sorted(required_nodes):
|
||||||
|
if cls not in installed_nodes:
|
||||||
|
entry = {"class_type": cls}
|
||||||
|
cmd = suggest_install_command(cls)
|
||||||
|
git_url = suggest_git_url(cls)
|
||||||
|
if cmd:
|
||||||
|
entry["fix_command"] = cmd
|
||||||
|
elif git_url:
|
||||||
|
entry["fix_git_url"] = git_url
|
||||||
|
entry["fix_hint"] = (
|
||||||
|
f"Not on registry. Install via Manager with this git URL: {git_url}"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
entry["fix_hint"] = (
|
||||||
|
"Search https://registry.comfy.org or "
|
||||||
|
"use ComfyUI-Manager UI to find the package providing this node."
|
||||||
|
)
|
||||||
|
missing_nodes.append(entry)
|
||||||
|
|
||||||
|
# ---- 2. Required models ----
|
||||||
|
model_cache: dict[str, tuple[set[str] | None, dict | None]] = {}
|
||||||
|
missing_models: list[dict] = []
|
||||||
|
folder_errors: dict[str, dict] = {}
|
||||||
|
|
||||||
|
for dep in iter_model_deps(workflow):
|
||||||
|
folder = dep["folder"]
|
||||||
|
if folder not in model_cache:
|
||||||
|
model_cache[folder] = fetch_models_for_folder(
|
||||||
|
base, folder, headers, is_cloud=is_cloud,
|
||||||
|
)
|
||||||
|
installed, err = model_cache[folder]
|
||||||
|
if installed is None:
|
||||||
|
# Couldn't enumerate this folder — record once
|
||||||
|
folder_errors.setdefault(folder, err or {})
|
||||||
|
# Don't flag as missing (we don't know); the folder_errors block surfaces this
|
||||||
|
continue
|
||||||
|
if not model_present(dep["value"], installed):
|
||||||
|
entry = dict(dep)
|
||||||
|
entry["fix_hint"] = (
|
||||||
|
f"comfy model download --url <URL> --relative-path models/{folder} "
|
||||||
|
f"--filename {dep['value']!r}"
|
||||||
|
)
|
||||||
|
missing_models.append(entry)
|
||||||
|
|
||||||
|
# ---- 3. Embedding refs in prompts ----
|
||||||
|
emb_installed, emb_err = fetch_embeddings(base, headers, is_cloud=is_cloud)
|
||||||
|
missing_embeddings: list[dict] = []
|
||||||
|
seen_emb: set[tuple[str, str]] = set()
|
||||||
|
for nid, emb_name in iter_embedding_refs(workflow):
|
||||||
|
if (nid, emb_name) in seen_emb:
|
||||||
|
continue
|
||||||
|
seen_emb.add((nid, emb_name))
|
||||||
|
if emb_installed is None:
|
||||||
|
# Couldn't enumerate — skip silently here, surface the error in the
|
||||||
|
# folder_errors block
|
||||||
|
continue
|
||||||
|
if not model_present(emb_name, emb_installed):
|
||||||
|
missing_embeddings.append({
|
||||||
|
"node_id": nid,
|
||||||
|
"embedding_name": emb_name,
|
||||||
|
"folder": "embeddings",
|
||||||
|
"fix_hint": (
|
||||||
|
f"Download {emb_name}.pt or .safetensors and place in "
|
||||||
|
f"models/embeddings/, or `comfy model download --url <URL> "
|
||||||
|
f"--relative-path models/embeddings`"
|
||||||
|
),
|
||||||
|
})
|
||||||
|
|
||||||
|
if emb_err and emb_installed is None:
|
||||||
|
folder_errors.setdefault("embeddings", emb_err)
|
||||||
|
|
||||||
|
is_ready = (
|
||||||
|
not node_check_skipped
|
||||||
|
and not missing_nodes
|
||||||
|
and not missing_models
|
||||||
|
and not missing_embeddings
|
||||||
|
)
|
||||||
|
|
||||||
|
return {
|
||||||
|
"is_ready": is_ready,
|
||||||
|
"node_check_skipped": node_check_skipped,
|
||||||
|
"node_check_skip_reason": obj_err if node_check_skipped else None,
|
||||||
|
"missing_nodes": missing_nodes,
|
||||||
|
"missing_models": missing_models,
|
||||||
|
"missing_embeddings": missing_embeddings,
|
||||||
|
"folder_errors": folder_errors,
|
||||||
|
# 0 is a legitimate count (e.g. empty server). Use None only when not queried.
|
||||||
|
"installed_node_count": len(installed_nodes) if installed_nodes is not None else None,
|
||||||
|
"required_node_count": len(required_nodes),
|
||||||
|
"required_nodes": sorted(required_nodes),
|
||||||
|
"host": base,
|
||||||
|
"is_cloud": is_cloud,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
p = argparse.ArgumentParser(description="Check ComfyUI workflow dependencies against a running server")
|
||||||
|
p.add_argument("workflow", help="Path to workflow API JSON file")
|
||||||
|
p.add_argument("--host", default=DEFAULT_LOCAL_HOST, help="ComfyUI server URL")
|
||||||
|
p.add_argument("--port", type=int, help="Server port (overrides --host port)")
|
||||||
|
p.add_argument("--api-key", help=f"API key for cloud (or set ${ENV_API_KEY} env var)")
|
||||||
|
p.add_argument("--strict", action="store_true",
|
||||||
|
help="Exit non-zero if node check is skipped (e.g. on cloud free tier)")
|
||||||
|
args = p.parse_args(argv)
|
||||||
|
|
||||||
|
host = args.host
|
||||||
|
if args.port is not None:
|
||||||
|
# Strip any port from host and append --port
|
||||||
|
from urllib.parse import urlparse, urlunparse
|
||||||
|
parsed = urlparse(host if "://" in host else f"http://{host}")
|
||||||
|
new_netloc = f"{parsed.hostname}:{args.port}"
|
||||||
|
host = urlunparse(parsed._replace(netloc=new_netloc))
|
||||||
|
|
||||||
|
api_key = resolve_api_key(args.api_key)
|
||||||
|
|
||||||
|
wf_path = Path(args.workflow).expanduser()
|
||||||
|
if not wf_path.exists():
|
||||||
|
emit_json({"error": f"Workflow file not found: {args.workflow}"})
|
||||||
|
return 1
|
||||||
|
try:
|
||||||
|
with wf_path.open() as f:
|
||||||
|
payload = json.load(f)
|
||||||
|
workflow = unwrap_workflow(payload)
|
||||||
|
except ValueError as e:
|
||||||
|
emit_json({"error": str(e)})
|
||||||
|
return 1
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
emit_json({"error": f"Invalid JSON: {e}"})
|
||||||
|
return 1
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = check_deps(workflow, host=host, api_key=api_key)
|
||||||
|
except Exception as e:
|
||||||
|
emit_json({"error": f"Dep check failed: {e}", "host": host})
|
||||||
|
return 1
|
||||||
|
|
||||||
|
emit_json(result)
|
||||||
|
|
||||||
|
if not result["is_ready"]:
|
||||||
|
return 1
|
||||||
|
if args.strict and result["node_check_skipped"]:
|
||||||
|
return 1
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,315 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
extract_schema.py — Analyze a ComfyUI API-format workflow and extract
|
||||||
|
controllable parameters.
|
||||||
|
|
||||||
|
Improvements over v1:
|
||||||
|
- Catalogs live in `_common.py`, shared with `check_deps.py`
|
||||||
|
- Coverage expanded for Flux / SD3 / Wan / Hunyuan / LTX / IPAdapter / rgthree
|
||||||
|
- Symmetric duplicate-name resolution: ALL duplicates get a node-id suffix
|
||||||
|
(instead of "first wins, second renamed"), so callers see consistent names
|
||||||
|
- Negative prompt detected by tracing `KSampler.negative` connections back to
|
||||||
|
the source CLIPTextEncode (more reliable than meta-title heuristic)
|
||||||
|
- Embedding references in prompt text are extracted as model dependencies
|
||||||
|
- Detects Primitive nodes that drive other nodes' inputs (and surfaces them
|
||||||
|
as the user-facing parameter)
|
||||||
|
- Reroutes are followed when tracing connections
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 extract_schema.py workflow_api.json
|
||||||
|
python3 extract_schema.py workflow_api.json --output schema.json
|
||||||
|
|
||||||
|
Stdlib-only. Python 3.10+.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||||
|
from _common import ( # noqa: E402
|
||||||
|
OUTPUT_NODES, PARAM_PATTERNS, PROMPT_FIELDS,
|
||||||
|
is_link, iter_embedding_refs, iter_model_deps, iter_nodes, unwrap_workflow,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# Sampler nodes whose `positive` / `negative` connections we trace
|
||||||
|
SAMPLER_NODE_FAMILY = {
|
||||||
|
"KSampler", "KSamplerAdvanced",
|
||||||
|
"SamplerCustom", "SamplerCustomAdvanced",
|
||||||
|
"BasicGuider", "CFGGuider", "DualCFGGuider",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def infer_type(value: Any) -> str:
|
||||||
|
if isinstance(value, bool):
|
||||||
|
return "bool"
|
||||||
|
if isinstance(value, int):
|
||||||
|
return "int"
|
||||||
|
if isinstance(value, float):
|
||||||
|
return "float"
|
||||||
|
if isinstance(value, str):
|
||||||
|
return "string"
|
||||||
|
if isinstance(value, list):
|
||||||
|
return "link"
|
||||||
|
if isinstance(value, dict):
|
||||||
|
return "object"
|
||||||
|
return "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def trace_to_node(workflow: dict, link: list, *, max_hops: int = 8) -> str | None:
|
||||||
|
"""Follow a [node_id, slot] link, hopping through Reroute / Primitive nodes
|
||||||
|
if needed, to find the *upstream* node id that holds the actual value/input.
|
||||||
|
|
||||||
|
Bounded by both `max_hops` AND a visited-set to prevent infinite loops on
|
||||||
|
pathological graphs.
|
||||||
|
"""
|
||||||
|
if not is_link(link):
|
||||||
|
return None
|
||||||
|
nid: str | None = link[0]
|
||||||
|
visited: set[str] = set()
|
||||||
|
for _ in range(max_hops):
|
||||||
|
if nid is None or nid in visited:
|
||||||
|
return nid
|
||||||
|
visited.add(nid)
|
||||||
|
node = workflow.get(nid)
|
||||||
|
if not isinstance(node, dict):
|
||||||
|
return None
|
||||||
|
cls = node.get("class_type", "")
|
||||||
|
# Reroute / Primitive / passthrough wrappers
|
||||||
|
if cls in {"Reroute", "PrimitiveNode", "Note", "easy showAnything"}:
|
||||||
|
inputs = node.get("inputs", {}) or {}
|
||||||
|
# Find first link-shaped input and follow it
|
||||||
|
next_link = next((v for v in inputs.values() if is_link(v)), None)
|
||||||
|
if next_link is None:
|
||||||
|
return nid
|
||||||
|
nid = next_link[0]
|
||||||
|
continue
|
||||||
|
return nid
|
||||||
|
return nid
|
||||||
|
|
||||||
|
|
||||||
|
def find_negative_prompt_node(workflow: dict) -> str | None:
|
||||||
|
"""Trace `negative` input of a sampler back to the source text encoder."""
|
||||||
|
for nid, node in iter_nodes(workflow):
|
||||||
|
if node["class_type"] not in SAMPLER_NODE_FAMILY:
|
||||||
|
continue
|
||||||
|
inputs = node.get("inputs", {}) or {}
|
||||||
|
neg = inputs.get("negative")
|
||||||
|
if not is_link(neg):
|
||||||
|
continue
|
||||||
|
src = trace_to_node(workflow, neg)
|
||||||
|
if src and isinstance(workflow.get(src), dict):
|
||||||
|
cls = workflow[src].get("class_type", "")
|
||||||
|
if cls.startswith("CLIPTextEncode") or cls in {"smZ CLIPTextEncode", "BNK_CLIPTextEncodeAdvanced"}:
|
||||||
|
return src
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def find_positive_prompt_node(workflow: dict) -> str | None:
|
||||||
|
for nid, node in iter_nodes(workflow):
|
||||||
|
if node["class_type"] not in SAMPLER_NODE_FAMILY:
|
||||||
|
continue
|
||||||
|
inputs = node.get("inputs", {}) or {}
|
||||||
|
pos = inputs.get("positive")
|
||||||
|
if not is_link(pos):
|
||||||
|
continue
|
||||||
|
src = trace_to_node(workflow, pos)
|
||||||
|
if src and isinstance(workflow.get(src), dict):
|
||||||
|
cls = workflow[src].get("class_type", "")
|
||||||
|
if cls.startswith("CLIPTextEncode") or cls in {"smZ CLIPTextEncode", "BNK_CLIPTextEncodeAdvanced"}:
|
||||||
|
return src
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def extract_schema(workflow: dict) -> dict:
|
||||||
|
"""Extract controllable parameters from a workflow.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
{
|
||||||
|
"parameters": { friendly_name: {node_id, field, type, value, ...} },
|
||||||
|
"output_nodes": [node_id, ...],
|
||||||
|
"model_dependencies": [{node_id, class_type, field, value, folder}],
|
||||||
|
"embedding_dependencies": [{node_id, embedding_name, found_in_field, value_excerpt}],
|
||||||
|
"summary": {...}
|
||||||
|
}
|
||||||
|
"""
|
||||||
|
output_nodes: list[str] = []
|
||||||
|
|
||||||
|
# First pass: identify positive / negative prompt nodes via connection tracing
|
||||||
|
pos_node = find_positive_prompt_node(workflow)
|
||||||
|
neg_node = find_negative_prompt_node(workflow)
|
||||||
|
|
||||||
|
# ----- collect raw parameter candidates -----
|
||||||
|
# Each candidate = (friendly_name, node_id, field, value)
|
||||||
|
# We resolve duplicate friendly_names AFTER the loop so dedup is symmetric.
|
||||||
|
raw_params: list[dict] = []
|
||||||
|
|
||||||
|
for node_id, node in iter_nodes(workflow):
|
||||||
|
cls = node["class_type"]
|
||||||
|
inputs = node.get("inputs", {}) or {}
|
||||||
|
|
||||||
|
if cls in OUTPUT_NODES:
|
||||||
|
output_nodes.append(node_id)
|
||||||
|
|
||||||
|
# Match this node against PARAM_PATTERNS
|
||||||
|
for p_class, p_field, friendly in PARAM_PATTERNS:
|
||||||
|
if cls != p_class:
|
||||||
|
continue
|
||||||
|
if p_field not in inputs:
|
||||||
|
continue
|
||||||
|
value = inputs[p_field]
|
||||||
|
t = infer_type(value)
|
||||||
|
if t == "link":
|
||||||
|
continue # connections aren't directly controllable
|
||||||
|
|
||||||
|
actual_name = friendly
|
||||||
|
|
||||||
|
# Disambiguate prompt vs negative_prompt by connection tracing
|
||||||
|
if friendly == "prompt":
|
||||||
|
if node_id == neg_node and pos_node != neg_node:
|
||||||
|
actual_name = "negative_prompt"
|
||||||
|
elif node_id == pos_node:
|
||||||
|
actual_name = "prompt"
|
||||||
|
else:
|
||||||
|
# Fallback: use _meta.title hints if present
|
||||||
|
meta_title = (node.get("_meta") or {}).get("title", "").lower()
|
||||||
|
if any(t_ in meta_title for t_ in ("negative", "neg", "-prompt", "anti")):
|
||||||
|
actual_name = "negative_prompt"
|
||||||
|
|
||||||
|
raw_params.append({
|
||||||
|
"name_hint": actual_name,
|
||||||
|
"node_id": node_id,
|
||||||
|
"field": p_field,
|
||||||
|
"type": t,
|
||||||
|
"value": value,
|
||||||
|
"class_type": cls,
|
||||||
|
})
|
||||||
|
|
||||||
|
# ----- symmetric duplicate-name resolution -----
|
||||||
|
# Group by name_hint. If a hint appears once, keep it. If multiple, suffix
|
||||||
|
# ALL with their node_id. Always-stable, always-uniquely-addressable.
|
||||||
|
by_name: dict[str, list[dict]] = {}
|
||||||
|
for r in raw_params:
|
||||||
|
by_name.setdefault(r["name_hint"], []).append(r)
|
||||||
|
|
||||||
|
parameters: dict[str, dict] = {}
|
||||||
|
for name, entries in by_name.items():
|
||||||
|
if len(entries) == 1:
|
||||||
|
r = entries[0]
|
||||||
|
parameters[name] = {
|
||||||
|
"node_id": r["node_id"], "field": r["field"],
|
||||||
|
"type": r["type"], "value": r["value"],
|
||||||
|
"class_type": r["class_type"],
|
||||||
|
}
|
||||||
|
else:
|
||||||
|
# Sort by node_id (string-natural) for stability
|
||||||
|
entries.sort(key=lambda x: (str(x["node_id"]).zfill(8), x["field"]))
|
||||||
|
for r in entries:
|
||||||
|
full_name = f"{name}_{r['node_id']}"
|
||||||
|
parameters[full_name] = {
|
||||||
|
"node_id": r["node_id"], "field": r["field"],
|
||||||
|
"type": r["type"], "value": r["value"],
|
||||||
|
"class_type": r["class_type"],
|
||||||
|
"alias_of": name,
|
||||||
|
}
|
||||||
|
|
||||||
|
# ----- model dependencies -----
|
||||||
|
model_deps = list(iter_model_deps(workflow))
|
||||||
|
|
||||||
|
# ----- embedding dependencies (in prompt text) -----
|
||||||
|
embedding_deps: list[dict] = []
|
||||||
|
seen_emb: set[tuple[str, str]] = set()
|
||||||
|
for nid, emb_name in iter_embedding_refs(workflow):
|
||||||
|
key = (nid, emb_name)
|
||||||
|
if key in seen_emb:
|
||||||
|
continue
|
||||||
|
seen_emb.add(key)
|
||||||
|
# Find which field had the reference, for context
|
||||||
|
node = workflow.get(nid, {})
|
||||||
|
inputs = node.get("inputs", {}) or {}
|
||||||
|
found_field = None
|
||||||
|
excerpt = None
|
||||||
|
for fname, fval in inputs.items():
|
||||||
|
if isinstance(fval, str) and fname in PROMPT_FIELDS and emb_name in fval:
|
||||||
|
found_field = fname
|
||||||
|
excerpt = fval[:120]
|
||||||
|
break
|
||||||
|
embedding_deps.append({
|
||||||
|
"node_id": nid,
|
||||||
|
"embedding_name": emb_name,
|
||||||
|
"field": found_field,
|
||||||
|
"value_excerpt": excerpt,
|
||||||
|
"folder": "embeddings",
|
||||||
|
})
|
||||||
|
|
||||||
|
# ----- summary -----
|
||||||
|
summary = {
|
||||||
|
"parameter_count": len(parameters),
|
||||||
|
"output_node_count": len(output_nodes),
|
||||||
|
"model_dep_count": len(model_deps),
|
||||||
|
"embedding_dep_count": len(embedding_deps),
|
||||||
|
"has_negative_prompt": "negative_prompt" in parameters,
|
||||||
|
"has_seed": "seed" in parameters or any(p.startswith("seed_") for p in parameters),
|
||||||
|
"is_video_workflow": any(
|
||||||
|
workflow.get(n, {}).get("class_type", "") in {
|
||||||
|
"VHS_VideoCombine", "SaveVideo", "SaveAnimatedWEBP", "SaveAnimatedPNG",
|
||||||
|
} for n in output_nodes
|
||||||
|
),
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
"parameters": parameters,
|
||||||
|
"output_nodes": output_nodes,
|
||||||
|
"model_dependencies": model_deps,
|
||||||
|
"embedding_dependencies": embedding_deps,
|
||||||
|
"summary": summary,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
p = argparse.ArgumentParser(description="Extract controllable parameters from a ComfyUI workflow")
|
||||||
|
p.add_argument("workflow", help="Path to workflow API JSON file")
|
||||||
|
p.add_argument("--output", "-o", help="Output file (default: stdout)")
|
||||||
|
p.add_argument("--summary-only", action="store_true",
|
||||||
|
help="Only print the summary block")
|
||||||
|
args = p.parse_args(argv)
|
||||||
|
|
||||||
|
wf_path = Path(args.workflow).expanduser()
|
||||||
|
if not wf_path.exists():
|
||||||
|
print(f"Error: {wf_path} not found", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
try:
|
||||||
|
with wf_path.open() as f:
|
||||||
|
payload = json.load(f)
|
||||||
|
workflow = unwrap_workflow(payload)
|
||||||
|
except ValueError as e:
|
||||||
|
print(f"Error: {e}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
except json.JSONDecodeError as e:
|
||||||
|
print(f"Error: invalid JSON — {e}", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
schema = extract_schema(workflow)
|
||||||
|
|
||||||
|
if args.summary_only:
|
||||||
|
out = json.dumps(schema["summary"], indent=2)
|
||||||
|
else:
|
||||||
|
out = json.dumps(schema, indent=2, default=str)
|
||||||
|
|
||||||
|
if args.output:
|
||||||
|
Path(args.output).write_text(out)
|
||||||
|
print(f"Schema written to {args.output}", file=sys.stderr)
|
||||||
|
else:
|
||||||
|
print(out)
|
||||||
|
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
+158
@@ -0,0 +1,158 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
fetch_logs.py — Retrieve workflow execution diagnostics from a ComfyUI server.
|
||||||
|
|
||||||
|
When a workflow errors, the server's /history (local) or /jobs (cloud) entry
|
||||||
|
contains the full Python traceback. This script makes it easy to fetch by
|
||||||
|
prompt_id, with sensible formatting.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
python3 fetch_logs.py <prompt_id>
|
||||||
|
python3 fetch_logs.py <prompt_id> --host https://cloud.comfy.org
|
||||||
|
python3 fetch_logs.py --tail-queue # show currently queued/running jobs
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||||
|
from _common import ( # noqa: E402
|
||||||
|
DEFAULT_LOCAL_HOST, ENV_API_KEY, emit_json, http_get, is_cloud_host,
|
||||||
|
resolve_api_key, resolve_url,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_history_entry(host: str, headers: dict, prompt_id: str, *, is_cloud: bool) -> dict:
|
||||||
|
if is_cloud:
|
||||||
|
# Try /jobs/{id} first
|
||||||
|
url = resolve_url(host, f"/jobs/{prompt_id}", is_cloud=True)
|
||||||
|
r = http_get(url, headers=headers, retries=2, timeout=30)
|
||||||
|
if r.status == 200:
|
||||||
|
try:
|
||||||
|
return {"ok": True, "entry": r.json(), "source": "/api/jobs"}
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
# Fallback to history_v2
|
||||||
|
url = resolve_url(host, f"/history/{prompt_id}", is_cloud=True)
|
||||||
|
r = http_get(url, headers=headers, retries=2, timeout=30)
|
||||||
|
try:
|
||||||
|
data = r.json()
|
||||||
|
except Exception:
|
||||||
|
data = None
|
||||||
|
if r.status == 200 and data:
|
||||||
|
return {"ok": True, "entry": data, "source": "/api/history_v2"}
|
||||||
|
return {"ok": False, "http_status": r.status, "body": r.text()[:500]}
|
||||||
|
|
||||||
|
url = resolve_url(host, f"/history/{prompt_id}", is_cloud=False)
|
||||||
|
r = http_get(url, headers=headers, retries=2, timeout=30)
|
||||||
|
if r.status != 200:
|
||||||
|
return {"ok": False, "http_status": r.status, "body": r.text()[:500]}
|
||||||
|
try:
|
||||||
|
data = r.json()
|
||||||
|
except Exception:
|
||||||
|
return {"ok": False, "reason": "non-JSON response"}
|
||||||
|
if not isinstance(data, dict) or prompt_id not in data:
|
||||||
|
return {"ok": False, "reason": "prompt_id not found in history",
|
||||||
|
"history_keys": list(data.keys())[:5] if isinstance(data, dict) else []}
|
||||||
|
return {"ok": True, "entry": data[prompt_id], "source": "/history"}
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_queue(host: str, headers: dict) -> dict:
|
||||||
|
url = resolve_url(host, "/queue")
|
||||||
|
r = http_get(url, headers=headers, retries=2, timeout=15)
|
||||||
|
try:
|
||||||
|
data = r.json()
|
||||||
|
except Exception:
|
||||||
|
data = {"raw": r.text()[:500]}
|
||||||
|
return {"http_status": r.status, "data": data}
|
||||||
|
|
||||||
|
|
||||||
|
def extract_diagnostics(entry: dict) -> dict:
|
||||||
|
"""Pull out the parts a human cares about: status, errors, traceback, timing."""
|
||||||
|
diag: dict = {}
|
||||||
|
status = entry.get("status") or {}
|
||||||
|
diag["status_str"] = status.get("status_str")
|
||||||
|
diag["completed"] = status.get("completed")
|
||||||
|
|
||||||
|
messages = status.get("messages") or []
|
||||||
|
diag["execution_log"] = []
|
||||||
|
for msg in messages:
|
||||||
|
if isinstance(msg, list) and len(msg) >= 2:
|
||||||
|
mtype, mdata = msg[0], msg[1]
|
||||||
|
diag["execution_log"].append({"type": mtype, "data": mdata})
|
||||||
|
else:
|
||||||
|
diag["execution_log"].append(msg)
|
||||||
|
|
||||||
|
# Look for execution_error inside messages
|
||||||
|
errors = []
|
||||||
|
for msg in messages:
|
||||||
|
if isinstance(msg, list) and len(msg) >= 2 and msg[0] == "execution_error":
|
||||||
|
errors.append(msg[1])
|
||||||
|
if errors:
|
||||||
|
diag["errors"] = errors
|
||||||
|
|
||||||
|
# Cloud's /jobs response shape: top-level outputs / status / etc.
|
||||||
|
if "outputs" in entry:
|
||||||
|
out = entry["outputs"] or {}
|
||||||
|
if isinstance(out, dict):
|
||||||
|
diag["output_node_ids"] = list(out.keys())
|
||||||
|
# Count file refs across all output buckets (images / video / etc.)
|
||||||
|
total = 0
|
||||||
|
for node_output in out.values():
|
||||||
|
if not isinstance(node_output, dict):
|
||||||
|
continue
|
||||||
|
for v in node_output.values():
|
||||||
|
if isinstance(v, list):
|
||||||
|
total += len(v)
|
||||||
|
diag["output_count"] = total
|
||||||
|
else:
|
||||||
|
diag["output_node_ids"] = []
|
||||||
|
diag["output_count"] = 0
|
||||||
|
return diag
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
p = argparse.ArgumentParser(description="Fetch workflow execution diagnostics")
|
||||||
|
p.add_argument("prompt_id", nargs="?", help="prompt_id to look up")
|
||||||
|
p.add_argument("--host", default=DEFAULT_LOCAL_HOST)
|
||||||
|
p.add_argument("--api-key", help=f"or set ${ENV_API_KEY}")
|
||||||
|
p.add_argument("--raw", action="store_true",
|
||||||
|
help="Print the full history entry instead of the digest")
|
||||||
|
p.add_argument("--tail-queue", action="store_true",
|
||||||
|
help="Show currently running/pending jobs instead")
|
||||||
|
args = p.parse_args(argv)
|
||||||
|
|
||||||
|
api_key = resolve_api_key(args.api_key)
|
||||||
|
headers = {"X-API-Key": api_key} if api_key else {}
|
||||||
|
is_cloud = is_cloud_host(args.host)
|
||||||
|
|
||||||
|
if args.tail_queue:
|
||||||
|
emit_json(fetch_queue(args.host, headers))
|
||||||
|
return 0
|
||||||
|
|
||||||
|
if not args.prompt_id:
|
||||||
|
print("Error: prompt_id is required (or use --tail-queue)", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
res = fetch_history_entry(args.host, headers, args.prompt_id, is_cloud=is_cloud)
|
||||||
|
if not res.get("ok"):
|
||||||
|
emit_json(res)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
if args.raw:
|
||||||
|
emit_json(res)
|
||||||
|
return 0
|
||||||
|
|
||||||
|
diag = extract_diagnostics(res["entry"])
|
||||||
|
diag["source"] = res.get("source")
|
||||||
|
diag["prompt_id"] = args.prompt_id
|
||||||
|
emit_json(diag)
|
||||||
|
return 0 if diag.get("status_str") not in {"error",} else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,198 @@
|
|||||||
|
# Design System: Revolut
|
||||||
|
|
||||||
|
|
||||||
|
> **Hermes Agent — Implementation Notes**
|
||||||
|
>
|
||||||
|
> The original site uses proprietary fonts. For self-contained HTML output, use these CDN substitutes:
|
||||||
|
> - **Primary:** `Inter` | **Mono:** `system monospace stack`
|
||||||
|
> - **Font stack (CSS):** `font-family: 'Inter', system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif;`
|
||||||
|
> - **Mono stack (CSS):** `font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, 'Liberation Mono', 'Courier New', monospace;`
|
||||||
|
> ```html
|
||||||
|
> <link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700&display=swap" rel="stylesheet">
|
||||||
|
> ```
|
||||||
|
> Use `write_file` to create HTML, serve via `generative-widgets` skill (cloudflared tunnel).
|
||||||
|
> Verify visual accuracy with `browser_vision` after generating.
|
||||||
|
|
||||||
|
## 1. Visual Theme & Atmosphere
|
||||||
|
|
||||||
|
Revolut's website is fintech confidence distilled into pixels — a design system that communicates "your money is in capable hands" through massive typography, generous whitespace, and a disciplined neutral palette. The visual language is built on Aeonik Pro, a geometric grotesque that creates billboard-scale headlines at 136px with weight 500 and aggressive negative tracking (-2.72px). This isn't subtle branding; it's fintech at stadium scale.
|
||||||
|
|
||||||
|
The color system is built on a comprehensive `--rui-*` (Revolut UI) token architecture with semantic naming for every state: danger (`#e23b4a`), warning (`#ec7e00`), teal (`#00a87e`), blue (`#494fdf`), deep-pink (`#e61e49`), and more. But the marketing surface itself is remarkably restrained — near-black (`#191c1f`) and pure white (`#ffffff`) dominate, with the colorful semantic tokens reserved for the product interface, not the marketing page.
|
||||||
|
|
||||||
|
What distinguishes Revolut is its pill-everything button system. Every button uses 9999px radius — primary dark (`#191c1f`), secondary light (`#f4f4f4`), outlined (`transparent + 2px solid`), and ghost on dark (`rgba(244,244,244,0.1) + 2px solid`). The padding is generous (14px 32px–34px), creating large, confident touch targets. Combined with Inter for body text at various weights and positive letter-spacing (0.16px–0.24px), the result is a design that feels both premium and accessible — banking for the modern era.
|
||||||
|
|
||||||
|
**Key Characteristics:**
|
||||||
|
- Aeonik Pro display at 136px weight 500 — billboard-scale fintech headlines
|
||||||
|
- Near-black (`#191c1f`) + white binary with comprehensive `--rui-*` semantic tokens
|
||||||
|
- Universal pill buttons (9999px radius) with generous padding (14px 32px)
|
||||||
|
- Inter for body text with positive letter-spacing (0.16px–0.24px)
|
||||||
|
- Rich semantic color system: blue, teal, pink, yellow, green, brown, danger, warning
|
||||||
|
- Zero shadows detected — depth through color contrast only
|
||||||
|
- Tight display line-heights (1.00) with relaxed body (1.50–1.56)
|
||||||
|
|
||||||
|
## 2. Color Palette & Roles
|
||||||
|
|
||||||
|
### Primary
|
||||||
|
- **Revolut Dark** (`#191c1f`): Primary dark surface, button background, near-black text
|
||||||
|
- **Pure White** (`#ffffff`): `--rui-color-action-label`, primary light surface
|
||||||
|
- **Light Surface** (`#f4f4f4`): Secondary button background, subtle surface
|
||||||
|
|
||||||
|
### Brand / Interactive
|
||||||
|
- **Revolut Blue** (`#494fdf`): `--rui-color-blue`, primary brand blue
|
||||||
|
- **Action Blue** (`#4f55f1`): `--rui-color-action-photo-header-text`, header accent
|
||||||
|
- **Blue Text** (`#376cd5`): `--website-color-blue-text`, link blue
|
||||||
|
|
||||||
|
### Semantic
|
||||||
|
- **Danger Red** (`#e23b4a`): `--rui-color-danger`, error/destructive
|
||||||
|
- **Deep Pink** (`#e61e49`): `--rui-color-deep-pink`, critical accent
|
||||||
|
- **Warning Orange** (`#ec7e00`): `--rui-color-warning`, warning states
|
||||||
|
- **Yellow** (`#b09000`): `--rui-color-yellow`, attention
|
||||||
|
- **Teal** (`#00a87e`): `--rui-color-teal`, success/positive
|
||||||
|
- **Light Green** (`#428619`): `--rui-color-light-green`, secondary success
|
||||||
|
- **Green Text** (`#006400`): `--website-color-green-text`, green text
|
||||||
|
- **Light Blue** (`#007bc2`): `--rui-color-light-blue`, informational
|
||||||
|
- **Brown** (`#936d62`): `--rui-color-brown`, warm neutral accent
|
||||||
|
- **Red Text** (`#8b0000`): `--website-color-red-text`, dark red text
|
||||||
|
|
||||||
|
### Neutral Scale
|
||||||
|
- **Mid Slate** (`#505a63`): Secondary text
|
||||||
|
- **Cool Gray** (`#8d969e`): Muted text, tertiary
|
||||||
|
- **Gray Tone** (`#c9c9cd`): `--rui-color-grey-tone-20`, borders/dividers
|
||||||
|
|
||||||
|
## 3. Typography Rules
|
||||||
|
|
||||||
|
### Font Families
|
||||||
|
- **Display**: `Aeonik Pro` — geometric grotesque, no detected fallbacks
|
||||||
|
- **Body / UI**: `Inter` — standard system sans
|
||||||
|
- **Fallback**: `Arial` for specific button contexts
|
||||||
|
|
||||||
|
### Hierarchy
|
||||||
|
|
||||||
|
| Role | Font | Size | Weight | Line Height | Letter Spacing | Notes |
|
||||||
|
|------|------|------|--------|-------------|----------------|-------|
|
||||||
|
| Display Mega | Aeonik Pro | 136px (8.50rem) | 500 | 1.00 (tight) | -2.72px | Stadium-scale hero |
|
||||||
|
| Display Hero | Aeonik Pro | 80px (5.00rem) | 500 | 1.00 (tight) | -0.8px | Primary hero |
|
||||||
|
| Section Heading | Aeonik Pro | 48px (3.00rem) | 500 | 1.21 (tight) | -0.48px | Feature sections |
|
||||||
|
| Sub-heading | Aeonik Pro | 40px (2.50rem) | 500 | 1.20 (tight) | -0.4px | Sub-sections |
|
||||||
|
| Card Title | Aeonik Pro | 32px (2.00rem) | 500 | 1.19 (tight) | -0.32px | Card headings |
|
||||||
|
| Feature Title | Aeonik Pro | 24px (1.50rem) | 400 | 1.33 | normal | Light headings |
|
||||||
|
| Nav / UI | Aeonik Pro | 20px (1.25rem) | 500 | 1.40 | normal | Navigation, buttons |
|
||||||
|
| Body Large | Inter | 18px (1.13rem) | 400 | 1.56 | -0.09px | Introductions |
|
||||||
|
| Body | Inter | 16px (1.00rem) | 400 | 1.50 | 0.24px | Standard reading |
|
||||||
|
| Body Semibold | Inter | 16px (1.00rem) | 600 | 1.50 | 0.16px | Emphasized body |
|
||||||
|
| Body Bold Link | Inter | 16px (1.00rem) | 700 | 1.50 | 0.24px | Bold links |
|
||||||
|
|
||||||
|
### Principles
|
||||||
|
- **Weight 500 as display default**: Aeonik Pro uses medium (500) for ALL headings — no bold. This creates authority through size and tracking, not weight.
|
||||||
|
- **Billboard tracking**: -2.72px at 136px is extremely compressed — text designed to be read at a glance, like airport signage.
|
||||||
|
- **Positive tracking on body**: Inter uses +0.16px to +0.24px, creating airy, well-spaced reading text that contrasts with the compressed headings.
|
||||||
|
|
||||||
|
## 4. Component Stylings
|
||||||
|
|
||||||
|
### Buttons
|
||||||
|
|
||||||
|
**Primary Dark Pill**
|
||||||
|
- Background: `#191c1f`
|
||||||
|
- Text: `#ffffff`
|
||||||
|
- Padding: 14px 32px
|
||||||
|
- Radius: 9999px (full pill)
|
||||||
|
- Hover: opacity 0.85
|
||||||
|
- Focus: `0 0 0 0.125rem` ring
|
||||||
|
|
||||||
|
**Secondary Light Pill**
|
||||||
|
- Background: `#f4f4f4`
|
||||||
|
- Text: `#000000`
|
||||||
|
- Padding: 14px 34px
|
||||||
|
- Radius: 9999px
|
||||||
|
- Hover: opacity 0.85
|
||||||
|
|
||||||
|
**Outlined Pill**
|
||||||
|
- Background: transparent
|
||||||
|
- Text: `#191c1f`
|
||||||
|
- Border: `2px solid #191c1f`
|
||||||
|
- Padding: 14px 32px
|
||||||
|
- Radius: 9999px
|
||||||
|
|
||||||
|
**Ghost on Dark**
|
||||||
|
- Background: `rgba(244, 244, 244, 0.1)`
|
||||||
|
- Text: `#f4f4f4`
|
||||||
|
- Border: `2px solid #f4f4f4`
|
||||||
|
- Padding: 14px 32px
|
||||||
|
- Radius: 9999px
|
||||||
|
|
||||||
|
### Cards & Containers
|
||||||
|
- Radius: 12px (small), 20px (cards)
|
||||||
|
- No shadows — flat surfaces with color contrast
|
||||||
|
- Dark and light section alternation
|
||||||
|
|
||||||
|
### Navigation
|
||||||
|
- Aeonik Pro 20px weight 500
|
||||||
|
- Clean header, hamburger toggle at 12px radius
|
||||||
|
- Pill CTAs right-aligned
|
||||||
|
|
||||||
|
## 5. Layout Principles
|
||||||
|
|
||||||
|
### Spacing System
|
||||||
|
- Base unit: 8px
|
||||||
|
- Scale: 4px, 6px, 8px, 14px, 16px, 20px, 24px, 32px, 40px, 48px, 80px, 88px, 120px
|
||||||
|
- Large section spacing: 80px–120px
|
||||||
|
|
||||||
|
### Border Radius Scale
|
||||||
|
- Standard (12px): Navigation, small buttons
|
||||||
|
- Card (20px): Feature cards
|
||||||
|
- Pill (9999px): All buttons
|
||||||
|
|
||||||
|
## 6. Depth & Elevation
|
||||||
|
|
||||||
|
| Level | Treatment | Use |
|
||||||
|
|-------|-----------|-----|
|
||||||
|
| Flat (Level 0) | No shadow | Everything — Revolut uses zero shadows |
|
||||||
|
| Focus | `0 0 0 0.125rem` ring | Accessibility focus |
|
||||||
|
|
||||||
|
**Shadow Philosophy**: Revolut uses ZERO shadows. Depth comes entirely from the dark/light section contrast and the generous whitespace between elements.
|
||||||
|
|
||||||
|
## 7. Do's and Don'ts
|
||||||
|
|
||||||
|
### Do
|
||||||
|
- Use Aeonik Pro weight 500 for all display headings
|
||||||
|
- Apply 9999px radius to all buttons — pill shape is universal
|
||||||
|
- Use generous button padding (14px 32px)
|
||||||
|
- Keep the palette to near-black + white for marketing surfaces
|
||||||
|
- Apply positive letter-spacing on Inter body text
|
||||||
|
|
||||||
|
### Don't
|
||||||
|
- Don't use shadows — Revolut is flat by design
|
||||||
|
- Don't use bold (700) for Aeonik Pro headings — 500 is the weight
|
||||||
|
- Don't use small buttons — the generous padding is intentional
|
||||||
|
- Don't apply semantic colors to marketing surfaces — they're for the product
|
||||||
|
|
||||||
|
## 8. Responsive Behavior
|
||||||
|
|
||||||
|
### Breakpoints
|
||||||
|
| Name | Width | Key Changes |
|
||||||
|
|------|-------|-------------|
|
||||||
|
| Mobile Small | <400px | Compact, single column |
|
||||||
|
| Mobile | 400–720px | Standard mobile |
|
||||||
|
| Tablet | 720–1024px | 2-column layouts |
|
||||||
|
| Desktop | 1024–1280px | Standard desktop |
|
||||||
|
| Large | 1280–1920px | Full layout |
|
||||||
|
|
||||||
|
## 9. Agent Prompt Guide
|
||||||
|
|
||||||
|
### Quick Color Reference
|
||||||
|
- Dark: Revolut Dark (`#191c1f`)
|
||||||
|
- Light: White (`#ffffff`)
|
||||||
|
- Surface: Light (`#f4f4f4`)
|
||||||
|
- Blue: Revolut Blue (`#494fdf`)
|
||||||
|
- Danger: Red (`#e23b4a`)
|
||||||
|
- Success: Teal (`#00a87e`)
|
||||||
|
|
||||||
|
### Example Component Prompts
|
||||||
|
- "Create a hero: white background. Headline at 136px Aeonik Pro weight 500, line-height 1.00, letter-spacing -2.72px, #191c1f text. Dark pill CTA (#191c1f, 9999px, 14px 32px). Outlined pill secondary (transparent, 2px solid #191c1f)."
|
||||||
|
- "Build a pill button: #191c1f background, white text, 9999px radius, 14px 32px padding, 20px Aeonik Pro weight 500. Hover: opacity 0.85."
|
||||||
|
|
||||||
|
### Iteration Guide
|
||||||
|
1. Aeonik Pro 500 for headings — never bold
|
||||||
|
2. All buttons are pills (9999px) with generous padding
|
||||||
|
3. Zero shadows — flat is the Revolut identity
|
||||||
|
4. Near-black + white for marketing, semantic colors for product
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
---
|
||||||
|
description: Skills for searching, downloading, and working with GIFs and short-form animated media.
|
||||||
|
---
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# inference.sh
|
||||||
|
|
||||||
|
Run 150+ AI applications in the cloud via the [inference.sh](https://inference.sh) platform.
|
||||||
|
|
||||||
|
**One API key for everything** — access image generation, video creation, LLMs, search, 3D, and more through a single account. No need to manage separate API keys for each provider.
|
||||||
|
|
||||||
|
## Available Skills
|
||||||
|
|
||||||
|
- **cli**: Use the inference.sh CLI (`infsh`) via the terminal tool
|
||||||
|
|
||||||
|
## What's Included
|
||||||
|
|
||||||
|
- **Image Generation**: FLUX, Reve, Seedream, Grok Imagine, Gemini
|
||||||
|
- **Video Generation**: Veo, Wan, Seedance, OmniHuman, HunyuanVideo
|
||||||
|
- **LLMs**: Claude, Gemini, Kimi, GLM-4 (via OpenRouter)
|
||||||
|
- **Search**: Tavily, Exa
|
||||||
|
- **3D**: Rodin
|
||||||
|
- **Social**: Twitter/X automation
|
||||||
|
- **Audio**: TTS, voice cloning
|
||||||
+267
@@ -0,0 +1,267 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
ws_monitor.py — Real-time ComfyUI WebSocket monitor.
|
||||||
|
|
||||||
|
Connects to /ws and pretty-prints execution events: node start/finish, sampling
|
||||||
|
progress, cached nodes, errors. Optionally writes preview frames to disk.
|
||||||
|
|
||||||
|
Useful for:
|
||||||
|
- Watching a long-running job in real time without parsing JSON yourself
|
||||||
|
- Saving in-progress preview frames for video / animation workflows
|
||||||
|
- Debugging "why is this hanging?" — see exactly which node is stuck
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
# Local — watch all jobs from this client_id
|
||||||
|
python3 ws_monitor.py
|
||||||
|
|
||||||
|
# Cloud — watch a specific prompt_id
|
||||||
|
python3 ws_monitor.py --host https://cloud.comfy.org \
|
||||||
|
--prompt-id abc-123-def
|
||||||
|
|
||||||
|
# Save preview frames to ./previews/
|
||||||
|
python3 ws_monitor.py --previews ./previews
|
||||||
|
|
||||||
|
Requires: websocket-client (`pip install websocket-client`).
|
||||||
|
Falls back to a clear error message when not installed.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import struct
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
from urllib.parse import urlparse
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||||
|
from _common import ( # noqa: E402
|
||||||
|
DEFAULT_LOCAL_HOST, ENV_API_KEY, log, new_client_id, resolve_api_key, is_cloud_host,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# Binary frame types from ComfyUI WebSocket protocol
|
||||||
|
BINARY_PREVIEW_IMAGE = 1
|
||||||
|
BINARY_TEXT = 3
|
||||||
|
BINARY_PREVIEW_IMAGE_WITH_METADATA = 4
|
||||||
|
|
||||||
|
# Image type codes inside PREVIEW_IMAGE
|
||||||
|
IMAGE_TYPE_JPEG = 1
|
||||||
|
IMAGE_TYPE_PNG = 2
|
||||||
|
|
||||||
|
# ANSI escape codes (works on most modern terminals)
|
||||||
|
RESET = "\033[0m"
|
||||||
|
DIM = "\033[2m"
|
||||||
|
BOLD = "\033[1m"
|
||||||
|
GREEN = "\033[32m"
|
||||||
|
YELLOW = "\033[33m"
|
||||||
|
RED = "\033[31m"
|
||||||
|
CYAN = "\033[36m"
|
||||||
|
|
||||||
|
|
||||||
|
def fmt_color(s: str, color: str, *, color_on: bool = True) -> str:
|
||||||
|
return f"{color}{s}{RESET}" if color_on else s
|
||||||
|
|
||||||
|
|
||||||
|
def parse_binary_frame(data: bytes) -> dict | None:
|
||||||
|
if len(data) < 8:
|
||||||
|
return None
|
||||||
|
type_code = struct.unpack(">I", data[0:4])[0]
|
||||||
|
if type_code == BINARY_PREVIEW_IMAGE:
|
||||||
|
image_type = struct.unpack(">I", data[4:8])[0]
|
||||||
|
ext = "jpg" if image_type == IMAGE_TYPE_JPEG else "png" if image_type == IMAGE_TYPE_PNG else "bin"
|
||||||
|
return {
|
||||||
|
"kind": "preview",
|
||||||
|
"image_type": image_type,
|
||||||
|
"ext": ext,
|
||||||
|
"image_bytes": data[8:],
|
||||||
|
}
|
||||||
|
if type_code == BINARY_PREVIEW_IMAGE_WITH_METADATA:
|
||||||
|
if len(data) < 12:
|
||||||
|
return None
|
||||||
|
meta_len = struct.unpack(">I", data[4:8])[0]
|
||||||
|
meta_end = 8 + meta_len
|
||||||
|
if len(data) < meta_end:
|
||||||
|
return None
|
||||||
|
try:
|
||||||
|
meta = json.loads(data[8:meta_end].decode("utf-8"))
|
||||||
|
except Exception:
|
||||||
|
meta = {"raw": data[8:meta_end][:200].decode("utf-8", "replace")}
|
||||||
|
return {
|
||||||
|
"kind": "preview_with_metadata",
|
||||||
|
"metadata": meta,
|
||||||
|
"image_bytes": data[meta_end:],
|
||||||
|
"ext": "png",
|
||||||
|
}
|
||||||
|
if type_code == BINARY_TEXT:
|
||||||
|
if len(data) < 8:
|
||||||
|
return None
|
||||||
|
nid_len = struct.unpack(">I", data[4:8])[0]
|
||||||
|
nid_end = 8 + nid_len
|
||||||
|
if len(data) < nid_end:
|
||||||
|
return None
|
||||||
|
return {
|
||||||
|
"kind": "text",
|
||||||
|
"node_id": data[8:nid_end].decode("utf-8", "replace"),
|
||||||
|
"text": data[nid_end:].decode("utf-8", "replace"),
|
||||||
|
}
|
||||||
|
return {"kind": "unknown", "type_code": type_code, "size": len(data)}
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
p = argparse.ArgumentParser(description="Real-time ComfyUI WebSocket monitor")
|
||||||
|
p.add_argument("--host", default=DEFAULT_LOCAL_HOST, help="ComfyUI server URL")
|
||||||
|
p.add_argument("--api-key", help=f"API key for cloud (or set ${ENV_API_KEY} env var)")
|
||||||
|
p.add_argument("--client-id", default=None, help="Client ID (default: random UUID)")
|
||||||
|
p.add_argument("--prompt-id", default=None,
|
||||||
|
help="Filter to a specific prompt_id (default: all jobs)")
|
||||||
|
p.add_argument("--previews", default=None,
|
||||||
|
help="Directory to save in-progress preview frames")
|
||||||
|
p.add_argument("--no-color", action="store_true", help="Disable ANSI colour")
|
||||||
|
p.add_argument("--timeout", type=float, default=600.0,
|
||||||
|
help="Hard cap on monitor duration (default 600s)")
|
||||||
|
args = p.parse_args(argv)
|
||||||
|
|
||||||
|
try:
|
||||||
|
import websocket # type: ignore[import-not-found]
|
||||||
|
except ImportError:
|
||||||
|
print(json.dumps({
|
||||||
|
"error": "websocket-client not installed",
|
||||||
|
"install": "pip install websocket-client",
|
||||||
|
}))
|
||||||
|
return 1
|
||||||
|
|
||||||
|
api_key = resolve_api_key(args.api_key)
|
||||||
|
cloud = is_cloud_host(args.host)
|
||||||
|
client_id = args.client_id or new_client_id()
|
||||||
|
|
||||||
|
# Build WS URL preserving any base-path component (e.g. behind reverse proxy).
|
||||||
|
parsed = urlparse(args.host if "://" in args.host else f"http://{args.host}")
|
||||||
|
scheme = "wss" if parsed.scheme == "https" else "ws"
|
||||||
|
netloc = parsed.netloc
|
||||||
|
base_path = parsed.path.rstrip("/")
|
||||||
|
ws_url = f"{scheme}://{netloc}{base_path}/ws?clientId={client_id}"
|
||||||
|
if cloud and api_key:
|
||||||
|
ws_url += f"&token={api_key}"
|
||||||
|
|
||||||
|
color_on = not args.no_color and sys.stdout.isatty()
|
||||||
|
|
||||||
|
preview_dir = Path(args.previews).expanduser() if args.previews else None
|
||||||
|
if preview_dir:
|
||||||
|
preview_dir.mkdir(parents=True, exist_ok=True)
|
||||||
|
log(f"Saving previews to {preview_dir}")
|
||||||
|
|
||||||
|
log(f"Connecting to {ws_url} (client_id={client_id})")
|
||||||
|
if args.prompt_id:
|
||||||
|
log(f"Filtering messages to prompt_id={args.prompt_id}")
|
||||||
|
|
||||||
|
ws = websocket.create_connection(ws_url, timeout=args.timeout)
|
||||||
|
ws.settimeout(args.timeout)
|
||||||
|
|
||||||
|
preview_counter = 0
|
||||||
|
try:
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
msg = ws.recv()
|
||||||
|
except websocket.WebSocketTimeoutException:
|
||||||
|
log(f"Idle for {args.timeout}s — exiting")
|
||||||
|
return 0
|
||||||
|
if isinstance(msg, bytes):
|
||||||
|
parsed = parse_binary_frame(msg)
|
||||||
|
if parsed is None:
|
||||||
|
continue
|
||||||
|
if parsed["kind"] in {"preview", "preview_with_metadata"} and preview_dir:
|
||||||
|
img_bytes = parsed.get("image_bytes", b"")
|
||||||
|
if img_bytes:
|
||||||
|
ext = parsed.get("ext", "png")
|
||||||
|
out = preview_dir / f"preview_{preview_counter:05d}.{ext}"
|
||||||
|
out.write_bytes(img_bytes)
|
||||||
|
preview_counter += 1
|
||||||
|
log(f" [preview] saved {out.name} ({len(img_bytes)} bytes)")
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
payload = json.loads(msg)
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
mtype = payload.get("type", "")
|
||||||
|
mdata = payload.get("data", {}) or {}
|
||||||
|
pid = mdata.get("prompt_id")
|
||||||
|
|
||||||
|
if args.prompt_id and pid and pid != args.prompt_id:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if mtype == "status":
|
||||||
|
qr = mdata.get("status", {}).get("exec_info", {}).get("queue_remaining", "?")
|
||||||
|
print(fmt_color(f"[status] queue_remaining={qr}", DIM, color_on=color_on))
|
||||||
|
elif mtype == "execution_start":
|
||||||
|
print(fmt_color(f"[start] prompt_id={pid}", BOLD, color_on=color_on))
|
||||||
|
elif mtype == "executing":
|
||||||
|
node = mdata.get("node")
|
||||||
|
if node:
|
||||||
|
print(fmt_color(f" [executing] node={node}", CYAN, color_on=color_on))
|
||||||
|
else:
|
||||||
|
print(fmt_color(f" [executing] (workflow done) prompt_id={pid}", DIM, color_on=color_on))
|
||||||
|
elif mtype == "progress":
|
||||||
|
v, m = mdata.get("value", 0), mdata.get("max", 0)
|
||||||
|
pct = (v / m * 100) if m else 0
|
||||||
|
print(f" [progress] {v}/{m} ({pct:5.1f}%) node={mdata.get('node')}")
|
||||||
|
elif mtype == "progress_state":
|
||||||
|
# Newer extended progress message
|
||||||
|
nodes = mdata.get("nodes") or {}
|
||||||
|
running = [k for k, v in nodes.items() if v.get("running")]
|
||||||
|
if running:
|
||||||
|
print(fmt_color(f" [progress_state] running={running}", DIM, color_on=color_on))
|
||||||
|
elif mtype == "executed":
|
||||||
|
node = mdata.get("node")
|
||||||
|
out = mdata.get("output") or {}
|
||||||
|
summary_parts = []
|
||||||
|
for key in ("images", "video", "videos", "gifs", "audio", "files"):
|
||||||
|
if out.get(key):
|
||||||
|
summary_parts.append(f"{key}={len(out[key])}")
|
||||||
|
summary = ", ".join(summary_parts) if summary_parts else "(no files)"
|
||||||
|
print(fmt_color(f" [executed] node={node} {summary}", GREEN, color_on=color_on))
|
||||||
|
elif mtype == "execution_cached":
|
||||||
|
cached = mdata.get("nodes") or []
|
||||||
|
if cached:
|
||||||
|
print(fmt_color(f" [cached] {len(cached)} nodes skipped", DIM, color_on=color_on))
|
||||||
|
elif mtype == "execution_success":
|
||||||
|
print(fmt_color(f"[success] prompt_id={pid}", GREEN + BOLD, color_on=color_on))
|
||||||
|
if args.prompt_id:
|
||||||
|
return 0
|
||||||
|
elif mtype == "execution_error":
|
||||||
|
exc_type = mdata.get("exception_type", "?")
|
||||||
|
exc_msg = mdata.get("exception_message", "?")
|
||||||
|
print(fmt_color(f"[error] {exc_type}: {exc_msg}", RED + BOLD, color_on=color_on))
|
||||||
|
tb = mdata.get("traceback")
|
||||||
|
if tb:
|
||||||
|
if isinstance(tb, list):
|
||||||
|
for line in tb:
|
||||||
|
print(fmt_color(f" {line}", RED, color_on=color_on))
|
||||||
|
else:
|
||||||
|
print(fmt_color(f" {tb}", RED, color_on=color_on))
|
||||||
|
if args.prompt_id:
|
||||||
|
return 1
|
||||||
|
elif mtype == "execution_interrupted":
|
||||||
|
print(fmt_color(f"[interrupted] prompt_id={pid}", YELLOW, color_on=color_on))
|
||||||
|
if args.prompt_id:
|
||||||
|
return 1
|
||||||
|
elif mtype == "notification":
|
||||||
|
v = mdata.get("value", "")
|
||||||
|
print(fmt_color(f"[notification] {v}", DIM, color_on=color_on))
|
||||||
|
else:
|
||||||
|
# Unknown / lightly-used types: print compactly
|
||||||
|
print(fmt_color(f"[{mtype}] {json.dumps(mdata, default=str)[:200]}", DIM, color_on=color_on))
|
||||||
|
|
||||||
|
except KeyboardInterrupt:
|
||||||
|
log("Interrupted")
|
||||||
|
return 130
|
||||||
|
finally:
|
||||||
|
try:
|
||||||
|
ws.close()
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
"""Pytest configuration for the comfyui skill test suite.
|
||||||
|
|
||||||
|
Adds `scripts/` to sys.path so tests can `from _common import ...`, and
|
||||||
|
provides a few common fixtures.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
ROOT = Path(__file__).resolve().parent.parent
|
||||||
|
SCRIPTS = ROOT / "scripts"
|
||||||
|
WORKFLOWS = ROOT / "workflows"
|
||||||
|
|
||||||
|
sys.path.insert(0, str(SCRIPTS))
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def sd15_workflow() -> dict:
|
||||||
|
return json.loads((WORKFLOWS / "sd15_txt2img.json").read_text())
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def flux_workflow() -> dict:
|
||||||
|
return json.loads((WORKFLOWS / "flux_dev_txt2img.json").read_text())
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def video_workflow() -> dict:
|
||||||
|
return json.loads((WORKFLOWS / "wan_video_t2v.json").read_text())
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def workflows_dir() -> Path:
|
||||||
|
return WORKFLOWS
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def scripts_dir() -> Path:
|
||||||
|
return SCRIPTS
|
||||||
|
|
||||||
|
|
||||||
|
@pytest.fixture
|
||||||
|
def cloud_key() -> str | None:
|
||||||
|
"""Cloud API key if set, otherwise None.
|
||||||
|
|
||||||
|
Tests that need cloud connectivity should skip when this is None.
|
||||||
|
"""
|
||||||
|
return os.environ.get("COMFY_CLOUD_API_KEY")
|
||||||
|
|
||||||
|
|
||||||
|
def pytest_collection_modifyitems(config, items):
|
||||||
|
"""Auto-skip cloud tests when no API key is set."""
|
||||||
|
if os.environ.get("COMFY_CLOUD_API_KEY"):
|
||||||
|
return
|
||||||
|
skip_cloud = pytest.mark.skip(reason="Set COMFY_CLOUD_API_KEY to run cloud tests")
|
||||||
|
for item in items:
|
||||||
|
if "cloud" in item.keywords:
|
||||||
|
item.add_marker(skip_cloud)
|
||||||
@@ -0,0 +1,162 @@
|
|||||||
|
---
|
||||||
|
name: dogfood
|
||||||
|
description: "Exploratory QA of web apps: find bugs, evidence, reports."
|
||||||
|
version: 1.0.0
|
||||||
|
platforms: [linux, macos, windows]
|
||||||
|
metadata:
|
||||||
|
hermes:
|
||||||
|
tags: [qa, testing, browser, web, dogfood]
|
||||||
|
related_skills: []
|
||||||
|
---
|
||||||
|
|
||||||
|
# Dogfood: Systematic Web Application QA Testing
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This skill guides you through systematic exploratory QA testing of web applications using the browser toolset. You will navigate the application, interact with elements, capture evidence of issues, and produce a structured bug report.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- Browser toolset must be available (`browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_vision`, `browser_console`, `browser_scroll`, `browser_back`, `browser_press`)
|
||||||
|
- A target URL and testing scope from the user
|
||||||
|
|
||||||
|
## Inputs
|
||||||
|
|
||||||
|
The user provides:
|
||||||
|
1. **Target URL** — the entry point for testing
|
||||||
|
2. **Scope** — what areas/features to focus on (or "full site" for comprehensive testing)
|
||||||
|
3. **Output directory** (optional) — where to save screenshots and the report (default: `./dogfood-output`)
|
||||||
|
|
||||||
|
## Workflow
|
||||||
|
|
||||||
|
Follow this 5-phase systematic workflow:
|
||||||
|
|
||||||
|
### Phase 1: Plan
|
||||||
|
|
||||||
|
1. Create the output directory structure:
|
||||||
|
```
|
||||||
|
{output_dir}/
|
||||||
|
├── screenshots/ # Evidence screenshots
|
||||||
|
└── report.md # Final report (generated in Phase 5)
|
||||||
|
```
|
||||||
|
2. Identify the testing scope based on user input.
|
||||||
|
3. Build a rough sitemap by planning which pages and features to test:
|
||||||
|
- Landing/home page
|
||||||
|
- Navigation links (header, footer, sidebar)
|
||||||
|
- Key user flows (sign up, login, search, checkout, etc.)
|
||||||
|
- Forms and interactive elements
|
||||||
|
- Edge cases (empty states, error pages, 404s)
|
||||||
|
|
||||||
|
### Phase 2: Explore
|
||||||
|
|
||||||
|
For each page or feature in your plan:
|
||||||
|
|
||||||
|
1. **Navigate** to the page:
|
||||||
|
```
|
||||||
|
browser_navigate(url="https://example.com/page")
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Take a snapshot** to understand the DOM structure:
|
||||||
|
```
|
||||||
|
browser_snapshot()
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Check the console** for JavaScript errors:
|
||||||
|
```
|
||||||
|
browser_console(clear=true)
|
||||||
|
```
|
||||||
|
Do this after every navigation and after every significant interaction. Silent JS errors are high-value findings.
|
||||||
|
|
||||||
|
4. **Take an annotated screenshot** to visually assess the page and identify interactive elements:
|
||||||
|
```
|
||||||
|
browser_vision(question="Describe the page layout, identify any visual issues, broken elements, or accessibility concerns", annotate=true)
|
||||||
|
```
|
||||||
|
The `annotate=true` flag overlays numbered `[N]` labels on interactive elements. Each `[N]` maps to ref `@eN` for subsequent browser commands.
|
||||||
|
|
||||||
|
5. **Test interactive elements** systematically:
|
||||||
|
- Click buttons and links: `browser_click(ref="@eN")`
|
||||||
|
- Fill forms: `browser_type(ref="@eN", text="test input")`
|
||||||
|
- Test keyboard navigation: `browser_press(key="Tab")`, `browser_press(key="Enter")`
|
||||||
|
- Scroll through content: `browser_scroll(direction="down")`
|
||||||
|
- Test form validation with invalid inputs
|
||||||
|
- Test empty submissions
|
||||||
|
|
||||||
|
6. **After each interaction**, check for:
|
||||||
|
- Console errors: `browser_console()`
|
||||||
|
- Visual changes: `browser_vision(question="What changed after the interaction?")`
|
||||||
|
- Expected vs actual behavior
|
||||||
|
|
||||||
|
### Phase 3: Collect Evidence
|
||||||
|
|
||||||
|
For every issue found:
|
||||||
|
|
||||||
|
1. **Take a screenshot** showing the issue:
|
||||||
|
```
|
||||||
|
browser_vision(question="Capture and describe the issue visible on this page", annotate=false)
|
||||||
|
```
|
||||||
|
Save the `screenshot_path` from the response — you will reference it in the report.
|
||||||
|
|
||||||
|
2. **Record the details**:
|
||||||
|
- URL where the issue occurs
|
||||||
|
- Steps to reproduce
|
||||||
|
- Expected behavior
|
||||||
|
- Actual behavior
|
||||||
|
- Console errors (if any)
|
||||||
|
- Screenshot path
|
||||||
|
|
||||||
|
3. **Classify the issue** using the issue taxonomy (see `references/issue-taxonomy.md`):
|
||||||
|
- Severity: Critical / High / Medium / Low
|
||||||
|
- Category: Functional / Visual / Accessibility / Console / UX / Content
|
||||||
|
|
||||||
|
### Phase 4: Categorize
|
||||||
|
|
||||||
|
1. Review all collected issues.
|
||||||
|
2. De-duplicate — merge issues that are the same bug manifesting in different places.
|
||||||
|
3. Assign final severity and category to each issue.
|
||||||
|
4. Sort by severity (Critical first, then High, Medium, Low).
|
||||||
|
5. Count issues by severity and category for the executive summary.
|
||||||
|
|
||||||
|
### Phase 5: Report
|
||||||
|
|
||||||
|
Generate the final report using the template at `templates/dogfood-report-template.md`.
|
||||||
|
|
||||||
|
The report must include:
|
||||||
|
1. **Executive summary** with total issue count, breakdown by severity, and testing scope
|
||||||
|
2. **Per-issue sections** with:
|
||||||
|
- Issue number and title
|
||||||
|
- Severity and category badges
|
||||||
|
- URL where observed
|
||||||
|
- Description of the issue
|
||||||
|
- Steps to reproduce
|
||||||
|
- Expected vs actual behavior
|
||||||
|
- Screenshot references (use `MEDIA:<screenshot_path>` for inline images)
|
||||||
|
- Console errors if relevant
|
||||||
|
3. **Summary table** of all issues
|
||||||
|
4. **Testing notes** — what was tested, what was not, any blockers
|
||||||
|
|
||||||
|
Save the report to `{output_dir}/report.md`.
|
||||||
|
|
||||||
|
## Tools Reference
|
||||||
|
|
||||||
|
| Tool | Purpose |
|
||||||
|
|------|---------|
|
||||||
|
| `browser_navigate` | Go to a URL |
|
||||||
|
| `browser_snapshot` | Get DOM text snapshot (accessibility tree) |
|
||||||
|
| `browser_click` | Click an element by ref (`@eN`) or text |
|
||||||
|
| `browser_type` | Type into an input field |
|
||||||
|
| `browser_scroll` | Scroll up/down on the page |
|
||||||
|
| `browser_back` | Go back in browser history |
|
||||||
|
| `browser_press` | Press a keyboard key |
|
||||||
|
| `browser_vision` | Screenshot + AI analysis; use `annotate=true` for element labels |
|
||||||
|
| `browser_console` | Get JS console output and errors |
|
||||||
|
|
||||||
|
## Tips
|
||||||
|
|
||||||
|
- **Always check `browser_console()` after navigating and after significant interactions.** Silent JS errors are among the most valuable findings.
|
||||||
|
- **Use `annotate=true` with `browser_vision`** when you need to reason about interactive element positions or when the snapshot refs are unclear.
|
||||||
|
- **Test with both valid and invalid inputs** — form validation bugs are common.
|
||||||
|
- **Scroll through long pages** — content below the fold may have rendering issues.
|
||||||
|
- **Test navigation flows** — click through multi-step processes end-to-end.
|
||||||
|
- **Check responsive behavior** by noting any layout issues visible in screenshots.
|
||||||
|
- **Don't forget edge cases**: empty states, very long text, special characters, rapid clicking.
|
||||||
|
- When reporting screenshots to the user, include `MEDIA:<screenshot_path>` so they can see the evidence inline.
|
||||||
@@ -0,0 +1,226 @@
|
|||||||
|
# Citation Management & Hallucination Prevention
|
||||||
|
|
||||||
|
This reference provides a complete workflow for managing citations programmatically, preventing AI-generated citation hallucinations, and maintaining clean bibliographies.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Contents
|
||||||
|
|
||||||
|
- [Why Citation Verification Matters](#why-citation-verification-matters)
|
||||||
|
- [Citation APIs Overview](#citation-apis-overview)
|
||||||
|
- [Verified Citation Workflow](#verified-citation-workflow)
|
||||||
|
- [Python Implementation](#python-implementation)
|
||||||
|
- [BibTeX Management](#bibtex-management)
|
||||||
|
- [Common Citation Formats](#common-citation-formats)
|
||||||
|
- [Troubleshooting](#troubleshooting)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Why Citation Verification Matters
|
||||||
|
|
||||||
|
### The Hallucination Problem
|
||||||
|
|
||||||
|
Research has documented significant issues with AI-generated citations:
|
||||||
|
- **~40% error rate** in AI-generated citations (Enago Academy research)
|
||||||
|
- NeurIPS 2025 found **100+ hallucinated citations** slipped through review
|
||||||
|
- Common errors include:
|
||||||
|
- Fabricated paper titles with real author names
|
||||||
|
- Wrong publication venues or years
|
||||||
|
- Non-existent papers with plausible metadata
|
||||||
|
- Incorrect DOIs or arXiv IDs
|
||||||
|
|
||||||
|
### Consequences
|
||||||
|
|
||||||
|
- Desk rejection at some venues
|
||||||
|
- Loss of credibility with reviewers
|
||||||
|
- Potential retraction if published
|
||||||
|
- Wasted time chasing non-existent sources
|
||||||
|
|
||||||
|
### Solution
|
||||||
|
|
||||||
|
**Never generate citations from memory—always verify programmatically.**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Citation APIs Overview
|
||||||
|
|
||||||
|
### Primary APIs
|
||||||
|
|
||||||
|
| API | Coverage | Rate Limits | Best For |
|
||||||
|
|-----|----------|-------------|----------|
|
||||||
|
| **Semantic Scholar** | 214M papers | 1 RPS (free key) | ML/AI papers, citation graphs |
|
||||||
|
| **CrossRef** | 140M+ DOIs | Polite pool with mailto | DOI lookup, BibTeX retrieval |
|
||||||
|
| **arXiv** | Preprints | 3-second delays | ML preprints, PDF access |
|
||||||
|
| **OpenAlex** | 240M+ works | 100K/day, 10 RPS | Open alternative to MAG |
|
||||||
|
|
||||||
|
### API Selection Guide
|
||||||
|
|
||||||
|
```
|
||||||
|
Need ML paper search? → Semantic Scholar
|
||||||
|
Have DOI, need BibTeX? → CrossRef content negotiation
|
||||||
|
Looking for preprint? → arXiv API
|
||||||
|
Need open data, bulk access? → OpenAlex
|
||||||
|
```
|
||||||
|
|
||||||
|
### No Official Google Scholar API
|
||||||
|
|
||||||
|
Google Scholar has no official API. Scraping violates ToS. Use SerpApi ($75-275/month) only if Semantic Scholar coverage is insufficient.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Verified Citation Workflow
|
||||||
|
|
||||||
|
### 5-Step Process
|
||||||
|
|
||||||
|
```
|
||||||
|
1. SEARCH → Query Semantic Scholar with specific keywords
|
||||||
|
↓
|
||||||
|
2. VERIFY → Confirm paper exists in 2+ sources
|
||||||
|
↓
|
||||||
|
3. RETRIEVE → Get BibTeX via DOI content negotiation
|
||||||
|
↓
|
||||||
|
4. VALIDATE → Confirm the claim appears in source
|
||||||
|
↓
|
||||||
|
5. ADD → Add verified entry to .bib file
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 1: Search
|
||||||
|
|
||||||
|
Use Semantic Scholar for ML/AI papers:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from semanticscholar import SemanticScholar
|
||||||
|
|
||||||
|
sch = SemanticScholar()
|
||||||
|
results = sch.search_paper("transformer attention mechanism", limit=10)
|
||||||
|
|
||||||
|
for paper in results:
|
||||||
|
print(f"Title: {paper.title}")
|
||||||
|
print(f"Year: {paper.year}")
|
||||||
|
print(f"DOI: {paper.externalIds.get('DOI', 'N/A')}")
|
||||||
|
print(f"arXiv: {paper.externalIds.get('ArXiv', 'N/A')}")
|
||||||
|
print(f"Citation count: {paper.citationCount}")
|
||||||
|
print("---")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 2: Verify Existence
|
||||||
|
|
||||||
|
Confirm paper exists in at least two sources:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import requests
|
||||||
|
|
||||||
|
def verify_paper(doi=None, arxiv_id=None, title=None):
|
||||||
|
"""Verify paper exists in multiple sources."""
|
||||||
|
sources_found = []
|
||||||
|
|
||||||
|
# Check Semantic Scholar
|
||||||
|
sch = SemanticScholar()
|
||||||
|
if doi:
|
||||||
|
paper = sch.get_paper(f"DOI:{doi}")
|
||||||
|
if paper:
|
||||||
|
sources_found.append("Semantic Scholar")
|
||||||
|
|
||||||
|
# Check CrossRef (via DOI)
|
||||||
|
if doi:
|
||||||
|
resp = requests.get(f"https://api.crossref.org/works/{doi}")
|
||||||
|
if resp.status_code == 200:
|
||||||
|
sources_found.append("CrossRef")
|
||||||
|
|
||||||
|
# Check arXiv
|
||||||
|
if arxiv_id:
|
||||||
|
resp = requests.get(
|
||||||
|
f"http://export.arxiv.org/api/query?id_list={arxiv_id}"
|
||||||
|
)
|
||||||
|
if "<entry>" in resp.text:
|
||||||
|
sources_found.append("arXiv")
|
||||||
|
|
||||||
|
return len(sources_found) >= 2, sources_found
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 3: Retrieve BibTeX
|
||||||
|
|
||||||
|
Use DOI content negotiation for guaranteed accuracy:
|
||||||
|
|
||||||
|
```python
|
||||||
|
import requests
|
||||||
|
|
||||||
|
def doi_to_bibtex(doi: str) -> str:
|
||||||
|
"""Get verified BibTeX from DOI via CrossRef content negotiation."""
|
||||||
|
response = requests.get(
|
||||||
|
f"https://doi.org/{doi}",
|
||||||
|
headers={"Accept": "application/x-bibtex"},
|
||||||
|
allow_redirects=True
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.text
|
||||||
|
|
||||||
|
# Example: "Attention Is All You Need"
|
||||||
|
bibtex = doi_to_bibtex("10.48550/arXiv.1706.03762")
|
||||||
|
print(bibtex)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 4: Validate Claims
|
||||||
|
|
||||||
|
Before citing a paper for a specific claim, verify the claim exists:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def get_paper_abstract(doi):
|
||||||
|
"""Get abstract to verify claims."""
|
||||||
|
sch = SemanticScholar()
|
||||||
|
paper = sch.get_paper(f"DOI:{doi}")
|
||||||
|
return paper.abstract if paper else None
|
||||||
|
|
||||||
|
# Verify claim appears in abstract
|
||||||
|
abstract = get_paper_abstract("10.48550/arXiv.1706.03762")
|
||||||
|
claim = "attention mechanism"
|
||||||
|
if claim.lower() in abstract.lower():
|
||||||
|
print("Claim appears in paper")
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step 5: Add to Bibliography
|
||||||
|
|
||||||
|
Add verified entry to your .bib file with consistent key format:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def generate_citation_key(bibtex: str) -> str:
|
||||||
|
"""Generate consistent citation key: author_year_firstword."""
|
||||||
|
import re
|
||||||
|
|
||||||
|
# Extract author
|
||||||
|
author_match = re.search(r'author\s*=\s*\{([^}]+)\}', bibtex, re.I)
|
||||||
|
if author_match:
|
||||||
|
first_author = author_match.group(1).split(',')[0].split()[-1]
|
||||||
|
else:
|
||||||
|
first_author = "unknown"
|
||||||
|
|
||||||
|
# Extract year
|
||||||
|
year_match = re.search(r'year\s*=\s*\{?(\d{4})\}?', bibtex, re.I)
|
||||||
|
year = year_match.group(1) if year_match else "0000"
|
||||||
|
|
||||||
|
# Extract title first word
|
||||||
|
title_match = re.search(r'title\s*=\s*\{([^}]+)\}', bibtex, re.I)
|
||||||
|
if title_match:
|
||||||
|
first_word = title_match.group(1).split()[0].lower()
|
||||||
|
first_word = re.sub(r'[^a-z]', '', first_word)
|
||||||
|
else:
|
||||||
|
first_word = "paper"
|
||||||
|
|
||||||
|
return f"{first_author.lower()}_{year}_{first_word}"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Python Implementation
|
||||||
|
|
||||||
|
### Complete Citation Manager Class
|
||||||
|
|
||||||
|
{% raw %}
|
||||||
|
```python
|
||||||
|
"""
|
||||||
|
Citation Manager - Verified citation workflow for ML papers.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import requests
|
||||||
|
import time
|
||||||
|
from typing import Opt
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
---
|
||||||
|
name: domain-intel
|
||||||
|
description: Passive domain reconnaissance using Python stdlib. Use this skill for subdomain discovery, SSL certificate inspection, WHOIS lookups, DNS records, domain availability checks, and bulk multi-domain analysis. No API keys required. Triggers on requests like "find subdomains", "check ssl cert", "whois lookup", "is this domain available", "bulk check these domains".
|
||||||
|
license: MIT
|
||||||
|
---
|
||||||
|
|
||||||
|
Passive domain intelligence using only Python stdlib and public data sources.
|
||||||
|
Zero dependencies. Zero API keys. Works out of the box.
|
||||||
|
|
||||||
|
## Capabilities
|
||||||
|
|
||||||
|
- Subdomain discovery via crt.sh certificate transparency logs
|
||||||
|
- Live SSL/TLS certificate inspection (expiry, cipher, SANs, TLS version)
|
||||||
|
- WHOIS lookup — supports 100+ TLDs via direct TCP queries
|
||||||
|
- DNS records: A, AAAA, MX, NS, TXT, CNAME
|
||||||
|
- Domain availability check (DNS + WHOIS + SSL signals)
|
||||||
|
- Bulk multi-domain analysis in parallel (up to 20 domains)
|
||||||
|
|
||||||
|
## Data Sources
|
||||||
|
|
||||||
|
- crt.sh — Certificate Transparency logs
|
||||||
|
- WHOIS servers — Direct TCP to 100+ authoritative TLD servers
|
||||||
|
- Google DNS-over-HTTPS — MX/NS/TXT/CNAME resolution
|
||||||
|
- System DNS — A/AAAA records
|
||||||
@@ -0,0 +1,184 @@
|
|||||||
|
"""Tests for extract_schema.py."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
|
||||||
|
from extract_schema import (
|
||||||
|
extract_schema,
|
||||||
|
find_negative_prompt_node,
|
||||||
|
find_positive_prompt_node,
|
||||||
|
trace_to_node,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Connection tracing
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
class TestConnectionTracing:
|
||||||
|
def test_direct_link(self):
|
||||||
|
wf = {
|
||||||
|
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "x"}},
|
||||||
|
"2": {"class_type": "KSampler",
|
||||||
|
"inputs": {"positive": ["1", 0], "negative": ["1", 0]}},
|
||||||
|
}
|
||||||
|
assert trace_to_node(wf, ["1", 0]) == "1"
|
||||||
|
|
||||||
|
def test_through_reroute(self):
|
||||||
|
wf = {
|
||||||
|
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "x"}},
|
||||||
|
"2": {"class_type": "Reroute", "inputs": {"input": ["1", 0]}},
|
||||||
|
"3": {"class_type": "Reroute", "inputs": {"input": ["2", 0]}},
|
||||||
|
}
|
||||||
|
assert trace_to_node(wf, ["3", 0]) == "1"
|
||||||
|
|
||||||
|
def test_circular_safe(self):
|
||||||
|
wf = {
|
||||||
|
"1": {"class_type": "Reroute", "inputs": {"input": ["2", 0]}},
|
||||||
|
"2": {"class_type": "Reroute", "inputs": {"input": ["1", 0]}},
|
||||||
|
}
|
||||||
|
# Should hit max_hops without infinite loop
|
||||||
|
result = trace_to_node(wf, ["1", 0], max_hops=5)
|
||||||
|
assert result in {"1", "2"} # any node, just don't hang
|
||||||
|
|
||||||
|
|
||||||
|
class TestPositiveNegativeDetection:
|
||||||
|
def test_basic(self, sd15_workflow):
|
||||||
|
# In sd15_workflow.json node 6 is positive, node 7 is negative
|
||||||
|
assert find_positive_prompt_node(sd15_workflow) == "6"
|
||||||
|
assert find_negative_prompt_node(sd15_workflow) == "7"
|
||||||
|
|
||||||
|
def test_swapped_order(self):
|
||||||
|
wf = {
|
||||||
|
"3": {"class_type": "KSampler",
|
||||||
|
"inputs": {
|
||||||
|
"positive": ["7", 0], "negative": ["6", 0],
|
||||||
|
"model": ["4", 0], "latent_image": ["5", 0],
|
||||||
|
"seed": 1, "steps": 20, "cfg": 7.5,
|
||||||
|
"sampler_name": "euler", "scheduler": "normal", "denoise": 1.0,
|
||||||
|
}},
|
||||||
|
"4": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "x"}},
|
||||||
|
"5": {"class_type": "EmptyLatentImage", "inputs": {"width": 512, "height": 512, "batch_size": 1}},
|
||||||
|
"6": {"class_type": "CLIPTextEncode", "inputs": {"text": "ugly", "clip": ["4", 1]}},
|
||||||
|
"7": {"class_type": "CLIPTextEncode", "inputs": {"text": "beautiful", "clip": ["4", 1]}},
|
||||||
|
}
|
||||||
|
# Now 7 is the positive (despite higher node ID)
|
||||||
|
assert find_positive_prompt_node(wf) == "7"
|
||||||
|
assert find_negative_prompt_node(wf) == "6"
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Schema extraction
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
class TestExtractSchema:
|
||||||
|
def test_basic_sd15(self, sd15_workflow):
|
||||||
|
schema = extract_schema(sd15_workflow)
|
||||||
|
params = schema["parameters"]
|
||||||
|
assert "prompt" in params
|
||||||
|
assert "negative_prompt" in params
|
||||||
|
assert "seed" in params
|
||||||
|
assert "steps" in params
|
||||||
|
assert "cfg" in params
|
||||||
|
assert "width" in params
|
||||||
|
assert "height" in params
|
||||||
|
|
||||||
|
def test_prompt_value_correct(self, sd15_workflow):
|
||||||
|
schema = extract_schema(sd15_workflow)
|
||||||
|
# The positive prompt in the example is the landscape one
|
||||||
|
assert "landscape" in schema["parameters"]["prompt"]["value"]
|
||||||
|
assert "ugly" in schema["parameters"]["negative_prompt"]["value"]
|
||||||
|
|
||||||
|
def test_model_dependencies(self, sd15_workflow):
|
||||||
|
schema = extract_schema(sd15_workflow)
|
||||||
|
deps = schema["model_dependencies"]
|
||||||
|
ckpts = [d["value"] for d in deps if d["folder"] == "checkpoints"]
|
||||||
|
assert "v1-5-pruned-emaonly.safetensors" in ckpts
|
||||||
|
|
||||||
|
def test_output_nodes(self, sd15_workflow):
|
||||||
|
schema = extract_schema(sd15_workflow)
|
||||||
|
assert "9" in schema["output_nodes"]
|
||||||
|
|
||||||
|
def test_summary(self, sd15_workflow):
|
||||||
|
schema = extract_schema(sd15_workflow)
|
||||||
|
s = schema["summary"]
|
||||||
|
assert s["has_negative_prompt"] is True
|
||||||
|
assert s["has_seed"] is True
|
||||||
|
assert s["is_video_workflow"] is False
|
||||||
|
assert s["parameter_count"] > 5
|
||||||
|
|
||||||
|
def test_flux_workflow(self, flux_workflow):
|
||||||
|
schema = extract_schema(flux_workflow)
|
||||||
|
# Flux uses RandomNoise for seed
|
||||||
|
assert schema["summary"]["has_seed"] is True
|
||||||
|
# Flux has only positive prompt (no negative encoder)
|
||||||
|
assert schema["summary"]["has_negative_prompt"] is False
|
||||||
|
|
||||||
|
def test_video_detected(self, video_workflow):
|
||||||
|
schema = extract_schema(video_workflow)
|
||||||
|
assert schema["summary"]["is_video_workflow"] is True
|
||||||
|
|
||||||
|
|
||||||
|
class TestEmbeddingDeps:
|
||||||
|
def test_extract_from_prompt(self):
|
||||||
|
wf = {
|
||||||
|
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "x"}},
|
||||||
|
"5": {"class_type": "EmptyLatentImage",
|
||||||
|
"inputs": {"width": 512, "height": 512, "batch_size": 1}},
|
||||||
|
"6": {"class_type": "CLIPTextEncode",
|
||||||
|
"inputs": {
|
||||||
|
"text": "a cat, embedding:goodvibes, embedding:art:1.2",
|
||||||
|
"clip": ["1", 1]
|
||||||
|
}},
|
||||||
|
"7": {"class_type": "CLIPTextEncode",
|
||||||
|
"inputs": {
|
||||||
|
"text": "ugly, embedding:badhands",
|
||||||
|
"clip": ["1", 1]
|
||||||
|
}},
|
||||||
|
"3": {"class_type": "KSampler",
|
||||||
|
"inputs": {
|
||||||
|
"positive": ["6", 0], "negative": ["7", 0],
|
||||||
|
"model": ["1", 0], "latent_image": ["5", 0],
|
||||||
|
"seed": 1, "steps": 20, "cfg": 7.5,
|
||||||
|
"sampler_name": "euler", "scheduler": "normal", "denoise": 1.0,
|
||||||
|
}},
|
||||||
|
"9": {"class_type": "SaveImage", "inputs": {"filename_prefix": "x", "images": ["3", 0]}},
|
||||||
|
}
|
||||||
|
schema = extract_schema(wf)
|
||||||
|
names = [d["embedding_name"] for d in schema["embedding_dependencies"]]
|
||||||
|
assert sorted(names) == ["art", "badhands", "goodvibes"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestDuplicateDeduplication:
|
||||||
|
def test_two_ksamplers_get_unique_names(self):
|
||||||
|
wf = {
|
||||||
|
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "x"}},
|
||||||
|
"5": {"class_type": "EmptyLatentImage",
|
||||||
|
"inputs": {"width": 512, "height": 512, "batch_size": 1}},
|
||||||
|
"6": {"class_type": "CLIPTextEncode", "inputs": {"text": "a", "clip": ["1", 1]}},
|
||||||
|
"7": {"class_type": "CLIPTextEncode", "inputs": {"text": "b", "clip": ["1", 1]}},
|
||||||
|
"3": {"class_type": "KSampler",
|
||||||
|
"inputs": {
|
||||||
|
"positive": ["6", 0], "negative": ["7", 0],
|
||||||
|
"model": ["1", 0], "latent_image": ["5", 0],
|
||||||
|
"seed": 42, "steps": 20, "cfg": 7.5,
|
||||||
|
"sampler_name": "euler", "scheduler": "normal", "denoise": 1.0,
|
||||||
|
}},
|
||||||
|
"4": {"class_type": "KSampler",
|
||||||
|
"inputs": {
|
||||||
|
"positive": ["6", 0], "negative": ["7", 0],
|
||||||
|
"model": ["1", 0], "latent_image": ["5", 0],
|
||||||
|
"seed": 99, "steps": 30, "cfg": 8.0,
|
||||||
|
"sampler_name": "euler", "scheduler": "normal", "denoise": 0.6,
|
||||||
|
}},
|
||||||
|
"9": {"class_type": "SaveImage", "inputs": {"filename_prefix": "x", "images": ["3", 0]}},
|
||||||
|
}
|
||||||
|
schema = extract_schema(wf)
|
||||||
|
params = schema["parameters"]
|
||||||
|
# Both seeds present with disambiguated names
|
||||||
|
seed_keys = [k for k in params if "seed" in k]
|
||||||
|
# Symmetric: both renamed (no bare "seed")
|
||||||
|
assert "seed" not in params
|
||||||
|
assert "seed_3" in params and "seed_4" in params
|
||||||
|
assert params["seed_3"]["value"] == 42
|
||||||
|
assert params["seed_4"]["value"] == 99
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
FIGURES_FOLDER := figures
|
||||||
|
PDFS := \
|
||||||
|
$(filter-out $(wildcard $(FIGURES_FOLDER)/*-crop.pdf),$(wildcard $(FIGURES_FOLDER)/*.pdf)) \
|
||||||
|
$(filter-out $(wildcard $(FIGURES_FOLDER)/**/*-crop.pdf),$(wildcard $(FIGURES_FOLDER)/**/*.pdf))
|
||||||
|
CROPPED_PDFS := $(PDFS:.pdf=-crop.pdf)
|
||||||
|
|
||||||
|
all: main.pdf
|
||||||
|
|
||||||
|
%.pdf: %.tex Makefile $(CROPPED_PDFS)
|
||||||
|
pdflatex -synctex=1 -interaction=nonstopmode $<
|
||||||
|
-bibtex $*.aux
|
||||||
|
pdflatex -synctex=1 -interaction=nonstopmode $<
|
||||||
|
pdflatex -synctex=1 -interaction=nonstopmode $<
|
||||||
|
|
||||||
|
.PHONY: figures
|
||||||
|
figures: $(CROPPED_PDFS)
|
||||||
|
|
||||||
|
.PRECIOUS: $(CROPPED_PDFS)
|
||||||
|
%-crop.pdf: %.pdf Makefile
|
||||||
|
pdfcrop $<
|
||||||
|
|
||||||
|
.PHONY: clean upgrade
|
||||||
|
clean:
|
||||||
|
find . -maxdepth 1 \
|
||||||
|
\( -name "*.aux" -o -name "*.bbl" -o -name "*.blg" -o \
|
||||||
|
-name "*.log" -o -name "*.out" -o -name "*.pdf" -o \
|
||||||
|
-name "*.synctex.gz" \) | xargs $(RM)
|
||||||
|
find $(FIGURES_FOLDER) -name "*-crop.pdf" | xargs $(RM)
|
||||||
|
|
||||||
|
YEAR := 2025
|
||||||
|
|
||||||
|
upgrade:
|
||||||
|
curl -O https://media.neurips.cc/Conferences/NeurIPS$(YEAR)/Styles.zip
|
||||||
|
unzip -u Styles.zip
|
||||||
|
mv Styles/neurips_${YEAR}.sty neurips.sty
|
||||||
|
$(RM) -r Styles.zip Styles
|
||||||
@@ -0,0 +1,185 @@
|
|||||||
|
"""Tests for extract_schema.py."""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from extract_schema import (
|
||||||
|
extract_schema,
|
||||||
|
find_negative_prompt_node,
|
||||||
|
find_positive_prompt_node,
|
||||||
|
trace_to_node,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Connection tracing
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
class TestConnectionTracing:
|
||||||
|
def test_direct_link(self):
|
||||||
|
wf = {
|
||||||
|
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "x"}},
|
||||||
|
"2": {"class_type": "KSampler",
|
||||||
|
"inputs": {"positive": ["1", 0], "negative": ["1", 0]}},
|
||||||
|
}
|
||||||
|
assert trace_to_node(wf, ["1", 0]) == "1"
|
||||||
|
|
||||||
|
def test_through_reroute(self):
|
||||||
|
wf = {
|
||||||
|
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "x"}},
|
||||||
|
"2": {"class_type": "Reroute", "inputs": {"input": ["1", 0]}},
|
||||||
|
"3": {"class_type": "Reroute", "inputs": {"input": ["2", 0]}},
|
||||||
|
}
|
||||||
|
assert trace_to_node(wf, ["3", 0]) == "1"
|
||||||
|
|
||||||
|
def test_circular_safe(self):
|
||||||
|
wf = {
|
||||||
|
"1": {"class_type": "Reroute", "inputs": {"input": ["2", 0]}},
|
||||||
|
"2": {"class_type": "Reroute", "inputs": {"input": ["1", 0]}},
|
||||||
|
}
|
||||||
|
# Should hit max_hops without infinite loop
|
||||||
|
result = trace_to_node(wf, ["1", 0], max_hops=5)
|
||||||
|
assert result in {"1", "2"} # any node, just don't hang
|
||||||
|
|
||||||
|
|
||||||
|
class TestPositiveNegativeDetection:
|
||||||
|
def test_basic(self, sd15_workflow):
|
||||||
|
# In sd15_workflow.json node 6 is positive, node 7 is negative
|
||||||
|
assert find_positive_prompt_node(sd15_workflow) == "6"
|
||||||
|
assert find_negative_prompt_node(sd15_workflow) == "7"
|
||||||
|
|
||||||
|
def test_swapped_order(self):
|
||||||
|
wf = {
|
||||||
|
"3": {"class_type": "KSampler",
|
||||||
|
"inputs": {
|
||||||
|
"positive": ["7", 0], "negative": ["6", 0],
|
||||||
|
"model": ["4", 0], "latent_image": ["5", 0],
|
||||||
|
"seed": 1, "steps": 20, "cfg": 7.5,
|
||||||
|
"sampler_name": "euler", "scheduler": "normal", "denoise": 1.0,
|
||||||
|
}},
|
||||||
|
"4": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "x"}},
|
||||||
|
"5": {"class_type": "EmptyLatentImage", "inputs": {"width": 512, "height": 512, "batch_size": 1}},
|
||||||
|
"6": {"class_type": "CLIPTextEncode", "inputs": {"text": "ugly", "clip": ["4", 1]}},
|
||||||
|
"7": {"class_type": "CLIPTextEncode", "inputs": {"text": "beautiful", "clip": ["4", 1]}},
|
||||||
|
}
|
||||||
|
# Now 7 is the positive (despite higher node ID)
|
||||||
|
assert find_positive_prompt_node(wf) == "7"
|
||||||
|
assert find_negative_prompt_node(wf) == "6"
|
||||||
|
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Schema extraction
|
||||||
|
# =============================================================================
|
||||||
|
|
||||||
|
class TestExtractSchema:
|
||||||
|
def test_basic_sd15(self, sd15_workflow):
|
||||||
|
schema = extract_schema(sd15_workflow)
|
||||||
|
params = schema["parameters"]
|
||||||
|
assert "prompt" in params
|
||||||
|
assert "negative_prompt" in params
|
||||||
|
assert "seed" in params
|
||||||
|
assert "steps" in params
|
||||||
|
assert "cfg" in params
|
||||||
|
assert "width" in params
|
||||||
|
assert "height" in params
|
||||||
|
|
||||||
|
def test_prompt_value_correct(self, sd15_workflow):
|
||||||
|
schema = extract_schema(sd15_workflow)
|
||||||
|
# The positive prompt in the example is the landscape one
|
||||||
|
assert "landscape" in schema["parameters"]["prompt"]["value"]
|
||||||
|
assert "ugly" in schema["parameters"]["negative_prompt"]["value"]
|
||||||
|
|
||||||
|
def test_model_dependencies(self, sd15_workflow):
|
||||||
|
schema = extract_schema(sd15_workflow)
|
||||||
|
deps = schema["model_dependencies"]
|
||||||
|
ckpts = [d["value"] for d in deps if d["folder"] == "checkpoints"]
|
||||||
|
assert "v1-5-pruned-emaonly.safetensors" in ckpts
|
||||||
|
|
||||||
|
def test_output_nodes(self, sd15_workflow):
|
||||||
|
schema = extract_schema(sd15_workflow)
|
||||||
|
assert "9" in schema["output_nodes"]
|
||||||
|
|
||||||
|
def test_summary(self, sd15_workflow):
|
||||||
|
schema = extract_schema(sd15_workflow)
|
||||||
|
s = schema["summary"]
|
||||||
|
assert s["has_negative_prompt"] is True
|
||||||
|
assert s["has_seed"] is True
|
||||||
|
assert s["is_video_workflow"] is False
|
||||||
|
assert s["parameter_count"] > 5
|
||||||
|
|
||||||
|
def test_flux_workflow(self, flux_workflow):
|
||||||
|
schema = extract_schema(flux_workflow)
|
||||||
|
# Flux uses RandomNoise for seed
|
||||||
|
assert schema["summary"]["has_seed"] is True
|
||||||
|
# Flux has only positive prompt (no negative encoder)
|
||||||
|
assert schema["summary"]["has_negative_prompt"] is False
|
||||||
|
|
||||||
|
def test_video_detected(self, video_workflow):
|
||||||
|
schema = extract_schema(video_workflow)
|
||||||
|
assert schema["summary"]["is_video_workflow"] is True
|
||||||
|
|
||||||
|
|
||||||
|
class TestEmbeddingDeps:
|
||||||
|
def test_extract_from_prompt(self):
|
||||||
|
wf = {
|
||||||
|
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "x"}},
|
||||||
|
"5": {"class_type": "EmptyLatentImage",
|
||||||
|
"inputs": {"width": 512, "height": 512, "batch_size": 1}},
|
||||||
|
"6": {"class_type": "CLIPTextEncode",
|
||||||
|
"inputs": {
|
||||||
|
"text": "a cat, embedding:goodvibes, embedding:art:1.2",
|
||||||
|
"clip": ["1", 1]
|
||||||
|
}},
|
||||||
|
"7": {"class_type": "CLIPTextEncode",
|
||||||
|
"inputs": {
|
||||||
|
"text": "ugly, embedding:badhands",
|
||||||
|
"clip": ["1", 1]
|
||||||
|
}},
|
||||||
|
"3": {"class_type": "KSampler",
|
||||||
|
"inputs": {
|
||||||
|
"positive": ["6", 0], "negative": ["7", 0],
|
||||||
|
"model": ["1", 0], "latent_image": ["5", 0],
|
||||||
|
"seed": 1, "steps": 20, "cfg": 7.5,
|
||||||
|
"sampler_name": "euler", "scheduler": "normal", "denoise": 1.0,
|
||||||
|
}},
|
||||||
|
"9": {"class_type": "SaveImage", "inputs": {"filename_prefix": "x", "images": ["3", 0]}},
|
||||||
|
}
|
||||||
|
schema = extract_schema(wf)
|
||||||
|
names = [d["embedding_name"] for d in schema["embedding_dependencies"]]
|
||||||
|
assert sorted(names) == ["art", "badhands", "goodvibes"]
|
||||||
|
|
||||||
|
|
||||||
|
class TestDuplicateDeduplication:
|
||||||
|
def test_two_ksamplers_get_unique_names(self):
|
||||||
|
wf = {
|
||||||
|
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "x"}},
|
||||||
|
"5": {"class_type": "EmptyLatentImage",
|
||||||
|
"inputs": {"width": 512, "height": 512, "batch_size": 1}},
|
||||||
|
"6": {"class_type": "CLIPTextEncode", "inputs": {"text": "a", "clip": ["1", 1]}},
|
||||||
|
"7": {"class_type": "CLIPTextEncode", "inputs": {"text": "b", "clip": ["1", 1]}},
|
||||||
|
"3": {"class_type": "KSampler",
|
||||||
|
"inputs": {
|
||||||
|
"positive": ["6", 0], "negative": ["7", 0],
|
||||||
|
"model": ["1", 0], "latent_image": ["5", 0],
|
||||||
|
"seed": 42, "steps": 20, "cfg": 7.5,
|
||||||
|
"sampler_name": "euler", "scheduler": "normal", "denoise": 1.0,
|
||||||
|
}},
|
||||||
|
"4": {"class_type": "KSampler",
|
||||||
|
"inputs": {
|
||||||
|
"positive": ["6", 0], "negative": ["7", 0],
|
||||||
|
"model": ["1", 0], "latent_image": ["5", 0],
|
||||||
|
"seed": 99, "steps": 30, "cfg": 8.0,
|
||||||
|
"sampler_name": "euler", "scheduler": "normal", "denoise": 0.6,
|
||||||
|
}},
|
||||||
|
"9": {"class_type": "SaveImage", "inputs": {"filename_prefix": "x", "images": ["3", 0]}},
|
||||||
|
}
|
||||||
|
schema = extract_schema(wf)
|
||||||
|
params = schema["parameters"]
|
||||||
|
# Both seeds present with disambiguated names
|
||||||
|
seed_keys = [k for k in params if "seed" in k]
|
||||||
|
# Symmetric: both renamed (no bare "seed")
|
||||||
|
assert "seed" not in params
|
||||||
|
assert "seed_3" in params and "seed_4" in params
|
||||||
|
assert params["seed_3"]["value"] == 42
|
||||||
|
assert params["seed_4"]["value"] == 99
|
||||||
@@ -0,0 +1,167 @@
|
|||||||
|
"""Named RGB palettes for pixel_art() and pixel_art_video().
|
||||||
|
|
||||||
|
Palette RGB values sourced from pixel-art-studio (MIT License)
|
||||||
|
https://github.com/Synero/pixel-art-studio — see ATTRIBUTION.md.
|
||||||
|
"""
|
||||||
|
|
||||||
|
PALETTES = {
|
||||||
|
# ── Hardware palettes ───────────────────────────────────────────────
|
||||||
|
"NES": [
|
||||||
|
(0, 0, 0), (124, 124, 124), (0, 0, 252), (0, 0, 188), (68, 40, 188),
|
||||||
|
(148, 0, 132), (168, 0, 32), (168, 16, 0), (136, 20, 0), (0, 116, 0),
|
||||||
|
(0, 148, 0), (0, 120, 0), (0, 88, 0), (0, 64, 88), (188, 188, 188),
|
||||||
|
(0, 120, 248), (0, 88, 248), (104, 68, 252), (216, 0, 204), (228, 0, 88),
|
||||||
|
(248, 56, 0), (228, 92, 16), (172, 124, 0), (0, 184, 0), (0, 168, 0),
|
||||||
|
(0, 168, 68), (0, 136, 136), (248, 248, 248), (60, 188, 252),
|
||||||
|
(104, 136, 252), (152, 120, 248), (248, 120, 248), (248, 88, 152),
|
||||||
|
(248, 120, 88), (252, 160, 68), (248, 184, 0), (184, 248, 24),
|
||||||
|
(88, 216, 84), (88, 248, 152), (0, 232, 216), (120, 120, 120),
|
||||||
|
(252, 252, 252), (164, 228, 252), (184, 184, 248), (216, 184, 248),
|
||||||
|
(248, 184, 248), (248, 164, 192), (240, 208, 176), (252, 224, 168),
|
||||||
|
(248, 216, 120), (216, 248, 120), (184, 248, 184), (184, 248, 216),
|
||||||
|
(0, 252, 252), (216, 216, 216),
|
||||||
|
],
|
||||||
|
"C64": [
|
||||||
|
(0, 0, 0), (255, 255, 255), (161, 77, 67), (106, 191, 199),
|
||||||
|
(161, 87, 164), (92, 172, 95), (64, 64, 223), (191, 206, 137),
|
||||||
|
(161, 104, 60), (108, 80, 21), (203, 126, 117), (98, 98, 98),
|
||||||
|
(137, 137, 137), (154, 226, 155), (124, 124, 255), (173, 173, 173),
|
||||||
|
],
|
||||||
|
"COMMODORE_64": [
|
||||||
|
(0, 0, 0), (255, 255, 255), (161, 77, 67), (106, 192, 200),
|
||||||
|
(161, 87, 165), (92, 172, 95), (64, 68, 227), (203, 214, 137),
|
||||||
|
(163, 104, 58), (110, 84, 11), (204, 127, 118), (99, 99, 99),
|
||||||
|
(139, 139, 139), (154, 227, 157), (139, 127, 205), (175, 175, 175),
|
||||||
|
],
|
||||||
|
"ZX_SPECTRUM": [
|
||||||
|
(0, 0, 0), (0, 39, 251), (252, 48, 22), (255, 63, 252),
|
||||||
|
(0, 249, 44), (0, 252, 254), (255, 253, 51), (255, 255, 255),
|
||||||
|
],
|
||||||
|
"APPLE_II_LO": [
|
||||||
|
(0, 0, 0), (133, 59, 81), (80, 71, 137), (234, 93, 240),
|
||||||
|
(0, 104, 82), (146, 146, 146), (0, 168, 241), (202, 195, 248),
|
||||||
|
(81, 92, 15), (235, 127, 35), (146, 146, 146), (246, 185, 202),
|
||||||
|
(0, 202, 41), (203, 211, 155), (155, 220, 203), (255, 255, 255),
|
||||||
|
],
|
||||||
|
"APPLE_II_HI": [
|
||||||
|
(0, 0, 0), (255, 0, 255), (0, 255, 0), (255, 255, 255),
|
||||||
|
(0, 175, 255), (255, 80, 0),
|
||||||
|
],
|
||||||
|
"GAMEBOY_ORIGINAL": [
|
||||||
|
(0, 63, 0), (46, 115, 32), (140, 191, 10), (160, 207, 10),
|
||||||
|
],
|
||||||
|
"GAMEBOY_POCKET": [
|
||||||
|
(0, 0, 0), (85, 85, 85), (170, 170, 170), (255, 255, 255),
|
||||||
|
],
|
||||||
|
"GAMEBOY_VIRTUALBOY": [
|
||||||
|
(239, 0, 0), (164, 0, 0), (85, 0, 0), (0, 0, 0),
|
||||||
|
],
|
||||||
|
"PICO_8": [
|
||||||
|
(0, 0, 0), (29, 43, 83), (126, 37, 83), (0, 135, 81), (171, 82, 54),
|
||||||
|
(95, 87, 79), (194, 195, 199), (255, 241, 232), (255, 0, 77),
|
||||||
|
(255, 163, 0), (255, 236, 39), (0, 228, 54), (41, 173, 255),
|
||||||
|
(131, 118, 156), (255, 119, 168), (255, 204, 170),
|
||||||
|
],
|
||||||
|
"TELETEXT": [
|
||||||
|
(0, 0, 0), (255, 0, 0), (0, 128, 0), (255, 255, 0),
|
||||||
|
(0, 0, 255), (255, 0, 255), (0, 255, 255), (255, 255, 255),
|
||||||
|
],
|
||||||
|
"CGA_MODE4_PAL1": [
|
||||||
|
(0, 0, 0), (255, 255, 255), (0, 255, 255), (255, 0, 255),
|
||||||
|
],
|
||||||
|
"MSX": [
|
||||||
|
(0, 0, 0), (62, 184, 73), (116, 208, 125), (89, 85, 224),
|
||||||
|
(128, 118, 241), (185, 94, 81), (101, 219, 239), (219, 101, 89),
|
||||||
|
(255, 137, 125), (204, 195, 94), (222, 208, 135), (58, 162, 65),
|
||||||
|
(183, 102, 181), (204, 204, 204), (255, 255, 255),
|
||||||
|
],
|
||||||
|
"MICROSOFT_WINDOWS_16": [
|
||||||
|
(0, 0, 0), (128, 0, 0), (0, 128, 0), (128, 128, 0), (0, 0, 128),
|
||||||
|
(128, 0, 128), (0, 128, 128), (192, 192, 192), (128, 128, 128),
|
||||||
|
(255, 0, 0), (0, 255, 0), (255, 255, 0), (0, 0, 255),
|
||||||
|
(255, 0, 255), (0, 255, 255), (255, 255, 255),
|
||||||
|
],
|
||||||
|
"MICROSOFT_WINDOWS_PAINT": [
|
||||||
|
(0, 0, 0), (255, 255, 255), (123, 123, 123), (189, 189, 189),
|
||||||
|
(123, 12, 2), (255, 37, 0), (123, 123, 2), (255, 251, 2),
|
||||||
|
(0, 123, 2), (2, 249, 2), (0, 123, 122), (2, 253, 254),
|
||||||
|
(2, 19, 122), (5, 50, 255), (123, 25, 122), (255, 64, 254),
|
||||||
|
(122, 57, 2), (255, 122, 57), (123, 123, 56), (255, 252, 122),
|
||||||
|
(2, 57, 57), (5, 250, 123), (0, 123, 255), (255, 44, 123),
|
||||||
|
],
|
||||||
|
"MONO_BW": [(0, 0, 0), (255, 255, 255)],
|
||||||
|
"MONO_AMBER": [(40, 40, 40), (255, 176, 0)],
|
||||||
|
"MONO_GREEN": [(40, 40, 40), (51, 255, 51)],
|
||||||
|
|
||||||
|
# ── Artistic palettes ───────────────────────────────────────────────
|
||||||
|
"PASTEL_DREAM": [
|
||||||
|
(255, 218, 233), (255, 229, 204), (255, 255, 204), (204, 255, 229),
|
||||||
|
(204, 229, 255), (229, 204, 255), (255, 204, 229), (204, 255, 255),
|
||||||
|
(255, 245, 220), (230, 230, 250),
|
||||||
|
],
|
||||||
|
"NEON_CYBER": [
|
||||||
|
(0, 0, 0), (255, 0, 128), (0, 255, 255), (255, 0, 255),
|
||||||
|
(0, 255, 128), (255, 255, 0), (128, 0, 255), (255, 128, 0),
|
||||||
|
(0, 128, 255), (255, 255, 255),
|
||||||
|
],
|
||||||
|
"RETRO_WARM": [
|
||||||
|
(62, 39, 35), (139, 69, 19), (210, 105, 30), (244, 164, 96),
|
||||||
|
(255, 218, 185), (255, 245, 238), (178, 34, 34), (205, 92, 92),
|
||||||
|
(255, 99, 71), (255, 160, 122),
|
||||||
|
],
|
||||||
|
"OCEAN_DEEP": [
|
||||||
|
(0, 25, 51), (0, 51, 102), (0, 76, 153), (0, 102, 178),
|
||||||
|
(0, 128, 204), (51, 153, 204), (102, 178, 204), (153, 204, 229),
|
||||||
|
(204, 229, 255), (229, 245, 255),
|
||||||
|
],
|
||||||
|
"FOREST_MOSS": [
|
||||||
|
(34, 51, 34), (51, 76, 51), (68, 102, 51), (85, 128, 68),
|
||||||
|
(102, 153, 85), (136, 170, 102), (170, 196, 136), (204, 221, 170),
|
||||||
|
(238, 238, 204), (245, 245, 220),
|
||||||
|
],
|
||||||
|
"SUNSET_FIRE": [
|
||||||
|
(51, 0, 0), (102, 0, 0), (153, 0, 0), (204, 0, 0), (255, 0, 0),
|
||||||
|
(255, 51, 0), (255, 102, 0), (255, 153, 0), (255, 204, 0),
|
||||||
|
(255, 255, 51),
|
||||||
|
],
|
||||||
|
"ARCTIC_ICE": [
|
||||||
|
(0, 0, 51), (0, 0, 102), (0, 51, 153), (0, 102, 153),
|
||||||
|
(51, 153, 204), (102, 204, 255), (153, 229, 255), (204, 242, 255),
|
||||||
|
(229, 247, 255), (255, 255, 255),
|
||||||
|
],
|
||||||
|
"VINTAGE_ROSE": [
|
||||||
|
(103, 58, 63), (137, 72, 81), (170, 91, 102), (196, 113, 122),
|
||||||
|
(219, 139, 147), (232, 168, 175), (240, 196, 199), (245, 215, 217),
|
||||||
|
(249, 232, 233), (255, 245, 245),
|
||||||
|
],
|
||||||
|
"EARTH_CLAY": [
|
||||||
|
(62, 39, 35), (89, 56, 47), (116, 73, 59), (143, 90, 71),
|
||||||
|
(170, 107, 83), (197, 124, 95), (210, 155, 126), (222, 186, 160),
|
||||||
|
(235, 217, 196), (248, 248, 232),
|
||||||
|
],
|
||||||
|
"ELECTRIC_VIOLET": [
|
||||||
|
(26, 0, 51), (51, 0, 102), (76, 0, 153), (102, 0, 204),
|
||||||
|
(128, 0, 255), (153, 51, 255), (178, 102, 255), (204, 153, 255),
|
||||||
|
(229, 204, 255), (245, 229, 255),
|
||||||
|
],
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def build_palette_image(palette_name):
|
||||||
|
"""Build a 1x1 PIL 'P'-mode image with the named palette for Image.quantize(palette=...)."""
|
||||||
|
from PIL import Image
|
||||||
|
|
||||||
|
if palette_name not in PALETTES:
|
||||||
|
raise ValueError(
|
||||||
|
f"Unknown palette {palette_name!r}. "
|
||||||
|
f"Choose from: {sorted(PALETTES)}"
|
||||||
|
)
|
||||||
|
flat = []
|
||||||
|
for (r, g, b) in PALETTES[palette_name]:
|
||||||
|
flat.extend([r, g, b])
|
||||||
|
# Pad to 768 bytes (256 colors) as PIL requires
|
||||||
|
while len(flat) < 768:
|
||||||
|
flat.append(0)
|
||||||
|
pal_img = Image.new("P", (1, 1))
|
||||||
|
pal_img.putpalette(flat)
|
||||||
|
return pal_img
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
---
|
||||||
|
description: Skills for academic research, paper discovery, literature review, domain reconnaissance, market data, content monitoring, and scientific knowledge retrieval.
|
||||||
|
---
|
||||||
@@ -0,0 +1,345 @@
|
|||||||
|
"""Pixel art video — overlay procedural animations onto a source image.
|
||||||
|
|
||||||
|
Takes any image (typically pre-processed with pixel_art()) and overlays
|
||||||
|
animated pixel effects (stars, rain, fireflies, etc.), then encodes to MP4
|
||||||
|
(and optionally GIF) via ffmpeg.
|
||||||
|
|
||||||
|
Scene animations ported from pixel-art-studio (MIT) — see ATTRIBUTION.md.
|
||||||
|
The generative/Pollinations code is intentionally dropped — Hermes uses
|
||||||
|
`image_generate` + `pixel_art()` for base frames instead.
|
||||||
|
|
||||||
|
Usage (import):
|
||||||
|
from pixel_art_video import pixel_art_video
|
||||||
|
pixel_art_video("frame.png", "out.mp4", scene="night", duration=6)
|
||||||
|
|
||||||
|
Usage (CLI):
|
||||||
|
python pixel_art_video.py frame.png out.mp4 --scene night --duration 6 --gif
|
||||||
|
"""
|
||||||
|
|
||||||
|
import math
|
||||||
|
import os
|
||||||
|
import random
|
||||||
|
import shutil
|
||||||
|
import subprocess
|
||||||
|
import tempfile
|
||||||
|
|
||||||
|
from PIL import Image, ImageDraw
|
||||||
|
|
||||||
|
|
||||||
|
# ── Pixel drawing helpers ──────────────────────────────────────────────
|
||||||
|
|
||||||
|
def _px(draw, x, y, color, size=2):
|
||||||
|
x, y = int(x), int(y)
|
||||||
|
W, H = draw.im.size
|
||||||
|
if 0 <= x < W and 0 <= y < H:
|
||||||
|
draw.rectangle([x, y, x + size - 1, y + size - 1], fill=color)
|
||||||
|
|
||||||
|
|
||||||
|
def _pixel_cross(draw, x, y, color, arm=2):
|
||||||
|
x, y = int(x), int(y)
|
||||||
|
for i in range(-arm, arm + 1):
|
||||||
|
_px(draw, x + i, y, color, 1)
|
||||||
|
_px(draw, x, y + i, color, 1)
|
||||||
|
|
||||||
|
|
||||||
|
# ── Animation init/draw pairs ──────────────────────────────────────────
|
||||||
|
|
||||||
|
def init_stars(rng, W, H):
|
||||||
|
return [(rng.randint(0, W), rng.randint(0, H // 2)) for _ in range(15)]
|
||||||
|
|
||||||
|
def draw_stars(draw, stars, t, W, H):
|
||||||
|
for i, (sx, sy) in enumerate(stars):
|
||||||
|
if math.sin(t * 2.0 + i * 0.7) > 0.65:
|
||||||
|
_pixel_cross(draw, sx, sy, (255, 255, 220), arm=2)
|
||||||
|
|
||||||
|
|
||||||
|
def init_fireflies(rng, W, H):
|
||||||
|
return [{"x": rng.randint(20, W - 20), "y": rng.randint(H // 4, H - 20),
|
||||||
|
"phase": rng.uniform(0, 6.28), "speed": rng.uniform(0.3, 0.8)}
|
||||||
|
for _ in range(10)]
|
||||||
|
|
||||||
|
def draw_fireflies(draw, ff, t, W, H):
|
||||||
|
for f in ff:
|
||||||
|
if math.sin(t * 1.5 + f["phase"]) < 0.15:
|
||||||
|
continue
|
||||||
|
_px(draw,
|
||||||
|
f["x"] + math.sin(t * f["speed"] + f["phase"]) * 3,
|
||||||
|
f["y"] + math.cos(t * f["speed"] * 0.7) * 2,
|
||||||
|
(200, 255, 100), 2)
|
||||||
|
|
||||||
|
|
||||||
|
def init_leaves(rng, W, H):
|
||||||
|
return [{"x": rng.randint(0, W), "y": rng.randint(-H, 0),
|
||||||
|
"speed": rng.uniform(0.5, 1.5), "wobble": rng.uniform(0.02, 0.05),
|
||||||
|
"phase": rng.uniform(0, 6.28),
|
||||||
|
"color": rng.choice([(180, 120, 50), (160, 100, 40), (200, 140, 60)])}
|
||||||
|
for _ in range(12)]
|
||||||
|
|
||||||
|
def draw_leaves(draw, leaves, t, W, H):
|
||||||
|
for leaf in leaves:
|
||||||
|
_px(draw,
|
||||||
|
leaf["x"] + math.sin(t * leaf["wobble"] + leaf["phase"]) * 15,
|
||||||
|
(leaf["y"] + t * leaf["speed"] * 20) % (H + 40) - 20,
|
||||||
|
leaf["color"], 2)
|
||||||
|
|
||||||
|
|
||||||
|
def init_dust_motes(rng, W, H):
|
||||||
|
return [{"x": rng.randint(30, W - 30), "y": rng.randint(30, H - 30),
|
||||||
|
"phase": rng.uniform(0, 6.28), "speed": rng.uniform(0.2, 0.5),
|
||||||
|
"amp": rng.uniform(2, 6)} for _ in range(20)]
|
||||||
|
|
||||||
|
def draw_dust_motes(draw, motes, t, W, H):
|
||||||
|
for m in motes:
|
||||||
|
if math.sin(t * 2.0 + m["phase"]) > 0.3:
|
||||||
|
_px(draw,
|
||||||
|
m["x"] + math.sin(t * 0.3 + m["phase"]) * m["amp"],
|
||||||
|
m["y"] - (m["speed"] * t * 15) % H,
|
||||||
|
(255, 210, 100), 1)
|
||||||
|
|
||||||
|
|
||||||
|
def init_sparkles(rng, W, H):
|
||||||
|
return [(rng.randint(W // 4, 3 * W // 4), rng.randint(H // 4, 3 * H // 4),
|
||||||
|
rng.uniform(0, 6.28),
|
||||||
|
rng.choice([(180, 200, 255), (255, 220, 150), (200, 180, 255)]))
|
||||||
|
for _ in range(10)]
|
||||||
|
|
||||||
|
def draw_sparkles(draw, sparkles, t, W, H):
|
||||||
|
for sx, sy, phase, color in sparkles:
|
||||||
|
if math.sin(t * 1.8 + phase) > 0.6:
|
||||||
|
_pixel_cross(draw, sx, sy, color, arm=2)
|
||||||
|
|
||||||
|
|
||||||
|
def init_rain(rng, W, H):
|
||||||
|
return [{"x": rng.randint(0, W), "y": rng.randint(0, H),
|
||||||
|
"speed": rng.uniform(4, 8)} for _ in range(30)]
|
||||||
|
|
||||||
|
def draw_rain(draw, rain, t, W, H):
|
||||||
|
for r in rain:
|
||||||
|
y = (r["y"] + t * r["speed"] * 20) % H
|
||||||
|
_px(draw, r["x"], y, (120, 150, 200), 1)
|
||||||
|
_px(draw, r["x"], y + 4, (100, 130, 180), 1)
|
||||||
|
|
||||||
|
|
||||||
|
def init_lightning(rng, W, H):
|
||||||
|
return {"timer": 0, "flash": False, "rng": rng}
|
||||||
|
|
||||||
|
def draw_lightning(draw, state, t, W, H):
|
||||||
|
state["timer"] += 1
|
||||||
|
if state["timer"] > 45 and state["rng"].random() < 0.04:
|
||||||
|
state["flash"] = True
|
||||||
|
state["timer"] = 0
|
||||||
|
if state["flash"]:
|
||||||
|
for x in range(0, W, 4):
|
||||||
|
for y in range(0, H // 3, 3):
|
||||||
|
if state["rng"].random() < 0.12:
|
||||||
|
_px(draw, x, y, (255, 255, 240), 2)
|
||||||
|
state["flash"] = False
|
||||||
|
|
||||||
|
|
||||||
|
def init_bubbles(rng, W, H):
|
||||||
|
return [{"x": rng.randint(20, W - 20), "y": rng.randint(H, H * 2),
|
||||||
|
"speed": rng.uniform(0.3, 0.8), "size": rng.choice([1, 2, 2])}
|
||||||
|
for _ in range(15)]
|
||||||
|
|
||||||
|
def draw_bubbles(draw, bubbles, t, W, H):
|
||||||
|
for b in bubbles:
|
||||||
|
x = b["x"] + math.sin(t * 0.5 + b["x"]) * 3
|
||||||
|
y = b["y"] - (t * b["speed"] * 20) % (H + 40)
|
||||||
|
if 0 < y < H:
|
||||||
|
_px(draw, x, y, (150, 200, 255), b["size"])
|
||||||
|
|
||||||
|
|
||||||
|
def init_embers(rng, W, H):
|
||||||
|
return [{"x": rng.randint(0, W), "y": rng.randint(0, H),
|
||||||
|
"speed": rng.uniform(0.3, 0.9), "phase": rng.uniform(0, 6.28),
|
||||||
|
"color": rng.choice([(255, 150, 30), (255, 100, 20), (255, 200, 50)])}
|
||||||
|
for _ in range(18)]
|
||||||
|
|
||||||
|
def draw_embers(draw, embers, t, W, H):
|
||||||
|
for e in embers:
|
||||||
|
x = e["x"] + math.sin(t * 0.4 + e["phase"]) * 5
|
||||||
|
y = e["y"] - (t * e["speed"] * 15) % H
|
||||||
|
if math.sin(t * 2.5 + e["phase"]) > 0.2:
|
||||||
|
_px(draw, x, y, e["color"], 2)
|
||||||
|
|
||||||
|
|
||||||
|
def init_snowflakes(rng, W, H):
|
||||||
|
return [{"x": rng.randint(0, W), "y": rng.randint(-H, 0),
|
||||||
|
"speed": rng.uniform(0.3, 0.6), "wobble": rng.uniform(0.04, 0.09),
|
||||||
|
"size": rng.choice([2, 2, 3])}
|
||||||
|
for _ in range(40)]
|
||||||
|
|
||||||
|
def draw_snowflakes(draw, flakes, t, W, H):
|
||||||
|
for f in flakes:
|
||||||
|
x = f["x"] + math.sin(t * f["wobble"] + f["x"]) * 20
|
||||||
|
y = (f["y"] + t * f["speed"] * 8) % (H + 20) - 10
|
||||||
|
if f["size"] >= 3:
|
||||||
|
_pixel_cross(draw, x, y, (230, 235, 255), arm=1)
|
||||||
|
else:
|
||||||
|
_px(draw, x, y, (230, 235, 255), 2)
|
||||||
|
|
||||||
|
|
||||||
|
def init_neon_pulse(rng, W, H):
|
||||||
|
return [(rng.randint(0, W), rng.randint(0, H), rng.uniform(0, 6.28),
|
||||||
|
rng.choice([(255, 0, 200), (0, 255, 255), (255, 50, 150)]))
|
||||||
|
for _ in range(8)]
|
||||||
|
|
||||||
|
def draw_neon_pulse(draw, points, t, W, H):
|
||||||
|
for x, y, phase, color in points:
|
||||||
|
if math.sin(t * 2.5 + phase) > 0.5:
|
||||||
|
_pixel_cross(draw, x, y, color, arm=3)
|
||||||
|
|
||||||
|
|
||||||
|
def init_heat_shimmer(rng, W, H):
|
||||||
|
return [{"x": rng.randint(0, W), "y": rng.randint(H // 2, H),
|
||||||
|
"phase": rng.uniform(0, 6.28)} for _ in range(12)]
|
||||||
|
|
||||||
|
def draw_heat_shimmer(draw, points, t, W, H):
|
||||||
|
for p in points:
|
||||||
|
x = p["x"] + math.sin(t * 0.8 + p["phase"]) * 2
|
||||||
|
y = p["y"] + math.sin(t * 1.2 + p["phase"]) * 1
|
||||||
|
if abs(math.sin(t * 1.5 + p["phase"])) > 0.6:
|
||||||
|
_px(draw, x, y, (255, 200, 100), 1)
|
||||||
|
|
||||||
|
|
||||||
|
# ── Scene → animation mapping ──────────────────────────────────────────
|
||||||
|
|
||||||
|
SCENES = {
|
||||||
|
"night": ["stars", "fireflies", "leaves"],
|
||||||
|
"dusk": ["fireflies", "sparkles"],
|
||||||
|
"tavern": ["dust_motes", "sparkles"],
|
||||||
|
"indoor": ["dust_motes"],
|
||||||
|
"urban": ["rain", "neon_pulse"],
|
||||||
|
"nature": ["leaves", "fireflies"],
|
||||||
|
"magic": ["sparkles", "fireflies"],
|
||||||
|
"storm": ["rain", "lightning"],
|
||||||
|
"underwater": ["bubbles", "sparkles"],
|
||||||
|
"fire": ["embers", "sparkles"],
|
||||||
|
"snow": ["snowflakes", "sparkles"],
|
||||||
|
"desert": ["heat_shimmer", "dust_motes"],
|
||||||
|
}
|
||||||
|
|
||||||
|
# Map scene layer name to (init_fn, draw_fn).
|
||||||
|
_LAYERS = {
|
||||||
|
"stars": (init_stars, draw_stars),
|
||||||
|
"fireflies": (init_fireflies, draw_fireflies),
|
||||||
|
"leaves": (init_leaves, draw_leaves),
|
||||||
|
"dust_motes": (init_dust_motes, draw_dust_motes),
|
||||||
|
"sparkles": (init_sparkles, draw_sparkles),
|
||||||
|
"rain": (init_rain, draw_rain),
|
||||||
|
"lightning": (init_lightning, draw_lightning),
|
||||||
|
"bubbles": (init_bubbles, draw_bubbles),
|
||||||
|
"embers": (init_embers, draw_embers),
|
||||||
|
"snowflakes": (init_snowflakes, draw_snowflakes),
|
||||||
|
"neon_pulse": (init_neon_pulse, draw_neon_pulse),
|
||||||
|
"heat_shimmer": (init_heat_shimmer, draw_heat_shimmer),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _ensure_ffmpeg():
|
||||||
|
if shutil.which("ffmpeg") is None:
|
||||||
|
raise RuntimeError(
|
||||||
|
"ffmpeg not found on PATH. Install via your package manager or "
|
||||||
|
"download from https://ffmpeg.org/"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def pixel_art_video(
|
||||||
|
base_image,
|
||||||
|
output_path,
|
||||||
|
scene="night",
|
||||||
|
duration=6,
|
||||||
|
fps=15,
|
||||||
|
seed=None,
|
||||||
|
export_gif=False,
|
||||||
|
):
|
||||||
|
"""Overlay pixel animations onto a base image and encode to MP4.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
base_image: path to source image (ideally already pixel-art styled)
|
||||||
|
output_path: path to MP4 output (GIF sibling written if export_gif=True)
|
||||||
|
scene: key from SCENES (night, urban, storm, snow, fire, ...)
|
||||||
|
duration: seconds of animation
|
||||||
|
fps: frames per second (default 15 for retro feel)
|
||||||
|
seed: optional int for reproducible animation placement
|
||||||
|
export_gif: also write a GIF alongside the MP4
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
(mp4_path, gif_path_or_None)
|
||||||
|
"""
|
||||||
|
if scene not in SCENES:
|
||||||
|
raise ValueError(
|
||||||
|
f"Unknown scene {scene!r}. Choose from: {sorted(SCENES)}"
|
||||||
|
)
|
||||||
|
_ensure_ffmpeg()
|
||||||
|
|
||||||
|
base = Image.open(base_image).convert("RGB")
|
||||||
|
W, H = base.size
|
||||||
|
|
||||||
|
rng = random.Random(seed if seed is not None else 42)
|
||||||
|
layers = []
|
||||||
|
for name in SCENES[scene]:
|
||||||
|
init_fn, draw_fn = _LAYERS[name]
|
||||||
|
layers.append((draw_fn, init_fn(rng, W, H)))
|
||||||
|
|
||||||
|
n_frames = fps * duration
|
||||||
|
os.makedirs(os.path.dirname(os.path.abspath(output_path)) or ".", exist_ok=True)
|
||||||
|
|
||||||
|
with tempfile.TemporaryDirectory(prefix="pixelart_frames_") as frames_dir:
|
||||||
|
for frame_idx in range(n_frames):
|
||||||
|
canvas = base.copy()
|
||||||
|
draw = ImageDraw.Draw(canvas)
|
||||||
|
t = frame_idx / fps
|
||||||
|
for draw_fn, state in layers:
|
||||||
|
draw_fn(draw, state, t, W, H)
|
||||||
|
canvas.save(os.path.join(frames_dir, f"frame_{frame_idx:04d}.png"))
|
||||||
|
|
||||||
|
subprocess.run(
|
||||||
|
["ffmpeg", "-y", "-loglevel", "error",
|
||||||
|
"-framerate", str(fps),
|
||||||
|
"-i", os.path.join(frames_dir, "frame_%04d.png"),
|
||||||
|
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-crf", "18",
|
||||||
|
output_path],
|
||||||
|
check=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
gif_path = None
|
||||||
|
if export_gif:
|
||||||
|
gif_path = output_path.rsplit(".", 1)[0] + ".gif"
|
||||||
|
subprocess.run(
|
||||||
|
["ffmpeg", "-y", "-loglevel", "error",
|
||||||
|
"-framerate", str(fps),
|
||||||
|
"-i", os.path.join(frames_dir, "frame_%04d.png"),
|
||||||
|
"-vf",
|
||||||
|
"scale=320:-1:flags=neighbor,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse",
|
||||||
|
"-loop", "0",
|
||||||
|
gif_path],
|
||||||
|
check=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
return output_path, gif_path
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
import argparse
|
||||||
|
p = argparse.ArgumentParser(description="Overlay pixel animations onto an image → MP4.")
|
||||||
|
p.add_argument("base_image")
|
||||||
|
p.add_argument("output")
|
||||||
|
p.add_argument("--scene", default="night", choices=sorted(SCENES))
|
||||||
|
p.add_argument("--duration", type=int, default=6)
|
||||||
|
p.add_argument("--fps", type=int, default=15)
|
||||||
|
p.add_argument("--seed", type=int, default=None)
|
||||||
|
p.add_argument("--gif", action="store_true")
|
||||||
|
args = p.parse_args()
|
||||||
|
mp4, gif = pixel_art_video(
|
||||||
|
args.base_image, args.output,
|
||||||
|
scene=args.scene, duration=args.duration,
|
||||||
|
fps=args.fps, seed=args.seed, export_gif=args.gif,
|
||||||
|
)
|
||||||
|
print(f"Wrote {mp4}")
|
||||||
|
if gif:
|
||||||
|
print(f"Wrote {gif}")
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Pipecat Bot Template — OpenAILLMService avec API Hermes
|
||||||
|
========================================================
|
||||||
|
Template minimal pour un bot texte via WebSocket.
|
||||||
|
Adapté de ~/workspace/jarvis-pipecat/bot.py
|
||||||
|
|
||||||
|
Prérequis: pipecat-ai[websocket]>=1.4.0, openai, uvicorn, websockets
|
||||||
|
"""
|
||||||
|
|
||||||
|
import json, os, subprocess, sys
|
||||||
|
from fastapi import FastAPI, WebSocket
|
||||||
|
from fastapi.responses import HTMLResponse
|
||||||
|
import uvicorn
|
||||||
|
|
||||||
|
# ⚠️ Pipecat 1.4.0 : utiliser `import module as m` (from-import cassé)
|
||||||
|
import pipecat.frames.frames as pframes
|
||||||
|
from pipecat.pipeline.pipeline import Pipeline
|
||||||
|
from pipecat.pipeline.runner import PipelineRunner
|
||||||
|
from pipecat.pipeline.task import PipelineTask
|
||||||
|
from pipecat.processors.frame_processor import FrameProcessor, FrameDirection
|
||||||
|
from pipecat.services.openai.llm import OpenAILLMService
|
||||||
|
from pipecat.transports.websocket.fastapi import FastAPIWebsocketParams, FastAPIWebsocketTransport
|
||||||
|
|
||||||
|
# ─── Config ──────────────────────────────────────────────────
|
||||||
|
PORT = int(os.environ.get("BOT_PORT", "7081"))
|
||||||
|
HERMES_URL = os.environ.get("HERMES_API_URL", "http://localhost:8642/v1")
|
||||||
|
HERMES_KEY = os.environ.get("HERMES_API_KEY", "")
|
||||||
|
SYSTEM_PROMPT = "Tu es un assistant vocal concis. Parle en français, 3 phrases max."
|
||||||
|
|
||||||
|
|
||||||
|
# ─── Processeur custom : texte → LLMMessagesFrame ────────────
|
||||||
|
class TextToLLMProcessor(FrameProcessor):
|
||||||
|
"""Remplace le context aggregator (buggé en 1.4.0)."""
|
||||||
|
|
||||||
|
def __init__(self, system_prompt: str):
|
||||||
|
super().__init__()
|
||||||
|
self._messages = [{"role": "system", "content": system_prompt}]
|
||||||
|
|
||||||
|
async def process_frame(self, frame, direction):
|
||||||
|
await self.push_frame(frame, direction)
|
||||||
|
if isinstance(frame, pframes.InputTransportMessageFrame):
|
||||||
|
text = frame.message
|
||||||
|
if isinstance(text, str) and text.strip():
|
||||||
|
self._messages.append({"role": "user", "content": text})
|
||||||
|
if len(self._messages) > 21:
|
||||||
|
self._messages = [self._messages[0]] + self._messages[-20:]
|
||||||
|
await self.push_frame(pframes.LLMMessagesFrame(messages=list(self._messages)))
|
||||||
|
elif isinstance(frame, pframes.LLMFullResponseStartFrame):
|
||||||
|
self._response_text = ""
|
||||||
|
elif isinstance(frame, pframes.TextFrame):
|
||||||
|
if hasattr(self, '_response_text'):
|
||||||
|
self._response_text += frame.text
|
||||||
|
elif isinstance(frame, pframes.LLMFullResponseEndFrame):
|
||||||
|
if hasattr(self, '_response_text') and self._response_text.strip():
|
||||||
|
self._messages.append({"role": "assistant", "content": self._response_text.strip()})
|
||||||
|
del self._response_text
|
||||||
|
|
||||||
|
|
||||||
|
# ─── App ─────────────────────────────────────────────────────
|
||||||
|
app = FastAPI()
|
||||||
|
|
||||||
|
|
||||||
|
@app.websocket("/ws")
|
||||||
|
async def ws_endpoint(websocket: WebSocket):
|
||||||
|
transport = FastAPIWebsocketTransport(
|
||||||
|
websocket=websocket,
|
||||||
|
params=FastAPIWebsocketParams(add_wav_header=False, session_timeout=None),
|
||||||
|
)
|
||||||
|
llm = OpenAILLMService(
|
||||||
|
api_key=HERMES_KEY,
|
||||||
|
base_url=HERMES_URL,
|
||||||
|
settings=OpenAILLMService.Settings(model="Hermes Agent", temperature=0.8, max_tokens=300, system_instruction=SYSTEM_PROMPT),
|
||||||
|
)
|
||||||
|
proc = TextToLLMProcessor(SYSTEM_PROMPT)
|
||||||
|
pipeline = Pipeline(processors=[transport.input(), proc, llm, transport.output()])
|
||||||
|
task = PipelineTask(pipeline)
|
||||||
|
runner = PipelineRunner()
|
||||||
|
await runner.run(task)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
uvicorn.run(app, host="0.0.0.0", port=PORT)
|
||||||
@@ -0,0 +1,88 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Scan ~/.hermes/skills/ for corrupted .py scripts.
|
||||||
|
|
||||||
|
Usage: python corruption-scanner.py [--fix-dir /path/to/output]
|
||||||
|
Output: JSON summary of clean vs corrupted files.
|
||||||
|
"""
|
||||||
|
import json, os, sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
def scan_skills(base_dir):
|
||||||
|
results = {"clean": [], "corrupted": [], "skipped": []}
|
||||||
|
|
||||||
|
for root, dirs, files in os.walk(base_dir):
|
||||||
|
if '__pycache__' in root or 'venv' in root:
|
||||||
|
continue
|
||||||
|
|
||||||
|
for fname in files:
|
||||||
|
if not fname.endswith('.py'):
|
||||||
|
continue
|
||||||
|
|
||||||
|
fpath = os.path.join(root, fname)
|
||||||
|
size = os.path.getsize(fpath)
|
||||||
|
|
||||||
|
# Skip empty __init__.py (normal)
|
||||||
|
if fname == '__init__.py' and size <= 50:
|
||||||
|
results["skipped"].append({"path": fpath, "reason": "empty __init__.py"})
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(fpath, 'r') as f:
|
||||||
|
content = f.read(1000)
|
||||||
|
|
||||||
|
first_line = content.strip().split('\n')[0].lower()
|
||||||
|
|
||||||
|
if '% partial rewrite' in first_line[:200] or 'latex' in first_line[:200]:
|
||||||
|
results["corrupted"].append({
|
||||||
|
"path": fpath,
|
||||||
|
"size": size,
|
||||||
|
"type": "latex_contamination"
|
||||||
|
})
|
||||||
|
elif first_line.startswith('%pdf') or '%PDF' in first_line[:20]:
|
||||||
|
results["corrupted"].append({
|
||||||
|
"path": fpath,
|
||||||
|
"size": size,
|
||||||
|
"type": "pdf_binary"
|
||||||
|
})
|
||||||
|
elif len(content) > 200000:
|
||||||
|
results["corrupted"].append({
|
||||||
|
"path": fpath,
|
||||||
|
"size": size,
|
||||||
|
"type": "oversized"
|
||||||
|
})
|
||||||
|
else:
|
||||||
|
results["clean"].append({"path": fpath, "size": size})
|
||||||
|
|
||||||
|
except UnicodeDecodeError:
|
||||||
|
results["skipped"].append({"path": fpath, "reason": "binary file"})
|
||||||
|
except Exception as e:
|
||||||
|
results["skipped"].append({"path": fpath, "reason": str(e)})
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
base = os.environ.get('HERMES_HOME', str(Path.home() / '.hermes'))
|
||||||
|
skills_dir = os.path.join(base, 'skills')
|
||||||
|
|
||||||
|
fix_dir = None
|
||||||
|
args = sys.argv[1:]
|
||||||
|
if '--fix-dir' in args:
|
||||||
|
idx = args.index('--fix-dir')
|
||||||
|
if idx + 1 < len(args):
|
||||||
|
fix_dir = args[idx + 1]
|
||||||
|
|
||||||
|
results = scan_skills(skills_dir)
|
||||||
|
|
||||||
|
print(json.dumps(results, indent=2, ensure_ascii=False))
|
||||||
|
|
||||||
|
# Summary
|
||||||
|
print(f"\n{'='*60}")
|
||||||
|
print(f"Scan complete:")
|
||||||
|
print(f" Clean: {len(results['clean'])}")
|
||||||
|
print(f" Corrupted: {len(results['corrupted'])}")
|
||||||
|
print(f" Skipped: {len(results['skipped'])}")
|
||||||
|
|
||||||
|
if results['corrupted']:
|
||||||
|
print(f"\nCorrupted files:")
|
||||||
|
for f in results['corrupted']:
|
||||||
|
print(f" ❌ {f['path']} ({f['size']:,} bytes) - {f['type']}")
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Fetch a YouTube video transcript and output it as structured JSON.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
uv run python3 fetch_transcript.py <url_or_video_id> [--language en,tr] [--timestamps]
|
||||||
|
|
||||||
|
Output (JSON):
|
||||||
|
{
|
||||||
|
"video_id": "...",
|
||||||
|
"language": "en",
|
||||||
|
"segments": [{"text": "...", "start": 0.0, "duration": 2.5}, ...],
|
||||||
|
"full_text": "complete transcript as plain text",
|
||||||
|
"timestamped_text": "00:00 first line\n00:05 second line\n..."
|
||||||
|
}
|
||||||
|
|
||||||
|
Install dependency: uv pip install youtube-transcript-api
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
|
||||||
|
def extract_video_id(url_or_id: str) -> str:
|
||||||
|
"""Extract the 11-character video ID from various YouTube URL formats."""
|
||||||
|
url_or_id = url_or_id.strip()
|
||||||
|
patterns = [
|
||||||
|
r'(?:v=|youtu\.be/|shorts/|embed/|live/)([a-zA-Z0-9_-]{11})',
|
||||||
|
r'^([a-zA-Z0-9_-]{11})$',
|
||||||
|
]
|
||||||
|
for pattern in patterns:
|
||||||
|
match = re.search(pattern, url_or_id)
|
||||||
|
if match:
|
||||||
|
return match.group(1)
|
||||||
|
return url_or_id
|
||||||
|
|
||||||
|
|
||||||
|
def format_timestamp(seconds: float) -> str:
|
||||||
|
"""Convert seconds to HH:MM:SS or MM:SS format."""
|
||||||
|
total = int(seconds)
|
||||||
|
h, remainder = divmod(total, 3600)
|
||||||
|
m, s = divmod(remainder, 60)
|
||||||
|
if h > 0:
|
||||||
|
return f"{h}:{m:02d}:{s:02d}"
|
||||||
|
return f"{m}:{s:02d}"
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_transcript(video_id: str, languages: list = None):
|
||||||
|
"""Fetch transcript segments from YouTube.
|
||||||
|
|
||||||
|
Returns a list of dicts with 'text', 'start', and 'duration' keys.
|
||||||
|
Compatible with youtube-transcript-api v1.x.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from youtube_transcript_api import YouTubeTranscriptApi
|
||||||
|
except ImportError:
|
||||||
|
print("Error: youtube-transcript-api not installed. Run: uv pip install youtube-transcript-api",
|
||||||
|
file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
api = YouTubeTranscriptApi()
|
||||||
|
if languages:
|
||||||
|
result = api.fetch(video_id, languages=languages)
|
||||||
|
else:
|
||||||
|
result = api.fetch(video_id)
|
||||||
|
|
||||||
|
# v1.x returns FetchedTranscriptSnippet objects; normalize to dicts
|
||||||
|
return [
|
||||||
|
{"text": seg.text, "start": seg.start, "duration": seg.duration}
|
||||||
|
for seg in result
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(description="Fetch YouTube transcript as JSON")
|
||||||
|
parser.add_argument("url", help="YouTube URL or video ID")
|
||||||
|
parser.add_argument("--language", "-l", default=None,
|
||||||
|
help="Comma-separated language codes (e.g. en,tr). Default: auto")
|
||||||
|
parser.add_argument("--timestamps", "-t", action="store_true",
|
||||||
|
help="Include timestamped text in output")
|
||||||
|
parser.add_argument("--text-only", action="store_true",
|
||||||
|
help="Output plain text instead of JSON")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
video_id = extract_video_id(args.url)
|
||||||
|
languages = [l.strip() for l in args.language.split(",")] if args.language else None
|
||||||
|
|
||||||
|
try:
|
||||||
|
segments = fetch_transcript(video_id, languages)
|
||||||
|
except Exception as e:
|
||||||
|
error_msg = str(e)
|
||||||
|
if "disabled" in error_msg.lower():
|
||||||
|
print(json.dumps({"error": "Transcripts are disabled for this video."}))
|
||||||
|
elif "no transcript" in error_msg.lower():
|
||||||
|
print(json.dumps({"error": f"No transcript found. Try specifying a language with --language."}))
|
||||||
|
else:
|
||||||
|
print(json.dumps({"error": error_msg}))
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
full_text = " ".join(seg["text"] for seg in segments)
|
||||||
|
timestamped = "\n".join(
|
||||||
|
f"{format_timestamp(seg['start'])} {seg['text']}" for seg in segments
|
||||||
|
)
|
||||||
|
|
||||||
|
if args.text_only:
|
||||||
|
print(timestamped if args.timestamps else full_text)
|
||||||
|
return
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"video_id": video_id,
|
||||||
|
"segment_count": len(segments),
|
||||||
|
"duration": format_timestamp(segments[-1]["start"] + segments[-1]["duration"]) if segments else "0:00",
|
||||||
|
"full_text": full_text,
|
||||||
|
}
|
||||||
|
if args.timestamps:
|
||||||
|
result["timestamped_text"] = timestamped
|
||||||
|
|
||||||
|
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,124 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Fetch a YouTube video transcript and output it as structured JSON.
|
||||||
|
|
||||||
|
Usage:
|
||||||
|
uv run python3 fetch_transcript.py <url_or_video_id> [--language en,tr] [--timestamps]
|
||||||
|
|
||||||
|
Output (JSON):
|
||||||
|
{
|
||||||
|
"video_id": "...",
|
||||||
|
"language": "en",
|
||||||
|
"segments": [{"text": "...", "start": 0.0, "duration": 2.5}, ...],
|
||||||
|
"full_text": "complete transcript as plain text",
|
||||||
|
"timestamped_text": "00:00 first line\n00:05 second line\n..."
|
||||||
|
}
|
||||||
|
|
||||||
|
Install dependency: uv pip install youtube-transcript-api
|
||||||
|
"""
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import re
|
||||||
|
import sys
|
||||||
|
|
||||||
|
|
||||||
|
def extract_video_id(url_or_id: str) -> str:
|
||||||
|
"""Extract the 11-character video ID from various YouTube URL formats."""
|
||||||
|
url_or_id = url_or_id.strip()
|
||||||
|
patterns = [
|
||||||
|
r'(?:v=|youtu\.be/|shorts/|embed/|live/)([a-zA-Z0-9_-]{11})',
|
||||||
|
r'^([a-zA-Z0-9_-]{11})$',
|
||||||
|
]
|
||||||
|
for pattern in patterns:
|
||||||
|
match = re.search(pattern, url_or_id)
|
||||||
|
if match:
|
||||||
|
return match.group(1)
|
||||||
|
return url_or_id
|
||||||
|
|
||||||
|
|
||||||
|
def format_timestamp(seconds: float) -> str:
|
||||||
|
"""Convert seconds to HH:MM:SS or MM:SS format."""
|
||||||
|
total = int(seconds)
|
||||||
|
h, remainder = divmod(total, 3600)
|
||||||
|
m, s = divmod(remainder, 60)
|
||||||
|
if h > 0:
|
||||||
|
return f"{h}:{m:02d}:{s:02d}"
|
||||||
|
return f"{m}:{s:02d}"
|
||||||
|
|
||||||
|
|
||||||
|
def fetch_transcript(video_id: str, languages: list = None):
|
||||||
|
"""Fetch transcript segments from YouTube.
|
||||||
|
|
||||||
|
Returns a list of dicts with 'text', 'start', and 'duration' keys.
|
||||||
|
Compatible with youtube-transcript-api v1.x.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from youtube_transcript_api import YouTubeTranscriptApi
|
||||||
|
except ImportError:
|
||||||
|
print("Error: youtube-transcript-api not installed. Run: uv pip install youtube-transcript-api",
|
||||||
|
file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
api = YouTubeTranscriptApi()
|
||||||
|
if languages:
|
||||||
|
result = api.fetch(video_id, languages=languages)
|
||||||
|
else:
|
||||||
|
result = api.fetch(video_id)
|
||||||
|
|
||||||
|
# v1.x returns FetchedTranscriptSnippet objects; normalize to dicts
|
||||||
|
return [
|
||||||
|
{"text": seg.text, "start": seg.start, "duration": seg.duration}
|
||||||
|
for seg in result
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
parser = argparse.ArgumentParser(description="Fetch YouTube transcript as JSON")
|
||||||
|
parser.add_argument("url", help="YouTube URL or video ID")
|
||||||
|
parser.add_argument("--language", "-l", default=None,
|
||||||
|
help="Comma-separated language codes (e.g. en,tr). Default: auto")
|
||||||
|
parser.add_argument("--timestamps", "-t", action="store_true",
|
||||||
|
help="Include timestamped text in output")
|
||||||
|
parser.add_argument("--text-only", action="store_true",
|
||||||
|
help="Output plain text instead of JSON")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
video_id = extract_video_id(args.url)
|
||||||
|
languages = [l.strip() for l in args.language.split(",")] if args.language else None
|
||||||
|
|
||||||
|
try:
|
||||||
|
segments = fetch_transcript(video_id, languages)
|
||||||
|
except Exception as e:
|
||||||
|
error_msg = str(e)
|
||||||
|
if "disabled" in error_msg.lower():
|
||||||
|
print(json.dumps({"error": "Transcripts are disabled for this video."}))
|
||||||
|
elif "no transcript" in error_msg.lower():
|
||||||
|
print(json.dumps({"error": f"No transcript found. Try specifying a language with --language."}))
|
||||||
|
else:
|
||||||
|
print(json.dumps({"error": error_msg}))
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
full_text = " ".join(seg["text"] for seg in segments)
|
||||||
|
timestamped = "\n".join(
|
||||||
|
f"{format_timestamp(seg['start'])} {seg['text']}" for seg in segments
|
||||||
|
)
|
||||||
|
|
||||||
|
if args.text_only:
|
||||||
|
print(timestamped if args.timestamps else full_text)
|
||||||
|
return
|
||||||
|
|
||||||
|
result = {
|
||||||
|
"video_id": video_id,
|
||||||
|
"segment_count": len(segments),
|
||||||
|
"duration": format_timestamp(segments[-1]["start"] + segments[-1]["duration"]) if segments else "0:00",
|
||||||
|
"full_text": full_text,
|
||||||
|
}
|
||||||
|
if args.timestamps:
|
||||||
|
result["timestamped_text"] = timestamped
|
||||||
|
|
||||||
|
print(json.dumps(result, ensure_ascii=False, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,54 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Analyse une image avec l'API Gemini et écrit le résultat dans un fichier.
|
||||||
|
Usage: python3 gemini_analyze_frame.py <image_path> <output_path>
|
||||||
|
|
||||||
|
La clé API Google doit être dans /tmp/gkey.txt (écrite par l'utilisateur
|
||||||
|
car Hermes masque les clés API dans tous les outils).
|
||||||
|
"""
|
||||||
|
import subprocess, base64, json, sys, os
|
||||||
|
|
||||||
|
IMAGE_PATH = sys.argv[1]
|
||||||
|
OUTPUT_PATH = sys.argv[2]
|
||||||
|
|
||||||
|
# Lire la clé depuis le fichier (contourne le masking Hermes)
|
||||||
|
with open("/tmp/gkey.txt") as f:
|
||||||
|
API_KEY = f.read().strip()
|
||||||
|
|
||||||
|
# Encoder l'image
|
||||||
|
with open(IMAGE_PATH, "rb") as f:
|
||||||
|
img_b64 = base64.b64encode(f.read()).decode()
|
||||||
|
|
||||||
|
# Construire la requête Gemini
|
||||||
|
body = {
|
||||||
|
"contents": [{
|
||||||
|
"parts": [
|
||||||
|
{"text": "Décris précisément cette image. Interface, texte, boutons, cases cochées, rôles, menus — tout ce qui est visible. Réponds en français."},
|
||||||
|
{"inline_data": {"mime_type": "image/png", "data": img_b64}}
|
||||||
|
]
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
# Appeler Gemini (curl pour éviter les timeouts Python HTTP)
|
||||||
|
result = subprocess.run([
|
||||||
|
"curl", "-s",
|
||||||
|
"--max-time", "30",
|
||||||
|
"-H", "Content-Type: application/json",
|
||||||
|
"-d", json.dumps(body),
|
||||||
|
f"https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key={API_KEY}"
|
||||||
|
], capture_output=True, text=True, timeout=35)
|
||||||
|
|
||||||
|
data = json.loads(result.stdout)
|
||||||
|
|
||||||
|
if "error" in data:
|
||||||
|
print(f"API ERROR: {data['error']}", file=sys.stderr)
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
text = data["candidates"][0]["content"]["parts"][0]["text"]
|
||||||
|
|
||||||
|
# Écrire dans le fichier de sortie (contourne CCR)
|
||||||
|
with open(OUTPUT_PATH, "w") as f:
|
||||||
|
f.write(text)
|
||||||
|
|
||||||
|
print(f"✅ {len(text)} chars → {OUTPUT_PATH}")
|
||||||
|
print(text[:300])
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Analyse toutes les frames d'un dossier via Gemini et écrit chaque résultat
|
||||||
|
dans un fichier séparé de < 100 caractères (contourne la compression CCR).
|
||||||
|
|
||||||
|
Usage: python3 gemini_analyze_frames.py <frames_dir> <keyfile>
|
||||||
|
Output: /tmp/fr0.txt, /tmp/fr1.txt, ...
|
||||||
|
"""
|
||||||
|
import subprocess, base64, json, sys, glob, os
|
||||||
|
|
||||||
|
FRAMES_DIR = sys.ar.../tmp/video_frames"
|
||||||
|
KEY_FILE = sys.ar...iven"
|
||||||
|
MODEL = "gemini-2.5-flash"
|
||||||
|
API = "https://generativelanguage.googleapis.com/v1beta/models"
|
||||||
|
|
||||||
|
with open(KEY_FILE) as f:
|
||||||
|
API_KEY = f.read().strip()
|
||||||
|
|
||||||
|
frames = sorted(glob.glob(f"{FRAMES_DIR}/frame*.png"))
|
||||||
|
if not frames:
|
||||||
|
frames = sorted(glob.glob(f"{FRAMES_DIR}/*.png"))
|
||||||
|
|
||||||
|
for i, frame in enumerate(frames):
|
||||||
|
try:
|
||||||
|
with open(frame, "rb") as f:
|
||||||
|
img_b64 = base64.b64encode(f.read()).decode()
|
||||||
|
|
||||||
|
body = {
|
||||||
|
"contents": [{
|
||||||
|
"parts": [
|
||||||
|
{"text": "Decris cette image en francais. Sois precis."},
|
||||||
|
{"inline_data": {"mime_type": "image/png", "data": img_b64}}
|
||||||
|
]
|
||||||
|
}]
|
||||||
|
}
|
||||||
|
|
||||||
|
with open("/tmp/gemini_req.json", "w") as f:
|
||||||
|
json.dump(body, f)
|
||||||
|
|
||||||
|
r = subprocess.run([
|
||||||
|
"curl", "-s", "-H", "Content-Type: application/json",
|
||||||
|
"-d", "@/tmp/gemini_req.json",
|
||||||
|
f"{API}/{MODEL}:generateContent?key={API_KEY}"
|
||||||
|
], capture_output=True, text=True, timeout=30)
|
||||||
|
|
||||||
|
data = json.loads(r.stdout)
|
||||||
|
if "candidates" in data and data["candidates"]:
|
||||||
|
text = data["candidates"][0]["content"]["parts"][0]["text"]
|
||||||
|
else:
|
||||||
|
text = f"ERR:{r.stdout[:150]}"
|
||||||
|
|
||||||
|
# Write to tiny file (CCR bypass)
|
||||||
|
with open(f"/tmp/fr{i}.txt", "w") as f:
|
||||||
|
f.write(f"FRAME {i}: {text[:100]}")
|
||||||
|
|
||||||
|
print(f"OK frame {i} → /tmp/fr{i}.txt")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
with open(f"/tmp/fr{i}.txt", "w") as f:
|
||||||
|
f.write(f"FRAME {i} EXC: {e}")
|
||||||
|
print(f"ERR frame {i}: {e}")
|
||||||
|
|
||||||
|
print(f"\nDONE: {len(frames)} frames → /tmp/fr0.txt .. /tmp/fr{len(frames)-1}.txt")
|
||||||
|
print("Lire avec: read_file /tmp/fr0.txt")
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
# Copyright (c) 2020-202x The virtualenv developers
|
||||||
|
#
|
||||||
|
# Permission is hereby granted, free of charge, to any person obtaining
|
||||||
|
# a copy of this software and associated documentation files (the
|
||||||
|
# "Software"), to deal in the Software without restriction, including
|
||||||
|
# without limitation the rights to use, copy, modify, merge, publish,
|
||||||
|
# distribute, sublicense, and/or sell copies of the Software, and to
|
||||||
|
# permit persons to whom the Software is furnished to do so, subject to
|
||||||
|
# the following conditions:
|
||||||
|
#
|
||||||
|
# The above copyright notice and this permission notice shall be
|
||||||
|
# included in all copies or substantial portions of the Software.
|
||||||
|
#
|
||||||
|
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||||
|
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
||||||
|
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
||||||
|
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
||||||
|
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
||||||
|
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
||||||
|
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Activate virtualenv for current interpreter:
|
||||||
|
|
||||||
|
import runpy
|
||||||
|
runpy.run_path(this_file)
|
||||||
|
|
||||||
|
This can be used when you must use an existing Python interpreter, not the virtualenv bin/python.
|
||||||
|
""" # noqa: D415
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import site
|
||||||
|
import sys
|
||||||
|
|
||||||
|
try:
|
||||||
|
abs_file = os.path.abspath(__file__)
|
||||||
|
except NameError as exc:
|
||||||
|
msg = "You must use import runpy; runpy.run_path(this_file)"
|
||||||
|
raise AssertionError(msg) from exc
|
||||||
|
|
||||||
|
bin_dir = os.path.dirname(abs_file)
|
||||||
|
base = bin_dir[: -len("bin") - 1] # strip away the bin part from the __file__, plus the path separator
|
||||||
|
|
||||||
|
# prepend bin to PATH (this file is inside the bin directory)
|
||||||
|
os.environ["PATH"] = os.pathsep.join([bin_dir, *os.environ.get("PATH", "").split(os.pathsep)])
|
||||||
|
os.environ["VIRTUAL_ENV"] = base # virtual env is right above bin directory
|
||||||
|
os.environ["VIRTUAL_ENV_PROMPT"] = "" or os.path.basename(base) # noqa: SIM222
|
||||||
|
|
||||||
|
# add the virtual environments libraries to the host python import mechanism
|
||||||
|
prev_length = len(sys.path)
|
||||||
|
for lib in "../lib/python3.11/site-packages".split(os.pathsep):
|
||||||
|
path = os.path.realpath(os.path.join(bin_dir, lib))
|
||||||
|
site.addsitedir(path)
|
||||||
|
sys.path[:] = sys.path[prev_length:] + sys.path[0:prev_length]
|
||||||
|
|
||||||
|
sys.real_prefix = sys.prefix
|
||||||
|
sys.prefix = base
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
"""Patches that are applied at runtime to the virtual environment."""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
|
||||||
|
VIRTUALENV_PATCH_FILE = os.path.join(__file__)
|
||||||
|
|
||||||
|
|
||||||
|
def patch_dist(dist):
|
||||||
|
"""
|
||||||
|
Distutils allows user to configure some arguments via a configuration file:
|
||||||
|
https://docs.python.org/3.11/install/index.html#distutils-configuration-files.
|
||||||
|
|
||||||
|
Some of this arguments though don't make sense in context of the virtual environment files, let's fix them up.
|
||||||
|
""" # noqa: D205
|
||||||
|
# we cannot allow some install config as that would get packages installed outside of the virtual environment
|
||||||
|
old_parse_config_files = dist.Distribution.parse_config_files
|
||||||
|
|
||||||
|
def parse_config_files(self, *args, **kwargs):
|
||||||
|
result = old_parse_config_files(self, *args, **kwargs)
|
||||||
|
install = self.get_option_dict("install")
|
||||||
|
|
||||||
|
if "prefix" in install: # the prefix governs where to install the libraries
|
||||||
|
install["prefix"] = VIRTUALENV_PATCH_FILE, os.path.abspath(sys.prefix)
|
||||||
|
for base in ("purelib", "platlib", "headers", "scripts", "data"):
|
||||||
|
key = f"install_{base}"
|
||||||
|
if key in install: # do not allow global configs to hijack venv paths
|
||||||
|
install.pop(key, None)
|
||||||
|
return result
|
||||||
|
|
||||||
|
dist.Distribution.parse_config_files = parse_config_files
|
||||||
|
|
||||||
|
|
||||||
|
# Import hook that patches some modules to ignore configuration values that break package installation in case
|
||||||
|
# of virtual environments.
|
||||||
|
_DISTUTILS_PATCH = "distutils.dist", "setuptools.dist"
|
||||||
|
# https://docs.python.org/3/library/importlib.html#setting-up-an-importer
|
||||||
|
|
||||||
|
|
||||||
|
class _Finder:
|
||||||
|
"""A meta path finder that allows patching the imported distutils modules."""
|
||||||
|
|
||||||
|
fullname = None
|
||||||
|
|
||||||
|
# lock[0] is threading.Lock(), but initialized lazily to avoid importing threading very early at startup,
|
||||||
|
# because there are gevent-based applications that need to be first to import threading by themselves.
|
||||||
|
# See https://github.com/pypa/virtualenv/issues/1895 for details.
|
||||||
|
lock = [] # noqa: RUF012
|
||||||
|
|
||||||
|
def find_spec(self, fullname, path, target=None): # noqa: ARG002
|
||||||
|
if fullname in _DISTUTILS_PATCH and self.fullname is None:
|
||||||
|
# initialize lock[0] lazily
|
||||||
|
if len(self.lock) == 0:
|
||||||
|
import threading
|
||||||
|
|
||||||
|
lock = threading.Lock()
|
||||||
|
# there is possibility that two threads T1 and T2 are simultaneously running into find_spec,
|
||||||
|
# observing .lock as empty, and further going into hereby initialization. However due to the GIL,
|
||||||
|
# list.append() operation is atomic and this way only one of the threads will "win" to put the lock
|
||||||
|
# - that every thread will use - into .lock[0].
|
||||||
|
# https://docs.python.org/3/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe
|
||||||
|
self.lock.append(lock)
|
||||||
|
|
||||||
|
from functools import partial
|
||||||
|
from importlib.util import find_spec
|
||||||
|
|
||||||
|
with self.lock[0]:
|
||||||
|
self.fullname = fullname
|
||||||
|
try:
|
||||||
|
spec = find_spec(fullname, path)
|
||||||
|
if spec is not None:
|
||||||
|
# https://www.python.org/dev/peps/pep-0451/#how-loading-will-work
|
||||||
|
is_new_api = hasattr(spec.loader, "exec_module")
|
||||||
|
func_name = "exec_module" if is_new_api else "load_module"
|
||||||
|
old = getattr(spec.loader, func_name)
|
||||||
|
func = self.exec_module if is_new_api else self.load_module
|
||||||
|
if old is not func:
|
||||||
|
try: # noqa: SIM105
|
||||||
|
setattr(spec.loader, func_name, partial(func, old))
|
||||||
|
except AttributeError:
|
||||||
|
pass # C-Extension loaders are r/o such as zipimporter with <3.7
|
||||||
|
return spec
|
||||||
|
finally:
|
||||||
|
self.fullname = None
|
||||||
|
return None
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def exec_module(old, module):
|
||||||
|
old(module)
|
||||||
|
if module.__name__ in _DISTUTILS_PATCH:
|
||||||
|
patch_dist(module)
|
||||||
|
|
||||||
|
@staticmethod
|
||||||
|
def load_module(old, name):
|
||||||
|
module = old(name)
|
||||||
|
if module.__name__ in _DISTUTILS_PATCH:
|
||||||
|
patch_dist(module)
|
||||||
|
return module
|
||||||
|
|
||||||
|
|
||||||
|
sys.meta_path.insert(0, _Finder())
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
from .core import contents, where
|
||||||
|
|
||||||
|
__all__ = ["contents", "where"]
|
||||||
|
__version__ = "2026.07.22"
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
import argparse
|
||||||
|
|
||||||
|
from certifi import contents, where
|
||||||
|
|
||||||
|
parser = argparse.ArgumentParser()
|
||||||
|
parser.add_argument("-c", "--contents", action="store_true")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
if args.contents:
|
||||||
|
print(contents())
|
||||||
|
else:
|
||||||
|
print(where())
|
||||||
@@ -0,0 +1,83 @@
|
|||||||
|
"""
|
||||||
|
certifi.py
|
||||||
|
~~~~~~~~~~
|
||||||
|
|
||||||
|
This module returns the installation location of cacert.pem or its contents.
|
||||||
|
"""
|
||||||
|
import sys
|
||||||
|
import atexit
|
||||||
|
|
||||||
|
def exit_cacert_ctx() -> None:
|
||||||
|
_CACERT_CTX.__exit__(None, None, None) # type: ignore[union-attr]
|
||||||
|
|
||||||
|
|
||||||
|
if sys.version_info >= (3, 11):
|
||||||
|
|
||||||
|
from importlib.resources import as_file, files
|
||||||
|
|
||||||
|
_CACERT_CTX = None
|
||||||
|
_CACERT_PATH = None
|
||||||
|
|
||||||
|
def where() -> str:
|
||||||
|
# This is slightly terrible, but we want to delay extracting the file
|
||||||
|
# in cases where we're inside of a zipimport situation until someone
|
||||||
|
# actually calls where(), but we don't want to re-extract the file
|
||||||
|
# on every call of where(), so we'll do it once then store it in a
|
||||||
|
# global variable.
|
||||||
|
global _CACERT_CTX
|
||||||
|
global _CACERT_PATH
|
||||||
|
if _CACERT_PATH is None:
|
||||||
|
# This is slightly janky, the importlib.resources API wants you to
|
||||||
|
# manage the cleanup of this file, so it doesn't actually return a
|
||||||
|
# path, it returns a context manager that will give you the path
|
||||||
|
# when you enter it and will do any cleanup when you leave it. In
|
||||||
|
# the common case of not needing a temporary file, it will just
|
||||||
|
# return the file system location and the __exit__() is a no-op.
|
||||||
|
#
|
||||||
|
# We also have to hold onto the actual context manager, because
|
||||||
|
# it will do the cleanup whenever it gets garbage collected, so
|
||||||
|
# we will also store that at the global level as well.
|
||||||
|
_CACERT_CTX = as_file(files("certifi").joinpath("cacert.pem"))
|
||||||
|
_CACERT_PATH = str(_CACERT_CTX.__enter__())
|
||||||
|
atexit.register(exit_cacert_ctx)
|
||||||
|
|
||||||
|
return _CACERT_PATH
|
||||||
|
|
||||||
|
def contents() -> str:
|
||||||
|
return files("certifi").joinpath("cacert.pem").read_text(encoding="ascii")
|
||||||
|
|
||||||
|
else:
|
||||||
|
|
||||||
|
from importlib.resources import path as get_path, read_text
|
||||||
|
|
||||||
|
_CACERT_CTX = None
|
||||||
|
_CACERT_PATH = None
|
||||||
|
|
||||||
|
def where() -> str:
|
||||||
|
# This is slightly terrible, but we want to delay extracting the
|
||||||
|
# file in cases where we're inside of a zipimport situation until
|
||||||
|
# someone actually calls where(), but we don't want to re-extract
|
||||||
|
# the file on every call of where(), so we'll do it once then store
|
||||||
|
# it in a global variable.
|
||||||
|
global _CACERT_CTX
|
||||||
|
global _CACERT_PATH
|
||||||
|
if _CACERT_PATH is None:
|
||||||
|
# This is slightly janky, the importlib.resources API wants you
|
||||||
|
# to manage the cleanup of this file, so it doesn't actually
|
||||||
|
# return a path, it returns a context manager that will give
|
||||||
|
# you the path when you enter it and will do any cleanup when
|
||||||
|
# you leave it. In the common case of not needing a temporary
|
||||||
|
# file, it will just return the file system location and the
|
||||||
|
# __exit__() is a no-op.
|
||||||
|
#
|
||||||
|
# We also have to hold onto the actual context manager, because
|
||||||
|
# it will do the cleanup whenever it gets garbage collected, so
|
||||||
|
# we will also store that at the global level as well.
|
||||||
|
_CACERT_CTX = get_path("certifi", "cacert.pem")
|
||||||
|
_CACERT_PATH = str(_CACERT_CTX.__enter__())
|
||||||
|
atexit.register(exit_cacert_ctx)
|
||||||
|
|
||||||
|
return _CACERT_PATH
|
||||||
|
|
||||||
|
def contents() -> str:
|
||||||
|
return read_text("certifi", "cacert.pem", encoding="ascii")
|
||||||
Executable
+18
@@ -0,0 +1,18 @@
|
|||||||
|
import os
|
||||||
|
import unittest
|
||||||
|
|
||||||
|
import certifi
|
||||||
|
|
||||||
|
|
||||||
|
class TestCertifi(unittest.TestCase):
|
||||||
|
def test_cabundle_exists(self) -> None:
|
||||||
|
assert os.path.exists(certifi.where())
|
||||||
|
|
||||||
|
def test_read_contents(self) -> None:
|
||||||
|
content = certifi.contents()
|
||||||
|
assert "-----BEGIN CERTIFICATE-----" in content
|
||||||
|
|
||||||
|
def test_py_typed_exists(self) -> None:
|
||||||
|
assert os.path.exists(
|
||||||
|
os.path.join(os.path.dirname(certifi.__file__), 'py.typed')
|
||||||
|
)
|
||||||
+48
@@ -0,0 +1,48 @@
|
|||||||
|
"""
|
||||||
|
Charset-Normalizer
|
||||||
|
~~~~~~~~~~~~~~
|
||||||
|
The Real First Universal Charset Detector.
|
||||||
|
A library that helps you read text from an unknown charset encoding.
|
||||||
|
Motivated by chardet, This package is trying to resolve the issue by taking a new approach.
|
||||||
|
All IANA character set names for which the Python core library provides codecs are supported.
|
||||||
|
|
||||||
|
Basic usage:
|
||||||
|
>>> from charset_normalizer import from_bytes
|
||||||
|
>>> results = from_bytes('Bсеки човек има право на образование. Oбразованието!'.encode('utf_8'))
|
||||||
|
>>> best_guess = results.best()
|
||||||
|
>>> str(best_guess)
|
||||||
|
'Bсеки човек има право на образование. Oбразованието!'
|
||||||
|
|
||||||
|
Others methods and usages are available - see the full documentation
|
||||||
|
at <https://github.com/Ousret/charset_normalizer>.
|
||||||
|
:copyright: (c) Ahmed TAHRI
|
||||||
|
:license: MIT, see LICENSE for more details.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
|
||||||
|
from .api import from_bytes, from_fp, from_path, is_binary
|
||||||
|
from .legacy import detect
|
||||||
|
from .models import CharsetMatch, CharsetMatches
|
||||||
|
from .utils import set_logging_handler
|
||||||
|
from .version import VERSION, __version__
|
||||||
|
|
||||||
|
__all__ = (
|
||||||
|
"from_fp",
|
||||||
|
"from_path",
|
||||||
|
"from_bytes",
|
||||||
|
"is_binary",
|
||||||
|
"detect",
|
||||||
|
"CharsetMatch",
|
||||||
|
"CharsetMatches",
|
||||||
|
"__version__",
|
||||||
|
"VERSION",
|
||||||
|
"set_logging_handler",
|
||||||
|
)
|
||||||
|
|
||||||
|
# Attach a NullHandler to the top level logger by default
|
||||||
|
# https://docs.python.org/3.3/howto/logging.html#configuring-logging-for-a-library
|
||||||
|
|
||||||
|
logging.getLogger("charset_normalizer").addHandler(logging.NullHandler())
|
||||||
+6
@@ -0,0 +1,6 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from .cli import cli_detect
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
cli_detect()
|
||||||
+934
@@ -0,0 +1,934 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from functools import lru_cache
|
||||||
|
from os import PathLike
|
||||||
|
from typing import BinaryIO
|
||||||
|
|
||||||
|
from .cd import (
|
||||||
|
coherence_ratio,
|
||||||
|
encoding_languages,
|
||||||
|
mb_encoding_languages,
|
||||||
|
merge_coherence_ratios,
|
||||||
|
)
|
||||||
|
from .constant import (
|
||||||
|
IANA_SUPPORTED,
|
||||||
|
IANA_SUPPORTED_SIMILAR,
|
||||||
|
TOO_BIG_SEQUENCE,
|
||||||
|
TOO_SMALL_SEQUENCE,
|
||||||
|
TRACE,
|
||||||
|
)
|
||||||
|
from .md import mess_ratio
|
||||||
|
from .models import CharsetMatch, CharsetMatches
|
||||||
|
from .utils import (
|
||||||
|
any_specified_encoding,
|
||||||
|
cut_sequence_chunks,
|
||||||
|
iana_name,
|
||||||
|
identify_sig_or_bom,
|
||||||
|
is_multi_byte_encoding,
|
||||||
|
should_strip_sig_or_bom,
|
||||||
|
)
|
||||||
|
|
||||||
|
logger = logging.getLogger("charset_normalizer")
|
||||||
|
explain_handler = logging.StreamHandler()
|
||||||
|
explain_handler.setFormatter(
|
||||||
|
logging.Formatter("%(asctime)s | %(levelname)s | %(message)s")
|
||||||
|
)
|
||||||
|
|
||||||
|
# Pre-compute a reordered encoding list: multibyte first, then single-byte.
|
||||||
|
# This allows the mb_definitive_match optimization to fire earlier, skipping
|
||||||
|
# all single-byte encodings for genuine CJK content. Multibyte codecs
|
||||||
|
# hard-fail (UnicodeDecodeError) on single-byte data almost instantly, so
|
||||||
|
# testing them first costs negligible time for non-CJK files.
|
||||||
|
# Stable sort on a boolean key: multibyte (False) first, IANA order kept
|
||||||
|
# within each group.
|
||||||
|
IANA_SUPPORTED_MB_FIRST: list[str] = sorted(
|
||||||
|
IANA_SUPPORTED, key=lambda encoding: not is_multi_byte_encoding(encoding)
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def from_bytes(
|
||||||
|
sequences: bytes | bytearray,
|
||||||
|
steps: int = 5,
|
||||||
|
chunk_size: int = 512,
|
||||||
|
threshold: float = 0.2,
|
||||||
|
cp_isolation: list[str] | None = None,
|
||||||
|
cp_exclusion: list[str] | None = None,
|
||||||
|
preemptive_behaviour: bool = True,
|
||||||
|
explain: bool = False,
|
||||||
|
language_threshold: float = 0.1,
|
||||||
|
enable_fallback: bool = True,
|
||||||
|
) -> CharsetMatches:
|
||||||
|
"""
|
||||||
|
Given a raw bytes sequence, return the best possibles charset usable to render str objects.
|
||||||
|
If there is no results, it is a strong indicator that the source is binary/not text.
|
||||||
|
By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence.
|
||||||
|
And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will.
|
||||||
|
|
||||||
|
The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page
|
||||||
|
but never take it for granted. Can improve the performance.
|
||||||
|
|
||||||
|
You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that
|
||||||
|
purpose.
|
||||||
|
|
||||||
|
This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32.
|
||||||
|
By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain'
|
||||||
|
toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging.
|
||||||
|
Custom logging format and handler can be set manually.
|
||||||
|
"""
|
||||||
|
|
||||||
|
if not isinstance(sequences, (bytearray, bytes)):
|
||||||
|
raise TypeError(
|
||||||
|
"Expected object of type bytes or bytearray, got: {}".format(
|
||||||
|
type(sequences)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if explain:
|
||||||
|
previous_logger_level: int = logger.level
|
||||||
|
logger.addHandler(explain_handler)
|
||||||
|
logger.setLevel(TRACE)
|
||||||
|
|
||||||
|
length: int = len(sequences)
|
||||||
|
|
||||||
|
if length == 0:
|
||||||
|
logger.debug("Encoding detection on empty bytes, assuming utf_8 intention.")
|
||||||
|
if explain: # Defensive: ensure exit path clean handler
|
||||||
|
logger.removeHandler(explain_handler)
|
||||||
|
logger.setLevel(previous_logger_level)
|
||||||
|
return CharsetMatches([CharsetMatch(sequences, "utf_8", 0.0, False, [], "")])
|
||||||
|
|
||||||
|
if cp_isolation is not None:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"cp_isolation is set. use this flag for debugging purpose. "
|
||||||
|
"limited list of encoding allowed : %s.",
|
||||||
|
", ".join(cp_isolation),
|
||||||
|
)
|
||||||
|
cp_isolation = [iana_name(cp, False) for cp in cp_isolation]
|
||||||
|
else:
|
||||||
|
cp_isolation = []
|
||||||
|
|
||||||
|
if cp_exclusion is not None:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"cp_exclusion is set. use this flag for debugging purpose. "
|
||||||
|
"limited list of encoding excluded : %s.",
|
||||||
|
", ".join(cp_exclusion),
|
||||||
|
)
|
||||||
|
cp_exclusion = [iana_name(cp, False) for cp in cp_exclusion]
|
||||||
|
else:
|
||||||
|
cp_exclusion = []
|
||||||
|
|
||||||
|
if length <= (chunk_size * steps):
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.",
|
||||||
|
steps,
|
||||||
|
chunk_size,
|
||||||
|
length,
|
||||||
|
)
|
||||||
|
steps = 1
|
||||||
|
chunk_size = length
|
||||||
|
|
||||||
|
if steps > 1 and length / steps < chunk_size:
|
||||||
|
chunk_size = int(length / steps)
|
||||||
|
|
||||||
|
is_too_small_sequence: bool = len(sequences) < TOO_SMALL_SEQUENCE
|
||||||
|
is_too_large_sequence: bool = len(sequences) >= TOO_BIG_SEQUENCE
|
||||||
|
|
||||||
|
if is_too_small_sequence:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Trying to detect encoding from a tiny portion of (%i) byte(s).",
|
||||||
|
length,
|
||||||
|
)
|
||||||
|
elif is_too_large_sequence:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Using lazy str decoding because the payload is quite large, (%i) byte(s).",
|
||||||
|
length,
|
||||||
|
)
|
||||||
|
|
||||||
|
prioritized_encodings: list[str] = []
|
||||||
|
|
||||||
|
specified_encoding: str | None = (
|
||||||
|
any_specified_encoding(sequences) if preemptive_behaviour else None
|
||||||
|
)
|
||||||
|
|
||||||
|
if specified_encoding is not None:
|
||||||
|
prioritized_encodings.append(specified_encoding)
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Detected declarative mark in sequence. Priority +1 given for %s.",
|
||||||
|
specified_encoding,
|
||||||
|
)
|
||||||
|
|
||||||
|
tested: set[str] = set()
|
||||||
|
tested_but_hard_failure: list[str] = []
|
||||||
|
tested_but_soft_failure: list[str] = []
|
||||||
|
soft_failure_skip: set[str] = set()
|
||||||
|
|
||||||
|
# Avoid unoptimized RSS usage.
|
||||||
|
# this cache is mostly interesting for
|
||||||
|
# local usage. Garbage collected at the
|
||||||
|
# end. Like it should.
|
||||||
|
cached_mess_ratio = lru_cache(maxsize=None)(mess_ratio)
|
||||||
|
cached_coherence_ratio = lru_cache(maxsize=None)(coherence_ratio)
|
||||||
|
|
||||||
|
# When a definitive result (chaos=0.0 and good coherence) is found after testing
|
||||||
|
# the prioritized encodings (ascii, utf_8), we can significantly reduce the remaining
|
||||||
|
# work. Encodings that target completely different language families (e.g., Cyrillic
|
||||||
|
# when the definitive match is Latin) are skipped entirely.
|
||||||
|
# Additionally, for same-family encodings that pass chaos probing, we reuse the
|
||||||
|
# definitive match's coherence ratios instead of recomputing them — a major savings
|
||||||
|
# since coherence_ratio accounts for ~30% of total time on slow Latin files.
|
||||||
|
definitive_match_found: bool = False
|
||||||
|
definitive_target_languages: set[str] = set()
|
||||||
|
# After the definitive match fires, we cap the number of additional same-family
|
||||||
|
# single-byte encodings that pass chaos probing. Once we've accumulated enough
|
||||||
|
# good candidates (N), further same-family SB encodings are unlikely to produce
|
||||||
|
# a better best() result and just waste mess_ratio + coherence_ratio time.
|
||||||
|
# The first encoding to trigger the definitive match is NOT counted (it's already in).
|
||||||
|
post_definitive_sb_success_count: int = 0
|
||||||
|
POST_DEFINITIVE_SB_CAP: int = 7
|
||||||
|
|
||||||
|
# When a non-UTF multibyte encoding passes chaos probing with significant multibyte
|
||||||
|
# content (decoded length < 98% of raw length), skip all remaining single-byte encodings.
|
||||||
|
# Rationale: multi-byte decoders (CJK) have strict byte-sequence validation — if they
|
||||||
|
# decode without error AND pass chaos probing with substantial multibyte content, the
|
||||||
|
# data is genuinely multibyte encoded. Single-byte encodings will always decode (every
|
||||||
|
# byte maps to something) but waste time on mess_ratio before failing.
|
||||||
|
# The 98% threshold prevents false triggers on files that happen to have a few valid
|
||||||
|
# multibyte pairs (e.g., cp424/_ude_1.txt where big5 decodes with 99% ratio).
|
||||||
|
mb_definitive_match_found: bool = False
|
||||||
|
|
||||||
|
fallback_ascii: CharsetMatch | None = None
|
||||||
|
fallback_u8: CharsetMatch | None = None
|
||||||
|
fallback_specified: CharsetMatch | None = None
|
||||||
|
|
||||||
|
results: CharsetMatches = CharsetMatches()
|
||||||
|
|
||||||
|
early_stop_results: CharsetMatches = CharsetMatches()
|
||||||
|
|
||||||
|
sig_encoding, sig_payload = identify_sig_or_bom(sequences)
|
||||||
|
|
||||||
|
if sig_encoding is not None:
|
||||||
|
prioritized_encodings.insert(0, sig_encoding)
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Detected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.",
|
||||||
|
len(sig_payload),
|
||||||
|
sig_encoding,
|
||||||
|
)
|
||||||
|
|
||||||
|
prioritized_encodings.append("ascii")
|
||||||
|
|
||||||
|
if "utf_8" not in prioritized_encodings:
|
||||||
|
prioritized_encodings.append("utf_8")
|
||||||
|
|
||||||
|
for encoding_iana in prioritized_encodings + IANA_SUPPORTED_MB_FIRST:
|
||||||
|
if cp_isolation and encoding_iana not in cp_isolation:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if cp_exclusion and encoding_iana in cp_exclusion:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if encoding_iana in tested:
|
||||||
|
continue
|
||||||
|
|
||||||
|
tested.add(encoding_iana)
|
||||||
|
|
||||||
|
decoded_payload: str | None = None
|
||||||
|
bom_or_sig_available: bool = sig_encoding == encoding_iana
|
||||||
|
strip_sig_or_bom: bool = bom_or_sig_available and should_strip_sig_or_bom(
|
||||||
|
encoding_iana
|
||||||
|
)
|
||||||
|
|
||||||
|
if encoding_iana in {"utf_16", "utf_32"} and not bom_or_sig_available:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.",
|
||||||
|
encoding_iana,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
if encoding_iana in {"utf_7"} and not bom_or_sig_available:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Encoding %s won't be tested as-is because detection is unreliable without BOM/SIG.",
|
||||||
|
encoding_iana,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Skip encodings similar to ones that already soft-failed (high mess ratio).
|
||||||
|
# Checked BEFORE the expensive decode attempt.
|
||||||
|
if encoding_iana in soft_failure_skip:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"%s is deemed too similar to a code page that was already considered unsuited. Continuing!",
|
||||||
|
encoding_iana,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
is_multi_byte_decoder: bool = is_multi_byte_encoding(encoding_iana)
|
||||||
|
except (ModuleNotFoundError, ImportError): # Defensive:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Encoding %s does not provide an IncrementalDecoder",
|
||||||
|
encoding_iana,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# When we've already found a definitive match (chaos=0.0 with good coherence)
|
||||||
|
# after testing the prioritized encodings, skip encodings that target
|
||||||
|
# completely different language families. This avoids running expensive
|
||||||
|
# mess_ratio + coherence_ratio on clearly unrelated candidates (e.g., Cyrillic
|
||||||
|
# when the definitive match is Latin-based).
|
||||||
|
if definitive_match_found:
|
||||||
|
if not is_multi_byte_decoder:
|
||||||
|
enc_languages = set(encoding_languages(encoding_iana))
|
||||||
|
else:
|
||||||
|
enc_languages = set(mb_encoding_languages(encoding_iana))
|
||||||
|
if not enc_languages.intersection(definitive_target_languages):
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Skipping %s: definitive match already found, this encoding targets different languages (%s vs %s).",
|
||||||
|
encoding_iana,
|
||||||
|
enc_languages,
|
||||||
|
definitive_target_languages,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# After the definitive match, cap the number of additional same-family
|
||||||
|
# single-byte encodings that pass chaos probing. This avoids testing the
|
||||||
|
# tail of rare, low-value same-family encodings (mac_iceland, cp860, etc.)
|
||||||
|
# that almost never change best() but each cost ~1-2ms of mess_ratio + coherence.
|
||||||
|
if (
|
||||||
|
definitive_match_found
|
||||||
|
and not is_multi_byte_decoder
|
||||||
|
and post_definitive_sb_success_count >= POST_DEFINITIVE_SB_CAP
|
||||||
|
):
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Skipping %s: already accumulated %d same-family results after definitive match (cap=%d).",
|
||||||
|
encoding_iana,
|
||||||
|
post_definitive_sb_success_count,
|
||||||
|
POST_DEFINITIVE_SB_CAP,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# When a multibyte encoding with significant multibyte content has already
|
||||||
|
# passed chaos probing, skip all single-byte encodings. They will either fail
|
||||||
|
# chaos probing (wasting mess_ratio time) or produce inferior results.
|
||||||
|
if mb_definitive_match_found and not is_multi_byte_decoder:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Skipping single-byte %s: multi-byte definitive match already found.",
|
||||||
|
encoding_iana,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Single-byte candidates of regular size defer the expensive whole
|
||||||
|
# payload decode until after chunk probing: single-byte codecs are
|
||||||
|
# stateless (1 byte == 1 char) so decoding chunk slices is provably
|
||||||
|
# identical to slicing the decoded payload, and candidates rejected
|
||||||
|
# by chaos probing (the common case) never pay the full decode nor
|
||||||
|
# the payload hash.
|
||||||
|
deferred_decoding: bool = (
|
||||||
|
not is_multi_byte_decoder and not is_too_large_sequence
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
if is_too_large_sequence and not is_multi_byte_decoder:
|
||||||
|
str(
|
||||||
|
(
|
||||||
|
sequences[: int(50e4)]
|
||||||
|
if not strip_sig_or_bom
|
||||||
|
else sequences[len(sig_payload) : int(50e4)]
|
||||||
|
),
|
||||||
|
encoding=encoding_iana,
|
||||||
|
)
|
||||||
|
elif not deferred_decoding:
|
||||||
|
# UTF-7 BOM is encoded in modified Base64 whose byte boundary
|
||||||
|
# can overlap with the next character. Stripping raw SIG bytes
|
||||||
|
# before decoding may leave stray bytes that decode as garbage.
|
||||||
|
# Decode the full sequence and remove the leading BOM char instead.
|
||||||
|
# see https://github.com/jawah/charset_normalizer/issues/718
|
||||||
|
# and https://github.com/jawah/charset_normalizer/issues/716
|
||||||
|
if encoding_iana == "utf_7" and bom_or_sig_available:
|
||||||
|
decoded_payload = str(
|
||||||
|
sequences,
|
||||||
|
encoding=encoding_iana,
|
||||||
|
)
|
||||||
|
if decoded_payload and decoded_payload[0] == "\ufeff":
|
||||||
|
decoded_payload = decoded_payload[1:]
|
||||||
|
else:
|
||||||
|
decoded_payload = str(
|
||||||
|
(
|
||||||
|
sequences
|
||||||
|
if not strip_sig_or_bom
|
||||||
|
else sequences[len(sig_payload) :]
|
||||||
|
),
|
||||||
|
encoding=encoding_iana,
|
||||||
|
)
|
||||||
|
except (UnicodeDecodeError, LookupError) as e:
|
||||||
|
if not isinstance(e, LookupError):
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Code page %s does not fit given bytes sequence at ALL. %s",
|
||||||
|
encoding_iana,
|
||||||
|
str(e),
|
||||||
|
)
|
||||||
|
tested_but_hard_failure.append(encoding_iana)
|
||||||
|
continue
|
||||||
|
|
||||||
|
r_ = range(
|
||||||
|
0 if not bom_or_sig_available else len(sig_payload),
|
||||||
|
length,
|
||||||
|
int(length / steps),
|
||||||
|
)
|
||||||
|
|
||||||
|
multi_byte_bonus: bool = (
|
||||||
|
is_multi_byte_decoder
|
||||||
|
and decoded_payload is not None
|
||||||
|
and len(decoded_payload) < length
|
||||||
|
)
|
||||||
|
|
||||||
|
if multi_byte_bonus:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Code page %s is a multi byte encoding table and it appear that at least one character "
|
||||||
|
"was encoded using n-bytes.",
|
||||||
|
encoding_iana,
|
||||||
|
)
|
||||||
|
|
||||||
|
max_chunk_gave_up: int = int(len(r_) / 4)
|
||||||
|
|
||||||
|
max_chunk_gave_up = max(max_chunk_gave_up, 2)
|
||||||
|
early_stop_count: int = 0
|
||||||
|
lazy_str_hard_failure = False
|
||||||
|
|
||||||
|
md_chunks: list[str] = []
|
||||||
|
md_ratios = []
|
||||||
|
|
||||||
|
try:
|
||||||
|
for chunk in cut_sequence_chunks(
|
||||||
|
sequences,
|
||||||
|
encoding_iana,
|
||||||
|
r_,
|
||||||
|
chunk_size,
|
||||||
|
bom_or_sig_available,
|
||||||
|
strip_sig_or_bom,
|
||||||
|
sig_payload,
|
||||||
|
is_multi_byte_decoder,
|
||||||
|
decoded_payload,
|
||||||
|
deferred_decoding,
|
||||||
|
):
|
||||||
|
md_chunks.append(chunk)
|
||||||
|
|
||||||
|
md_ratios.append(
|
||||||
|
cached_mess_ratio(
|
||||||
|
chunk,
|
||||||
|
threshold,
|
||||||
|
explain and 1 <= len(cp_isolation) <= 2,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if md_ratios[-1] >= threshold:
|
||||||
|
early_stop_count += 1
|
||||||
|
|
||||||
|
if (early_stop_count >= max_chunk_gave_up) or (
|
||||||
|
bom_or_sig_available and not strip_sig_or_bom
|
||||||
|
):
|
||||||
|
break
|
||||||
|
except (
|
||||||
|
UnicodeDecodeError,
|
||||||
|
LookupError,
|
||||||
|
) as e: # Lazy str loading may have missed something there
|
||||||
|
if deferred_decoding:
|
||||||
|
# Deferred single-byte validation failed on a chunk (or the
|
||||||
|
# codec is unavailable on this interpreter build): identical
|
||||||
|
# outcome and bookkeeping to the eager full-decode failure.
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Code page %s does not fit given bytes sequence at ALL. %s",
|
||||||
|
encoding_iana,
|
||||||
|
str(e),
|
||||||
|
)
|
||||||
|
tested_but_hard_failure.append(encoding_iana)
|
||||||
|
continue
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"LazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %s",
|
||||||
|
encoding_iana,
|
||||||
|
str(e),
|
||||||
|
)
|
||||||
|
early_stop_count = max_chunk_gave_up
|
||||||
|
lazy_str_hard_failure = True
|
||||||
|
|
||||||
|
mean_mess_ratio: float = sum(md_ratios) / len(md_ratios) if md_ratios else 0.0
|
||||||
|
|
||||||
|
# We might want to check the sequence again with the whole content,
|
||||||
|
# but only if initial MD tests passed.
|
||||||
|
if (
|
||||||
|
not lazy_str_hard_failure
|
||||||
|
and is_too_large_sequence
|
||||||
|
and not is_multi_byte_decoder
|
||||||
|
and mean_mess_ratio < threshold
|
||||||
|
and early_stop_count < max_chunk_gave_up
|
||||||
|
):
|
||||||
|
try:
|
||||||
|
sequences[int(50e3) :].decode(encoding_iana, errors="strict")
|
||||||
|
except UnicodeDecodeError as e:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %s",
|
||||||
|
encoding_iana,
|
||||||
|
str(e),
|
||||||
|
)
|
||||||
|
tested_but_hard_failure.append(encoding_iana)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if mean_mess_ratio >= threshold or early_stop_count >= max_chunk_gave_up:
|
||||||
|
tested_but_soft_failure.append(encoding_iana)
|
||||||
|
if encoding_iana in IANA_SUPPORTED_SIMILAR:
|
||||||
|
soft_failure_skip.update(IANA_SUPPORTED_SIMILAR[encoding_iana])
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"%s was excluded because of initial chaos probing. Gave up %i time(s). "
|
||||||
|
"Computed mean chaos is %f %%.",
|
||||||
|
encoding_iana,
|
||||||
|
early_stop_count,
|
||||||
|
round(mean_mess_ratio * 100, ndigits=3),
|
||||||
|
)
|
||||||
|
# Preparing those fallbacks in case we got nothing.
|
||||||
|
if (
|
||||||
|
enable_fallback
|
||||||
|
and encoding_iana
|
||||||
|
in ["ascii", "utf_8", specified_encoding, "utf_16", "utf_32"]
|
||||||
|
and not lazy_str_hard_failure
|
||||||
|
):
|
||||||
|
# Always fully decode payload before.
|
||||||
|
# We've missed a UnicodeDecodeError proof
|
||||||
|
# while issuing release 3.4.8
|
||||||
|
# see https://github.com/jawah/charset_normalizer/issues/771
|
||||||
|
if decoded_payload is None:
|
||||||
|
try:
|
||||||
|
decoded_payload = str(
|
||||||
|
(
|
||||||
|
sequences
|
||||||
|
if not strip_sig_or_bom
|
||||||
|
else sequences[len(sig_payload) :]
|
||||||
|
),
|
||||||
|
encoding=encoding_iana,
|
||||||
|
)
|
||||||
|
except (UnicodeDecodeError, LookupError):
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"%s does not decode the whole payload: fallback entry withheld.",
|
||||||
|
encoding_iana,
|
||||||
|
)
|
||||||
|
continue
|
||||||
|
if is_too_large_sequence:
|
||||||
|
# Don't retain huge payload in RAM.
|
||||||
|
decoded_payload = None
|
||||||
|
|
||||||
|
fallback_entry = CharsetMatch(
|
||||||
|
sequences,
|
||||||
|
encoding_iana,
|
||||||
|
threshold,
|
||||||
|
bom_or_sig_available,
|
||||||
|
[],
|
||||||
|
decoded_payload,
|
||||||
|
preemptive_declaration=specified_encoding,
|
||||||
|
)
|
||||||
|
if encoding_iana == specified_encoding:
|
||||||
|
fallback_specified = fallback_entry
|
||||||
|
elif encoding_iana == "ascii":
|
||||||
|
fallback_ascii = fallback_entry
|
||||||
|
else:
|
||||||
|
fallback_u8 = fallback_entry
|
||||||
|
continue
|
||||||
|
|
||||||
|
if deferred_decoding:
|
||||||
|
# The candidate passed chaos probing: perform the whole payload
|
||||||
|
# decode (validation + payload reuse) that was deferred earlier.
|
||||||
|
try:
|
||||||
|
decoded_payload = str(
|
||||||
|
(
|
||||||
|
sequences
|
||||||
|
if not strip_sig_or_bom
|
||||||
|
else sequences[len(sig_payload) :]
|
||||||
|
),
|
||||||
|
encoding=encoding_iana,
|
||||||
|
)
|
||||||
|
except (UnicodeDecodeError, LookupError) as e:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Code page %s does not fit given bytes sequence at ALL. %s",
|
||||||
|
encoding_iana,
|
||||||
|
str(e),
|
||||||
|
)
|
||||||
|
tested_but_hard_failure.append(encoding_iana)
|
||||||
|
continue
|
||||||
|
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"%s passed initial chaos probing. Mean measured chaos is %f %%",
|
||||||
|
encoding_iana,
|
||||||
|
round(mean_mess_ratio * 100, ndigits=3),
|
||||||
|
)
|
||||||
|
|
||||||
|
if not is_multi_byte_decoder:
|
||||||
|
target_languages: list[str] = encoding_languages(encoding_iana)
|
||||||
|
else:
|
||||||
|
target_languages = mb_encoding_languages(encoding_iana)
|
||||||
|
|
||||||
|
if target_languages:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"%s should target any language(s) of %s",
|
||||||
|
encoding_iana,
|
||||||
|
target_languages,
|
||||||
|
)
|
||||||
|
|
||||||
|
cd_ratios = []
|
||||||
|
|
||||||
|
# Run coherence detection on all chunks. We previously tried limiting to
|
||||||
|
# 1-2 chunks for post-definitive encodings to save time, but this caused
|
||||||
|
# coverage regressions by producing unrepresentative coherence scores.
|
||||||
|
# The SB cap and language-family skip optimizations provide sufficient
|
||||||
|
# speedup without sacrificing coherence accuracy.
|
||||||
|
if encoding_iana != "ascii":
|
||||||
|
# We shall skip the CD when its about ASCII
|
||||||
|
# Most of the time its not relevant to run "language-detection" on it.
|
||||||
|
lg_inclusion: str | None = (
|
||||||
|
",".join(target_languages) if target_languages else None
|
||||||
|
)
|
||||||
|
|
||||||
|
for chunk in md_chunks:
|
||||||
|
chunk_languages = cached_coherence_ratio(
|
||||||
|
chunk,
|
||||||
|
language_threshold,
|
||||||
|
lg_inclusion,
|
||||||
|
)
|
||||||
|
|
||||||
|
cd_ratios.append(chunk_languages)
|
||||||
|
|
||||||
|
cd_ratios_merged = merge_coherence_ratios(cd_ratios)
|
||||||
|
|
||||||
|
if cd_ratios_merged:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"We detected language %s using %s",
|
||||||
|
cd_ratios_merged,
|
||||||
|
encoding_iana,
|
||||||
|
)
|
||||||
|
|
||||||
|
current_match = CharsetMatch(
|
||||||
|
sequences,
|
||||||
|
encoding_iana,
|
||||||
|
mean_mess_ratio,
|
||||||
|
bom_or_sig_available,
|
||||||
|
cd_ratios_merged,
|
||||||
|
(
|
||||||
|
decoded_payload
|
||||||
|
if (
|
||||||
|
not is_too_large_sequence
|
||||||
|
or encoding_iana in [specified_encoding, "ascii", "utf_8"]
|
||||||
|
)
|
||||||
|
else None
|
||||||
|
),
|
||||||
|
preemptive_declaration=specified_encoding,
|
||||||
|
)
|
||||||
|
|
||||||
|
results.append(current_match)
|
||||||
|
|
||||||
|
# Count post-definitive same-family SB successes for the early termination cap.
|
||||||
|
# Only count low-mess encodings (< 2%) toward the cap. High-mess encodings are
|
||||||
|
# marginal results that shouldn't prevent better-quality candidates from being
|
||||||
|
# tested. For example, iso8859_4 (mess=0%) should not be skipped just because
|
||||||
|
# 7 high-mess Latin encodings (cp1252 at 8%, etc.) were tried first.
|
||||||
|
if (
|
||||||
|
definitive_match_found
|
||||||
|
and not is_multi_byte_decoder
|
||||||
|
and mean_mess_ratio < 0.02
|
||||||
|
):
|
||||||
|
post_definitive_sb_success_count += 1
|
||||||
|
|
||||||
|
if (
|
||||||
|
encoding_iana in [specified_encoding, "ascii", "utf_8"]
|
||||||
|
and mean_mess_ratio < 0.1
|
||||||
|
):
|
||||||
|
# If md says nothing to worry about, then... stop immediately!
|
||||||
|
if mean_mess_ratio == 0.0:
|
||||||
|
logger.debug(
|
||||||
|
"Encoding detection: %s is most likely the one.",
|
||||||
|
current_match.encoding,
|
||||||
|
)
|
||||||
|
if explain: # Defensive: ensure exit path clean handler
|
||||||
|
logger.removeHandler(explain_handler)
|
||||||
|
logger.setLevel(previous_logger_level)
|
||||||
|
return CharsetMatches([current_match])
|
||||||
|
|
||||||
|
early_stop_results.append(current_match)
|
||||||
|
|
||||||
|
if (
|
||||||
|
len(early_stop_results)
|
||||||
|
and (specified_encoding is None or specified_encoding in tested)
|
||||||
|
and "ascii" in tested
|
||||||
|
and "utf_8" in tested
|
||||||
|
):
|
||||||
|
probable_result = early_stop_results.best()
|
||||||
|
assert probable_result is not None
|
||||||
|
logger.debug(
|
||||||
|
"Encoding detection: %s is most likely the one.",
|
||||||
|
probable_result.encoding,
|
||||||
|
)
|
||||||
|
if explain: # Defensive: ensure exit path clean handler
|
||||||
|
logger.removeHandler(explain_handler)
|
||||||
|
logger.setLevel(previous_logger_level)
|
||||||
|
|
||||||
|
return CharsetMatches([probable_result])
|
||||||
|
|
||||||
|
# Once we find a result with good coherence (>= 0.5) after testing the
|
||||||
|
# prioritized encodings (ascii, utf_8), activate "definitive mode": skip
|
||||||
|
# encodings that target completely different language families. This avoids
|
||||||
|
# running expensive mess_ratio + coherence_ratio on clearly unrelated
|
||||||
|
# candidates (e.g., Cyrillic encodings when the match is Latin-based).
|
||||||
|
# We require coherence >= 0.5 to avoid false positives (e.g., cp1251 decoding
|
||||||
|
# Hebrew text with 0.0 chaos but wrong language detection at coherence 0.33).
|
||||||
|
if not definitive_match_found and not is_multi_byte_decoder:
|
||||||
|
best_coherence = (
|
||||||
|
max((v for _, v in cd_ratios_merged), default=0.0)
|
||||||
|
if cd_ratios_merged
|
||||||
|
else 0.0
|
||||||
|
)
|
||||||
|
if best_coherence >= 0.5 and "ascii" in tested and "utf_8" in tested:
|
||||||
|
definitive_match_found = True
|
||||||
|
definitive_target_languages.update(target_languages)
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Definitive match found: %s (chaos=%.3f, coherence=%.2f). Encodings targeting different language families will be skipped.",
|
||||||
|
encoding_iana,
|
||||||
|
mean_mess_ratio,
|
||||||
|
best_coherence,
|
||||||
|
)
|
||||||
|
|
||||||
|
# When a non-UTF multibyte encoding passes chaos probing with significant
|
||||||
|
# multibyte content (decoded < 98% of raw), activate mb_definitive_match.
|
||||||
|
# This skips all remaining single-byte encodings which would either soft-fail
|
||||||
|
# (running expensive mess_ratio for nothing) or produce inferior results.
|
||||||
|
if (
|
||||||
|
not mb_definitive_match_found
|
||||||
|
and is_multi_byte_decoder
|
||||||
|
and multi_byte_bonus
|
||||||
|
and decoded_payload is not None
|
||||||
|
and len(decoded_payload) < length * 0.98
|
||||||
|
and encoding_iana
|
||||||
|
not in {
|
||||||
|
"utf_8",
|
||||||
|
"utf_8_sig",
|
||||||
|
"utf_16",
|
||||||
|
"utf_16_be",
|
||||||
|
"utf_16_le",
|
||||||
|
"utf_32",
|
||||||
|
"utf_32_be",
|
||||||
|
"utf_32_le",
|
||||||
|
"utf_7",
|
||||||
|
}
|
||||||
|
and "ascii" in tested
|
||||||
|
and "utf_8" in tested
|
||||||
|
):
|
||||||
|
mb_definitive_match_found = True
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Multi-byte definitive match: %s (chaos=%.3f, decoded=%d/%d=%.1f%%). Single-byte encodings will be skipped.",
|
||||||
|
encoding_iana,
|
||||||
|
mean_mess_ratio,
|
||||||
|
len(decoded_payload),
|
||||||
|
length,
|
||||||
|
len(decoded_payload) / length * 100,
|
||||||
|
)
|
||||||
|
|
||||||
|
if encoding_iana == sig_encoding:
|
||||||
|
logger.debug(
|
||||||
|
"Encoding detection: %s is most likely the one as we detected a BOM or SIG within "
|
||||||
|
"the beginning of the sequence.",
|
||||||
|
encoding_iana,
|
||||||
|
)
|
||||||
|
if explain: # Defensive: ensure exit path clean handler
|
||||||
|
logger.removeHandler(explain_handler)
|
||||||
|
logger.setLevel(previous_logger_level)
|
||||||
|
return CharsetMatches([results[encoding_iana]])
|
||||||
|
|
||||||
|
if len(results) == 0:
|
||||||
|
if fallback_u8 or fallback_ascii or fallback_specified:
|
||||||
|
logger.log(
|
||||||
|
TRACE,
|
||||||
|
"Nothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.",
|
||||||
|
)
|
||||||
|
|
||||||
|
if fallback_specified:
|
||||||
|
logger.debug(
|
||||||
|
"Encoding detection: %s will be used as a fallback match",
|
||||||
|
fallback_specified.encoding,
|
||||||
|
)
|
||||||
|
results.append(fallback_specified)
|
||||||
|
elif (
|
||||||
|
(fallback_u8 and fallback_ascii is None)
|
||||||
|
or (
|
||||||
|
fallback_u8
|
||||||
|
and fallback_ascii
|
||||||
|
and fallback_u8.fingerprint != fallback_ascii.fingerprint
|
||||||
|
)
|
||||||
|
or (fallback_u8 is not None)
|
||||||
|
):
|
||||||
|
logger.debug("Encoding detection: utf_8 will be used as a fallback match")
|
||||||
|
results.append(fallback_u8)
|
||||||
|
elif fallback_ascii:
|
||||||
|
logger.debug("Encoding detection: ascii will be used as a fallback match")
|
||||||
|
results.append(fallback_ascii)
|
||||||
|
|
||||||
|
if results:
|
||||||
|
logger.debug(
|
||||||
|
"Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.",
|
||||||
|
results.best().encoding, # type: ignore
|
||||||
|
len(results) - 1,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
logger.debug("Encoding detection: Unable to determine any suitable charset.")
|
||||||
|
|
||||||
|
if explain:
|
||||||
|
logger.removeHandler(explain_handler)
|
||||||
|
logger.setLevel(previous_logger_level)
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def from_fp(
|
||||||
|
fp: BinaryIO,
|
||||||
|
steps: int = 5,
|
||||||
|
chunk_size: int = 512,
|
||||||
|
threshold: float = 0.20,
|
||||||
|
cp_isolation: list[str] | None = None,
|
||||||
|
cp_exclusion: list[str] | None = None,
|
||||||
|
preemptive_behaviour: bool = True,
|
||||||
|
explain: bool = False,
|
||||||
|
language_threshold: float = 0.1,
|
||||||
|
enable_fallback: bool = True,
|
||||||
|
) -> CharsetMatches:
|
||||||
|
"""
|
||||||
|
Same thing than the function from_bytes but using a file pointer that is already ready.
|
||||||
|
Will not close the file pointer.
|
||||||
|
"""
|
||||||
|
return from_bytes(
|
||||||
|
fp.read(),
|
||||||
|
steps,
|
||||||
|
chunk_size,
|
||||||
|
threshold,
|
||||||
|
cp_isolation,
|
||||||
|
cp_exclusion,
|
||||||
|
preemptive_behaviour,
|
||||||
|
explain,
|
||||||
|
language_threshold,
|
||||||
|
enable_fallback,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def from_path(
|
||||||
|
path: str | bytes | PathLike, # type: ignore[type-arg]
|
||||||
|
steps: int = 5,
|
||||||
|
chunk_size: int = 512,
|
||||||
|
threshold: float = 0.20,
|
||||||
|
cp_isolation: list[str] | None = None,
|
||||||
|
cp_exclusion: list[str] | None = None,
|
||||||
|
preemptive_behaviour: bool = True,
|
||||||
|
explain: bool = False,
|
||||||
|
language_threshold: float = 0.1,
|
||||||
|
enable_fallback: bool = True,
|
||||||
|
) -> CharsetMatches:
|
||||||
|
"""
|
||||||
|
Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode.
|
||||||
|
Can raise IOError.
|
||||||
|
"""
|
||||||
|
with open(path, "rb") as fp:
|
||||||
|
return from_fp(
|
||||||
|
fp,
|
||||||
|
steps,
|
||||||
|
chunk_size,
|
||||||
|
threshold,
|
||||||
|
cp_isolation,
|
||||||
|
cp_exclusion,
|
||||||
|
preemptive_behaviour,
|
||||||
|
explain,
|
||||||
|
language_threshold,
|
||||||
|
enable_fallback,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def is_binary(
|
||||||
|
fp_or_path_or_payload: PathLike | str | BinaryIO | bytes, # type: ignore[type-arg]
|
||||||
|
steps: int = 5,
|
||||||
|
chunk_size: int = 512,
|
||||||
|
threshold: float = 0.20,
|
||||||
|
cp_isolation: list[str] | None = None,
|
||||||
|
cp_exclusion: list[str] | None = None,
|
||||||
|
preemptive_behaviour: bool = True,
|
||||||
|
explain: bool = False,
|
||||||
|
language_threshold: float = 0.1,
|
||||||
|
enable_fallback: bool = False,
|
||||||
|
) -> bool:
|
||||||
|
"""
|
||||||
|
Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string.
|
||||||
|
Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match
|
||||||
|
are disabled to be stricter around ASCII-compatible but unlikely to be a string.
|
||||||
|
"""
|
||||||
|
if isinstance(fp_or_path_or_payload, (str, PathLike)):
|
||||||
|
guesses = from_path(
|
||||||
|
fp_or_path_or_payload,
|
||||||
|
steps=steps,
|
||||||
|
chunk_size=chunk_size,
|
||||||
|
threshold=threshold,
|
||||||
|
cp_isolation=cp_isolation,
|
||||||
|
cp_exclusion=cp_exclusion,
|
||||||
|
preemptive_behaviour=preemptive_behaviour,
|
||||||
|
explain=explain,
|
||||||
|
language_threshold=language_threshold,
|
||||||
|
enable_fallback=enable_fallback,
|
||||||
|
)
|
||||||
|
elif isinstance(
|
||||||
|
fp_or_path_or_payload,
|
||||||
|
(
|
||||||
|
bytes,
|
||||||
|
bytearray,
|
||||||
|
),
|
||||||
|
):
|
||||||
|
guesses = from_bytes(
|
||||||
|
fp_or_path_or_payload,
|
||||||
|
steps=steps,
|
||||||
|
chunk_size=chunk_size,
|
||||||
|
threshold=threshold,
|
||||||
|
cp_isolation=cp_isolation,
|
||||||
|
cp_exclusion=cp_exclusion,
|
||||||
|
preemptive_behaviour=preemptive_behaviour,
|
||||||
|
explain=explain,
|
||||||
|
language_threshold=language_threshold,
|
||||||
|
enable_fallback=enable_fallback,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
guesses = from_fp(
|
||||||
|
fp_or_path_or_payload,
|
||||||
|
steps=steps,
|
||||||
|
chunk_size=chunk_size,
|
||||||
|
threshold=threshold,
|
||||||
|
cp_isolation=cp_isolation,
|
||||||
|
cp_exclusion=cp_exclusion,
|
||||||
|
preemptive_behaviour=preemptive_behaviour,
|
||||||
|
explain=explain,
|
||||||
|
language_threshold=language_threshold,
|
||||||
|
enable_fallback=enable_fallback,
|
||||||
|
)
|
||||||
|
|
||||||
|
return not guesses
|
||||||
+467
@@ -0,0 +1,467 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib
|
||||||
|
from codecs import IncrementalDecoder
|
||||||
|
from functools import lru_cache
|
||||||
|
|
||||||
|
from .constant import (
|
||||||
|
FREQUENCIES,
|
||||||
|
KO_NAMES,
|
||||||
|
LANGUAGE_SUPPORTED_COUNT,
|
||||||
|
TOO_SMALL_SEQUENCE,
|
||||||
|
ZH_NAMES,
|
||||||
|
_FREQUENCIES_SET,
|
||||||
|
_FREQUENCIES_RANK,
|
||||||
|
)
|
||||||
|
from .md import _ASCII_CHAR_INFO, _char_info, is_suspiciously_successive_range
|
||||||
|
from .models import CoherenceMatches
|
||||||
|
from .utils import (
|
||||||
|
is_multi_byte_encoding,
|
||||||
|
is_unicode_range_secondary,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def encoding_unicode_range(iana_name: str) -> list[str]:
|
||||||
|
"""
|
||||||
|
Return associated unicode ranges in a single byte code page.
|
||||||
|
"""
|
||||||
|
if is_multi_byte_encoding(iana_name):
|
||||||
|
raise OSError( # Defensive:
|
||||||
|
"Function not supported on multi-byte code page"
|
||||||
|
)
|
||||||
|
|
||||||
|
decoder = importlib.import_module(f"encodings.{iana_name}").IncrementalDecoder
|
||||||
|
|
||||||
|
p: IncrementalDecoder = decoder(errors="ignore")
|
||||||
|
seen_ranges: dict[str, int] = {}
|
||||||
|
character_count: int = 0
|
||||||
|
|
||||||
|
for i in range(0x40, 0xFF):
|
||||||
|
chunk: str = p.decode(bytes([i]))
|
||||||
|
|
||||||
|
if chunk:
|
||||||
|
chunk_codepoint = ord(chunk)
|
||||||
|
character_range: str | None = (
|
||||||
|
_ASCII_CHAR_INFO[chunk_codepoint].range
|
||||||
|
if chunk_codepoint < 128
|
||||||
|
else _char_info(chunk).range
|
||||||
|
)
|
||||||
|
|
||||||
|
if character_range is None:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if not is_unicode_range_secondary(character_range):
|
||||||
|
if character_range not in seen_ranges:
|
||||||
|
seen_ranges[character_range] = 0
|
||||||
|
seen_ranges[character_range] += 1
|
||||||
|
character_count += 1
|
||||||
|
|
||||||
|
return sorted(
|
||||||
|
[
|
||||||
|
character_range
|
||||||
|
for character_range in seen_ranges
|
||||||
|
if seen_ranges[character_range] / character_count >= 0.15
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def unicode_range_languages(primary_range: str) -> list[str]:
|
||||||
|
"""
|
||||||
|
Return inferred languages used with a unicode range.
|
||||||
|
"""
|
||||||
|
languages: list[str] = []
|
||||||
|
|
||||||
|
for language, characters in FREQUENCIES.items():
|
||||||
|
for character in characters:
|
||||||
|
codepoint = ord(character)
|
||||||
|
info = (
|
||||||
|
_ASCII_CHAR_INFO[codepoint]
|
||||||
|
if codepoint < 128
|
||||||
|
else _char_info(character)
|
||||||
|
)
|
||||||
|
if info.range == primary_range:
|
||||||
|
languages.append(language)
|
||||||
|
break
|
||||||
|
|
||||||
|
return languages
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache()
|
||||||
|
def encoding_languages(iana_name: str) -> list[str]:
|
||||||
|
"""
|
||||||
|
Single-byte encoding language association. Some code page are heavily linked to particular language(s).
|
||||||
|
This function does the correspondence.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
unicode_ranges: list[str] = encoding_unicode_range(iana_name)
|
||||||
|
except ImportError: # Defensive: encoding unavailable on this build.
|
||||||
|
return []
|
||||||
|
|
||||||
|
primary_range: str | None = None
|
||||||
|
|
||||||
|
for specified_range in unicode_ranges:
|
||||||
|
if "Latin" not in specified_range:
|
||||||
|
primary_range = specified_range
|
||||||
|
break
|
||||||
|
|
||||||
|
if primary_range is None:
|
||||||
|
return ["Latin Based"]
|
||||||
|
|
||||||
|
return unicode_range_languages(primary_range)
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache()
|
||||||
|
def mb_encoding_languages(iana_name: str) -> list[str]:
|
||||||
|
"""
|
||||||
|
Multi-byte encoding language association. Some code page are heavily linked to particular language(s).
|
||||||
|
This function does the correspondence.
|
||||||
|
"""
|
||||||
|
if (
|
||||||
|
iana_name.startswith("shift_")
|
||||||
|
or iana_name.startswith("iso2022_jp")
|
||||||
|
or iana_name.startswith("euc_j")
|
||||||
|
or iana_name == "cp932"
|
||||||
|
):
|
||||||
|
return ["Japanese"]
|
||||||
|
if iana_name.startswith("gb") or iana_name in ZH_NAMES:
|
||||||
|
return ["Chinese"]
|
||||||
|
if iana_name.startswith("iso2022_kr") or iana_name in KO_NAMES:
|
||||||
|
return ["Korean"]
|
||||||
|
|
||||||
|
return []
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=LANGUAGE_SUPPORTED_COUNT)
|
||||||
|
def get_target_features(language: str) -> tuple[bool, bool]:
|
||||||
|
"""
|
||||||
|
Determine main aspects from a supported language if it contains accents and if is pure Latin.
|
||||||
|
"""
|
||||||
|
target_have_accents: bool = False
|
||||||
|
target_pure_latin: bool = True
|
||||||
|
|
||||||
|
for character in FREQUENCIES[language]:
|
||||||
|
codepoint = ord(character)
|
||||||
|
info = _ASCII_CHAR_INFO[codepoint] if codepoint < 128 else _char_info(character)
|
||||||
|
if not target_have_accents and info.accentuated:
|
||||||
|
target_have_accents = True
|
||||||
|
if target_pure_latin and not info.latin:
|
||||||
|
target_pure_latin = False
|
||||||
|
|
||||||
|
return target_have_accents, target_pure_latin
|
||||||
|
|
||||||
|
|
||||||
|
def alphabet_languages(
|
||||||
|
characters: list[str], ignore_non_latin: bool = False
|
||||||
|
) -> list[str]:
|
||||||
|
"""
|
||||||
|
Return associated languages associated to given characters.
|
||||||
|
"""
|
||||||
|
languages: list[tuple[str, float]] = []
|
||||||
|
|
||||||
|
characters_set: frozenset[str] = frozenset(characters)
|
||||||
|
source_have_accents = False
|
||||||
|
for character in characters:
|
||||||
|
codepoint = ord(character)
|
||||||
|
info = _ASCII_CHAR_INFO[codepoint] if codepoint < 128 else _char_info(character)
|
||||||
|
if info.accentuated:
|
||||||
|
source_have_accents = True
|
||||||
|
break
|
||||||
|
|
||||||
|
for language, language_characters in FREQUENCIES.items():
|
||||||
|
target_have_accents, target_pure_latin = get_target_features(language)
|
||||||
|
|
||||||
|
if ignore_non_latin and not target_pure_latin:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if not target_have_accents and source_have_accents:
|
||||||
|
continue
|
||||||
|
|
||||||
|
character_count: int = len(language_characters)
|
||||||
|
|
||||||
|
character_match_count: int = len(_FREQUENCIES_SET[language] & characters_set)
|
||||||
|
|
||||||
|
ratio: float = character_match_count / character_count
|
||||||
|
|
||||||
|
if ratio >= 0.2:
|
||||||
|
languages.append((language, ratio))
|
||||||
|
|
||||||
|
languages = sorted(languages, key=lambda x: x[1], reverse=True)
|
||||||
|
|
||||||
|
return [compatible_language[0] for compatible_language in languages]
|
||||||
|
|
||||||
|
|
||||||
|
def characters_popularity_compare(
|
||||||
|
language: str, ordered_characters: list[str]
|
||||||
|
) -> float:
|
||||||
|
"""
|
||||||
|
Determine if a ordered characters list (by occurrence from most appearance to rarest) match a particular language.
|
||||||
|
The result is a ratio between 0. (absolutely no correspondence) and 1. (near perfect fit).
|
||||||
|
Beware that is function is not strict on the match in order to ease the detection. (Meaning close match is 1.)
|
||||||
|
"""
|
||||||
|
if language not in FREQUENCIES:
|
||||||
|
raise ValueError(f"{language} not available") # Defensive:
|
||||||
|
|
||||||
|
character_approved_count: int = 0
|
||||||
|
lang_rank: dict[str, int] = _FREQUENCIES_RANK[language]
|
||||||
|
|
||||||
|
ordered_characters_count: int = len(ordered_characters)
|
||||||
|
target_language_characters_count: int = len(FREQUENCIES[language])
|
||||||
|
|
||||||
|
large_alphabet: bool = target_language_characters_count > 26
|
||||||
|
large_alphabet_threshold: float = target_language_characters_count / 3
|
||||||
|
|
||||||
|
expected_projection_ratio: float = (
|
||||||
|
target_language_characters_count / ordered_characters_count
|
||||||
|
)
|
||||||
|
|
||||||
|
# Single pass: characters present in the language vocabulary, as
|
||||||
|
# (language rank, popularity rank) pairs. The scoring below only ever
|
||||||
|
# needs ranks, never the characters themselves.
|
||||||
|
common_lr: list[int] = []
|
||||||
|
common_orr: list[int] = []
|
||||||
|
for popularity_rank, character in enumerate(ordered_characters):
|
||||||
|
language_rank = lang_rank.get(character)
|
||||||
|
if language_rank is not None:
|
||||||
|
common_lr.append(language_rank)
|
||||||
|
common_orr.append(popularity_rank)
|
||||||
|
|
||||||
|
for character_rank_in_language, character_rank in zip(common_lr, common_orr):
|
||||||
|
character_rank_projection: int = int(character_rank * expected_projection_ratio)
|
||||||
|
|
||||||
|
if (
|
||||||
|
not large_alphabet
|
||||||
|
and abs(character_rank_projection - character_rank_in_language) > 4
|
||||||
|
):
|
||||||
|
continue
|
||||||
|
|
||||||
|
if (
|
||||||
|
large_alphabet
|
||||||
|
and abs(character_rank_projection - character_rank_in_language)
|
||||||
|
< large_alphabet_threshold
|
||||||
|
):
|
||||||
|
character_approved_count += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
if character_rank_in_language == 0:
|
||||||
|
# before_match_count is structurally 0 here (no pair can have a
|
||||||
|
# smaller language rank): the historic "before <= 4" acceptance
|
||||||
|
# always holds. (The symmetric "after_len == 0" case is
|
||||||
|
# impossible: language ranks are strictly below the language
|
||||||
|
# character count, hence after_len >= 1.)
|
||||||
|
character_approved_count += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
after_len: int = target_language_characters_count - character_rank_in_language
|
||||||
|
|
||||||
|
# Count how many characters appear "before" in both orderings, and
|
||||||
|
# how many appear "at or after" in both orderings. Both counts grow
|
||||||
|
# monotonically and the approval thresholds
|
||||||
|
# (before / rank >= 0.4 or after / after_len >= 0.4) are known
|
||||||
|
# upfront, expressed below as exact integer comparisons: exit as
|
||||||
|
# soon as one is crossed.
|
||||||
|
before_match_count: int = 0
|
||||||
|
after_match_count: int = 0
|
||||||
|
|
||||||
|
for lr_i, orr_i in zip(common_lr, common_orr):
|
||||||
|
if lr_i < character_rank_in_language:
|
||||||
|
if orr_i < character_rank:
|
||||||
|
before_match_count += 1
|
||||||
|
if 5 * before_match_count >= 2 * character_rank_in_language:
|
||||||
|
character_approved_count += 1
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
if orr_i >= character_rank:
|
||||||
|
after_match_count += 1
|
||||||
|
if 5 * after_match_count >= 2 * after_len:
|
||||||
|
character_approved_count += 1
|
||||||
|
break
|
||||||
|
|
||||||
|
return character_approved_count / len(ordered_characters)
|
||||||
|
|
||||||
|
|
||||||
|
def alpha_unicode_split(decoded_sequence: str) -> list[str]:
|
||||||
|
"""
|
||||||
|
Given a decoded text sequence, return a list of str. Unicode range / alphabet separation.
|
||||||
|
Ex. a text containing English/Latin with a bit a Hebrew will return two items in the resulting list;
|
||||||
|
One containing the latin letters and the other hebrew.
|
||||||
|
"""
|
||||||
|
layers: dict[str, list[str]] = {}
|
||||||
|
|
||||||
|
# Fast path: track single-layer key to skip dict iteration for single-script text.
|
||||||
|
single_layer_key: str | None = None
|
||||||
|
multi_layer: bool = False
|
||||||
|
|
||||||
|
# Cache the last character_range and its resolved layer to avoid repeated
|
||||||
|
# is_suspiciously_successive_range calls for consecutive same-range chars.
|
||||||
|
prev_character_range: str | None = None
|
||||||
|
prev_layer_target: str | None = None
|
||||||
|
|
||||||
|
for character in decoded_sequence:
|
||||||
|
# Reuse the per-codepoint CharInfo cache: info.alpha and info.range
|
||||||
|
# are computed with the very same str.isalpha() / unicode_range()
|
||||||
|
# calls this loop historically made per character occurrence.
|
||||||
|
codepoint: int = ord(character)
|
||||||
|
if codepoint < 128:
|
||||||
|
info = _ASCII_CHAR_INFO[codepoint]
|
||||||
|
else:
|
||||||
|
info = _char_info(character)
|
||||||
|
|
||||||
|
if not info.alpha:
|
||||||
|
continue
|
||||||
|
|
||||||
|
character_range: str | None = info.range
|
||||||
|
|
||||||
|
if character_range is None:
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Fast path: same range as previous character → reuse cached layer target.
|
||||||
|
if character_range == prev_character_range:
|
||||||
|
if prev_layer_target is not None:
|
||||||
|
layers[prev_layer_target].append(character)
|
||||||
|
continue
|
||||||
|
|
||||||
|
layer_target_range: str | None = None
|
||||||
|
|
||||||
|
if multi_layer:
|
||||||
|
for discovered_range in layers:
|
||||||
|
if not is_suspiciously_successive_range(
|
||||||
|
discovered_range, character_range
|
||||||
|
):
|
||||||
|
layer_target_range = discovered_range
|
||||||
|
break
|
||||||
|
elif single_layer_key is not None:
|
||||||
|
if not is_suspiciously_successive_range(single_layer_key, character_range):
|
||||||
|
layer_target_range = single_layer_key
|
||||||
|
|
||||||
|
if layer_target_range is None:
|
||||||
|
layer_target_range = character_range
|
||||||
|
|
||||||
|
if layer_target_range not in layers:
|
||||||
|
layers[layer_target_range] = []
|
||||||
|
if single_layer_key is None:
|
||||||
|
single_layer_key = layer_target_range
|
||||||
|
else:
|
||||||
|
multi_layer = True
|
||||||
|
|
||||||
|
layers[layer_target_range].append(character)
|
||||||
|
|
||||||
|
# Cache for next iteration
|
||||||
|
prev_character_range = character_range
|
||||||
|
prev_layer_target = layer_target_range
|
||||||
|
|
||||||
|
return ["".join(chars).lower() for chars in layers.values()]
|
||||||
|
|
||||||
|
|
||||||
|
def merge_coherence_ratios(results: list[CoherenceMatches]) -> CoherenceMatches:
|
||||||
|
"""
|
||||||
|
This function merge results previously given by the function coherence_ratio.
|
||||||
|
The return type is the same as coherence_ratio.
|
||||||
|
"""
|
||||||
|
per_language_ratios: dict[str, list[float]] = {}
|
||||||
|
for result in results:
|
||||||
|
for sub_result in result:
|
||||||
|
language, ratio = sub_result
|
||||||
|
if language not in per_language_ratios:
|
||||||
|
per_language_ratios[language] = [ratio]
|
||||||
|
continue
|
||||||
|
per_language_ratios[language].append(ratio)
|
||||||
|
|
||||||
|
merge = [
|
||||||
|
(
|
||||||
|
language,
|
||||||
|
round(
|
||||||
|
sum(per_language_ratios[language]) / len(per_language_ratios[language]),
|
||||||
|
4,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
for language in per_language_ratios
|
||||||
|
]
|
||||||
|
|
||||||
|
return sorted(merge, key=lambda x: x[1], reverse=True)
|
||||||
|
|
||||||
|
|
||||||
|
def filter_alt_coherence_matches(results: CoherenceMatches) -> CoherenceMatches:
|
||||||
|
"""
|
||||||
|
We shall NOT return "English—" in CoherenceMatches because it is an alternative
|
||||||
|
of "English". This function only keeps the best match and remove the em-dash in it.
|
||||||
|
"""
|
||||||
|
index_results: dict[str, list[float]] = dict()
|
||||||
|
|
||||||
|
for result in results:
|
||||||
|
language, ratio = result
|
||||||
|
no_em_name: str = language.replace("—", "")
|
||||||
|
|
||||||
|
if no_em_name not in index_results:
|
||||||
|
index_results[no_em_name] = []
|
||||||
|
|
||||||
|
index_results[no_em_name].append(ratio)
|
||||||
|
|
||||||
|
if any(len(index_results[e]) > 1 for e in index_results):
|
||||||
|
filtered_results: CoherenceMatches = []
|
||||||
|
|
||||||
|
for language in index_results:
|
||||||
|
filtered_results.append((language, max(index_results[language])))
|
||||||
|
|
||||||
|
return filtered_results
|
||||||
|
|
||||||
|
return results
|
||||||
|
|
||||||
|
|
||||||
|
def coherence_ratio(
|
||||||
|
decoded_sequence: str, threshold: float = 0.1, lg_inclusion: str | None = None
|
||||||
|
) -> CoherenceMatches:
|
||||||
|
"""
|
||||||
|
Detect ANY language that can be identified in given sequence. The sequence will be analysed by layers.
|
||||||
|
A layer = Character extraction by alphabets/ranges.
|
||||||
|
"""
|
||||||
|
|
||||||
|
results: list[tuple[str, float]] = []
|
||||||
|
ignore_non_latin: bool = False
|
||||||
|
|
||||||
|
sufficient_match_count: int = 0
|
||||||
|
|
||||||
|
lg_inclusion_list = lg_inclusion.split(",") if lg_inclusion is not None else []
|
||||||
|
if "Latin Based" in lg_inclusion_list:
|
||||||
|
ignore_non_latin = True
|
||||||
|
lg_inclusion_list.remove("Latin Based")
|
||||||
|
|
||||||
|
for layer in alpha_unicode_split(decoded_sequence):
|
||||||
|
# Native counting + stable sort reproduce Counter.most_common()
|
||||||
|
# ordering exactly (ties keep first-appearance order) without the
|
||||||
|
# interpreted Counter machinery in the compiled hot path.
|
||||||
|
char_counts: dict[str, int] = {}
|
||||||
|
for layer_character in layer:
|
||||||
|
char_counts[layer_character] = char_counts.get(layer_character, 0) + 1
|
||||||
|
|
||||||
|
character_count: int = len(layer)
|
||||||
|
|
||||||
|
if character_count <= TOO_SMALL_SEQUENCE:
|
||||||
|
continue
|
||||||
|
|
||||||
|
popular_character_ordered: list[str] = [
|
||||||
|
item[0]
|
||||||
|
for item in sorted(
|
||||||
|
char_counts.items(), key=lambda item: item[1], reverse=True
|
||||||
|
)
|
||||||
|
]
|
||||||
|
|
||||||
|
for language in lg_inclusion_list or alphabet_languages(
|
||||||
|
popular_character_ordered, ignore_non_latin
|
||||||
|
):
|
||||||
|
ratio: float = characters_popularity_compare(
|
||||||
|
language, popular_character_ordered
|
||||||
|
)
|
||||||
|
|
||||||
|
if ratio < threshold:
|
||||||
|
continue
|
||||||
|
elif ratio >= 0.8:
|
||||||
|
sufficient_match_count += 1
|
||||||
|
|
||||||
|
results.append((language, round(ratio, 4)))
|
||||||
|
|
||||||
|
if sufficient_match_count >= 3:
|
||||||
|
break
|
||||||
|
|
||||||
|
return sorted(
|
||||||
|
filter_alt_coherence_matches(results), key=lambda x: x[1], reverse=True
|
||||||
|
)
|
||||||
+8
@@ -0,0 +1,8 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from .__main__ import cli_detect, query_yes_no
|
||||||
|
|
||||||
|
__all__ = (
|
||||||
|
"cli_detect",
|
||||||
|
"query_yes_no",
|
||||||
|
)
|
||||||
+363
@@ -0,0 +1,363 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sys
|
||||||
|
import typing
|
||||||
|
from os.path import abspath, basename, dirname, join, realpath
|
||||||
|
from platform import python_version
|
||||||
|
from unicodedata import unidata_version
|
||||||
|
|
||||||
|
import charset_normalizer.md as md_module
|
||||||
|
from charset_normalizer import from_fp
|
||||||
|
from charset_normalizer.models import CliDetectionResult
|
||||||
|
from charset_normalizer.version import __version__
|
||||||
|
|
||||||
|
|
||||||
|
def query_yes_no(question: str, default: str = "yes") -> bool: # Defensive:
|
||||||
|
"""Ask a yes/no question via input() and return the answer as a bool."""
|
||||||
|
prompt = " [Y/n] " if default == "yes" else " [y/N] "
|
||||||
|
|
||||||
|
while True:
|
||||||
|
choice = input(question + prompt).strip().lower()
|
||||||
|
if not choice:
|
||||||
|
return default == "yes"
|
||||||
|
if choice in ("y", "yes"):
|
||||||
|
return True
|
||||||
|
if choice in ("n", "no"):
|
||||||
|
return False
|
||||||
|
print("Please respond with 'y' or 'n'.")
|
||||||
|
|
||||||
|
|
||||||
|
class FileType:
|
||||||
|
"""Factory for creating file object types
|
||||||
|
|
||||||
|
Instances of FileType are typically passed as type= arguments to the
|
||||||
|
ArgumentParser add_argument() method.
|
||||||
|
|
||||||
|
Keyword Arguments:
|
||||||
|
- mode -- A string indicating how the file is to be opened. Accepts the
|
||||||
|
same values as the builtin open() function.
|
||||||
|
- bufsize -- The file's desired buffer size. Accepts the same values as
|
||||||
|
the builtin open() function.
|
||||||
|
- encoding -- The file's encoding. Accepts the same values as the
|
||||||
|
builtin open() function.
|
||||||
|
- errors -- A string indicating how encoding and decoding errors are to
|
||||||
|
be handled. Accepts the same value as the builtin open() function.
|
||||||
|
|
||||||
|
Backported from CPython 3.12
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
mode: str = "r",
|
||||||
|
bufsize: int = -1,
|
||||||
|
encoding: str | None = None,
|
||||||
|
errors: str | None = None,
|
||||||
|
):
|
||||||
|
self._mode = mode
|
||||||
|
self._bufsize = bufsize
|
||||||
|
self._encoding = encoding
|
||||||
|
self._errors = errors
|
||||||
|
|
||||||
|
def __call__(self, string: str) -> typing.IO: # type: ignore[type-arg]
|
||||||
|
# the special argument "-" means sys.std{in,out}
|
||||||
|
if string == "-":
|
||||||
|
if "r" in self._mode:
|
||||||
|
return sys.stdin.buffer if "b" in self._mode else sys.stdin
|
||||||
|
elif any(c in self._mode for c in "wax"):
|
||||||
|
return sys.stdout.buffer if "b" in self._mode else sys.stdout
|
||||||
|
else:
|
||||||
|
msg = f'argument "-" with mode {self._mode}'
|
||||||
|
raise ValueError(msg)
|
||||||
|
|
||||||
|
# all other arguments are used as file names
|
||||||
|
try:
|
||||||
|
return open(string, self._mode, self._bufsize, self._encoding, self._errors)
|
||||||
|
except OSError as e:
|
||||||
|
message = f"can't open '{string}': {e}"
|
||||||
|
raise argparse.ArgumentTypeError(message)
|
||||||
|
|
||||||
|
def __repr__(self) -> str:
|
||||||
|
args = self._mode, self._bufsize
|
||||||
|
kwargs = [("encoding", self._encoding), ("errors", self._errors)]
|
||||||
|
args_str = ", ".join(
|
||||||
|
[repr(arg) for arg in args if arg != -1]
|
||||||
|
+ [f"{kw}={arg!r}" for kw, arg in kwargs if arg is not None]
|
||||||
|
)
|
||||||
|
return f"{type(self).__name__}({args_str})"
|
||||||
|
|
||||||
|
|
||||||
|
def cli_detect(argv: list[str] | None = None) -> int:
|
||||||
|
"""
|
||||||
|
CLI assistant using ARGV and ArgumentParser
|
||||||
|
:param argv:
|
||||||
|
:return: 0 if everything is fine, anything else equal trouble
|
||||||
|
"""
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
description="The Real First Universal Charset Detector. "
|
||||||
|
"Discover originating encoding used on text file. "
|
||||||
|
"Normalize text to unicode."
|
||||||
|
)
|
||||||
|
|
||||||
|
parser.add_argument(
|
||||||
|
"files", type=FileType("rb"), nargs="+", help="File(s) to be analysed"
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"-v",
|
||||||
|
"--verbose",
|
||||||
|
action="store_true",
|
||||||
|
default=False,
|
||||||
|
dest="verbose",
|
||||||
|
help="Display complementary information about file if any. "
|
||||||
|
"Stdout will contain logs about the detection process.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"-a",
|
||||||
|
"--with-alternative",
|
||||||
|
action="store_true",
|
||||||
|
default=False,
|
||||||
|
dest="alternatives",
|
||||||
|
help="Output complementary possibilities if any. Top-level JSON WILL be a list.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"-n",
|
||||||
|
"--normalize",
|
||||||
|
action="store_true",
|
||||||
|
default=False,
|
||||||
|
dest="normalize",
|
||||||
|
help="Permit to normalize input file. If not set, program does not write anything.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"-m",
|
||||||
|
"--minimal",
|
||||||
|
action="store_true",
|
||||||
|
default=False,
|
||||||
|
dest="minimal",
|
||||||
|
help="Only output the charset detected to STDOUT. Disabling JSON output.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"-r",
|
||||||
|
"--replace",
|
||||||
|
action="store_true",
|
||||||
|
default=False,
|
||||||
|
dest="replace",
|
||||||
|
help="Replace file when trying to normalize it instead of creating a new one.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"-f",
|
||||||
|
"--force",
|
||||||
|
action="store_true",
|
||||||
|
default=False,
|
||||||
|
dest="force",
|
||||||
|
help="Replace file without asking if you are sure, use this flag with caution.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"-i",
|
||||||
|
"--no-preemptive",
|
||||||
|
action="store_true",
|
||||||
|
default=False,
|
||||||
|
dest="no_preemptive",
|
||||||
|
help="Disable looking at a charset declaration to hint the detector.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"-t",
|
||||||
|
"--threshold",
|
||||||
|
action="store",
|
||||||
|
default=0.2,
|
||||||
|
type=float,
|
||||||
|
dest="threshold",
|
||||||
|
help="Define a custom maximum amount of noise allowed in decoded content. 0. <= noise <= 1.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--version",
|
||||||
|
action="version",
|
||||||
|
version="Charset-Normalizer {} - Python {} - Unicode {} - SpeedUp {}".format(
|
||||||
|
__version__,
|
||||||
|
python_version(),
|
||||||
|
unidata_version,
|
||||||
|
"OFF" if md_module.__file__.lower().endswith(".py") else "ON",
|
||||||
|
),
|
||||||
|
help="Show version information and exit.",
|
||||||
|
)
|
||||||
|
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
|
||||||
|
if args.replace is True and args.normalize is False:
|
||||||
|
if args.files:
|
||||||
|
for my_file in args.files:
|
||||||
|
my_file.close()
|
||||||
|
print("Use --replace in addition of --normalize only.", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
if args.force is True and args.replace is False:
|
||||||
|
if args.files:
|
||||||
|
for my_file in args.files:
|
||||||
|
my_file.close()
|
||||||
|
print("Use --force in addition of --replace only.", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
if args.threshold < 0.0 or args.threshold > 1.0:
|
||||||
|
if args.files:
|
||||||
|
for my_file in args.files:
|
||||||
|
my_file.close()
|
||||||
|
print("--threshold VALUE should be between 0. AND 1.", file=sys.stderr)
|
||||||
|
return 1
|
||||||
|
|
||||||
|
x_ = []
|
||||||
|
|
||||||
|
for my_file in args.files:
|
||||||
|
matches = from_fp(
|
||||||
|
my_file,
|
||||||
|
threshold=args.threshold,
|
||||||
|
explain=args.verbose,
|
||||||
|
preemptive_behaviour=args.no_preemptive is False,
|
||||||
|
)
|
||||||
|
|
||||||
|
best_guess = matches.best()
|
||||||
|
|
||||||
|
if best_guess is None:
|
||||||
|
print(
|
||||||
|
'Unable to identify originating encoding for "{}". {}'.format(
|
||||||
|
my_file.name,
|
||||||
|
(
|
||||||
|
"Maybe try increasing maximum amount of chaos."
|
||||||
|
if args.threshold < 1.0
|
||||||
|
else ""
|
||||||
|
),
|
||||||
|
),
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
x_.append(
|
||||||
|
CliDetectionResult(
|
||||||
|
abspath(my_file.name),
|
||||||
|
None,
|
||||||
|
[],
|
||||||
|
[],
|
||||||
|
"Unknown",
|
||||||
|
[],
|
||||||
|
False,
|
||||||
|
1.0,
|
||||||
|
0.0,
|
||||||
|
None,
|
||||||
|
True,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
cli_result = CliDetectionResult(
|
||||||
|
abspath(my_file.name),
|
||||||
|
best_guess.encoding,
|
||||||
|
best_guess.encoding_aliases,
|
||||||
|
[
|
||||||
|
cp
|
||||||
|
for cp in best_guess.could_be_from_charset
|
||||||
|
if cp != best_guess.encoding
|
||||||
|
],
|
||||||
|
best_guess.language,
|
||||||
|
best_guess.alphabets,
|
||||||
|
best_guess.bom,
|
||||||
|
best_guess.percent_chaos,
|
||||||
|
best_guess.percent_coherence,
|
||||||
|
None,
|
||||||
|
True,
|
||||||
|
)
|
||||||
|
x_.append(cli_result)
|
||||||
|
|
||||||
|
if len(matches) > 1 and args.alternatives:
|
||||||
|
for el in matches:
|
||||||
|
if el != best_guess:
|
||||||
|
x_.append(
|
||||||
|
CliDetectionResult(
|
||||||
|
abspath(my_file.name),
|
||||||
|
el.encoding,
|
||||||
|
el.encoding_aliases,
|
||||||
|
[
|
||||||
|
cp
|
||||||
|
for cp in el.could_be_from_charset
|
||||||
|
if cp != el.encoding
|
||||||
|
],
|
||||||
|
el.language,
|
||||||
|
el.alphabets,
|
||||||
|
el.bom,
|
||||||
|
el.percent_chaos,
|
||||||
|
el.percent_coherence,
|
||||||
|
None,
|
||||||
|
False,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
if args.normalize is True:
|
||||||
|
if best_guess.encoding.startswith("utf") is True:
|
||||||
|
print(
|
||||||
|
'"{}" file does not need to be normalized, as it already came from unicode.'.format(
|
||||||
|
my_file.name
|
||||||
|
),
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
if my_file.closed is False:
|
||||||
|
my_file.close()
|
||||||
|
continue
|
||||||
|
|
||||||
|
dir_path = dirname(realpath(my_file.name))
|
||||||
|
file_name = basename(realpath(my_file.name))
|
||||||
|
|
||||||
|
o_: list[str] = file_name.split(".")
|
||||||
|
|
||||||
|
if args.replace is False:
|
||||||
|
o_.insert(-1, best_guess.encoding)
|
||||||
|
if my_file.closed is False:
|
||||||
|
my_file.close()
|
||||||
|
elif (
|
||||||
|
args.force is False
|
||||||
|
and query_yes_no(
|
||||||
|
'Are you sure to normalize "{}" by replacing it ?'.format(
|
||||||
|
my_file.name
|
||||||
|
),
|
||||||
|
"no",
|
||||||
|
)
|
||||||
|
is False
|
||||||
|
):
|
||||||
|
if my_file.closed is False:
|
||||||
|
my_file.close()
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
cli_result.unicode_path = join(dir_path, ".".join(o_))
|
||||||
|
|
||||||
|
with open(cli_result.unicode_path, "wb") as fp:
|
||||||
|
fp.write(best_guess.output())
|
||||||
|
except OSError as e: # Defensive:
|
||||||
|
print(str(e), file=sys.stderr)
|
||||||
|
if my_file.closed is False:
|
||||||
|
my_file.close()
|
||||||
|
return 2
|
||||||
|
|
||||||
|
if my_file.closed is False:
|
||||||
|
my_file.close()
|
||||||
|
|
||||||
|
if args.minimal is False:
|
||||||
|
from json import dumps
|
||||||
|
|
||||||
|
print(
|
||||||
|
dumps(
|
||||||
|
[el.__dict__ for el in x_] if len(x_) > 1 else x_[0].__dict__,
|
||||||
|
ensure_ascii=True,
|
||||||
|
indent=4,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
for my_file in args.files:
|
||||||
|
print(
|
||||||
|
", ".join(
|
||||||
|
[
|
||||||
|
el.encoding or "undefined"
|
||||||
|
for el in x_
|
||||||
|
if el.path == abspath(my_file.name)
|
||||||
|
]
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__": # Defensive:
|
||||||
|
cli_detect()
|
||||||
+2496
File diff suppressed because it is too large
Load Diff
+79
@@ -0,0 +1,79 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import TYPE_CHECKING, Any
|
||||||
|
from warnings import warn
|
||||||
|
|
||||||
|
from .api import from_bytes
|
||||||
|
from .constant import CHARDET_CORRESPONDENCE, TOO_SMALL_SEQUENCE
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from typing import TypedDict
|
||||||
|
|
||||||
|
class ResultDict(TypedDict):
|
||||||
|
encoding: str | None
|
||||||
|
language: str
|
||||||
|
confidence: float | None
|
||||||
|
|
||||||
|
|
||||||
|
def detect(
|
||||||
|
byte_str: bytes, should_rename_legacy: bool = False, **kwargs: Any
|
||||||
|
) -> ResultDict:
|
||||||
|
"""
|
||||||
|
chardet legacy method
|
||||||
|
Detect the encoding of the given byte string. It should be mostly backward-compatible.
|
||||||
|
Encoding name will match Chardet own writing whenever possible. (Not on encoding name unsupported by it)
|
||||||
|
This function is deprecated and should be used to migrate your project easily, consult the documentation for
|
||||||
|
further information. Not planned for removal.
|
||||||
|
|
||||||
|
:param byte_str: The byte sequence to examine.
|
||||||
|
:param should_rename_legacy: Should we rename legacy encodings
|
||||||
|
to their more modern equivalents?
|
||||||
|
"""
|
||||||
|
if len(kwargs):
|
||||||
|
warn(
|
||||||
|
f"charset-normalizer disregard arguments '{','.join(list(kwargs.keys()))}' in legacy function detect()"
|
||||||
|
)
|
||||||
|
|
||||||
|
if not isinstance(byte_str, (bytearray, bytes)):
|
||||||
|
raise TypeError( # pragma: nocover
|
||||||
|
f"Expected object of type bytes or bytearray, got: {type(byte_str)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
if isinstance(byte_str, bytearray):
|
||||||
|
byte_str = bytes(byte_str)
|
||||||
|
|
||||||
|
r = from_bytes(byte_str).best()
|
||||||
|
|
||||||
|
encoding = r.encoding if r is not None else None
|
||||||
|
language = r.language if r is not None and r.language != "Unknown" else ""
|
||||||
|
confidence = 1.0 - r.chaos if r is not None else None
|
||||||
|
|
||||||
|
# automatically lower confidence
|
||||||
|
# on small bytes samples.
|
||||||
|
# https://github.com/jawah/charset_normalizer/issues/391
|
||||||
|
if (
|
||||||
|
confidence is not None
|
||||||
|
and confidence >= 0.9
|
||||||
|
and encoding
|
||||||
|
not in {
|
||||||
|
"utf_8",
|
||||||
|
"ascii",
|
||||||
|
}
|
||||||
|
and not r.bom # type: ignore[union-attr]
|
||||||
|
and len(byte_str) < TOO_SMALL_SEQUENCE
|
||||||
|
):
|
||||||
|
confidence -= 0.2
|
||||||
|
|
||||||
|
# Note: CharsetNormalizer does not return 'UTF-8-SIG' as the sig get stripped in the detection/normalization process
|
||||||
|
# but chardet does return 'utf-8-sig' and it is a valid codec name.
|
||||||
|
if r is not None and encoding == "utf_8" and r.bom:
|
||||||
|
encoding += "_sig"
|
||||||
|
|
||||||
|
if not should_rename_legacy and encoding in CHARDET_CORRESPONDENCE:
|
||||||
|
encoding = CHARDET_CORRESPONDENCE[encoding]
|
||||||
|
|
||||||
|
return {
|
||||||
|
"encoding": encoding,
|
||||||
|
"language": language,
|
||||||
|
"confidence": confidence,
|
||||||
|
}
|
||||||
+1063
File diff suppressed because it is too large
Load Diff
+387
@@ -0,0 +1,387 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from encodings.aliases import aliases
|
||||||
|
from re import sub
|
||||||
|
from typing import Any, Iterator, List, Tuple
|
||||||
|
|
||||||
|
from .constant import RE_POSSIBLE_ENCODING_INDICATION, TOO_BIG_SEQUENCE
|
||||||
|
from .utils import iana_name, is_multi_byte_encoding, unicode_range
|
||||||
|
|
||||||
|
|
||||||
|
class CharsetMatch:
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
payload: bytes | bytearray,
|
||||||
|
guessed_encoding: str,
|
||||||
|
mean_mess_ratio: float,
|
||||||
|
has_sig_or_bom: bool,
|
||||||
|
languages: CoherenceMatches,
|
||||||
|
decoded_payload: str | None = None,
|
||||||
|
preemptive_declaration: str | None = None,
|
||||||
|
):
|
||||||
|
self._payload: bytes | bytearray = payload
|
||||||
|
|
||||||
|
self._encoding: str = guessed_encoding
|
||||||
|
self._mean_mess_ratio: float = mean_mess_ratio
|
||||||
|
self._languages: CoherenceMatches = languages
|
||||||
|
self._has_sig_or_bom: bool = has_sig_or_bom
|
||||||
|
self._unicode_ranges: list[str] | None = None
|
||||||
|
|
||||||
|
self._leaves: list[CharsetMatch] = []
|
||||||
|
self._mean_coherence_ratio: float = 0.0
|
||||||
|
|
||||||
|
self._output_payload: bytes | None = None
|
||||||
|
self._output_encoding: str | None = None
|
||||||
|
|
||||||
|
self._string: str | None = decoded_payload
|
||||||
|
|
||||||
|
self._preemptive_declaration: str | None = preemptive_declaration
|
||||||
|
|
||||||
|
def __eq__(self, other: object) -> bool:
|
||||||
|
if not isinstance(other, CharsetMatch):
|
||||||
|
if isinstance(other, str):
|
||||||
|
# Use non-strict iana_name so an operand that is not a known
|
||||||
|
# encoding alias compares unequal instead of raising, keeping
|
||||||
|
# __eq__ total as the data model requires (mirrors the lookup
|
||||||
|
# in CharsetMatches.__getitem__).
|
||||||
|
return iana_name(other, False) == self.encoding
|
||||||
|
return False
|
||||||
|
return self.encoding == other.encoding and self.fingerprint == other.fingerprint
|
||||||
|
|
||||||
|
def __lt__(self, other: object) -> bool:
|
||||||
|
"""
|
||||||
|
Implemented to make sorted available upon CharsetMatches items.
|
||||||
|
"""
|
||||||
|
if not isinstance(other, CharsetMatch):
|
||||||
|
raise ValueError
|
||||||
|
|
||||||
|
chaos_difference: float = abs(self.chaos - other.chaos)
|
||||||
|
coherence_difference: float = abs(self.coherence - other.coherence)
|
||||||
|
|
||||||
|
# Below 0.5% difference --> Use Coherence
|
||||||
|
if chaos_difference < 0.005 and coherence_difference > 0.02:
|
||||||
|
return self.coherence > other.coherence
|
||||||
|
elif chaos_difference < 0.005 and coherence_difference <= 0.02:
|
||||||
|
# When having a difficult decision, use the result that decoded as many multi-byte as possible.
|
||||||
|
# preserve RAM usage!
|
||||||
|
if len(self._payload) >= TOO_BIG_SEQUENCE:
|
||||||
|
return self.chaos < other.chaos
|
||||||
|
return self.multi_byte_usage > other.multi_byte_usage
|
||||||
|
|
||||||
|
return self.chaos < other.chaos
|
||||||
|
|
||||||
|
@property
|
||||||
|
def multi_byte_usage(self) -> float:
|
||||||
|
# Empty payloads are valid (from_bytes(b"") returns a match); avoid /0.
|
||||||
|
raw_len = len(self.raw)
|
||||||
|
if raw_len == 0:
|
||||||
|
return 0.0
|
||||||
|
return 1.0 - (len(str(self)) / raw_len)
|
||||||
|
|
||||||
|
def __str__(self) -> str:
|
||||||
|
# Lazy Str Loading
|
||||||
|
if self._string is None:
|
||||||
|
self._string = str(self._payload, self._encoding, "strict")
|
||||||
|
# UTF-7 BOM is encoded in modified Base64 whose byte boundary
|
||||||
|
# can overlap with the next character, so raw-byte stripping
|
||||||
|
# is unreliable. Strip the decoded BOM character instead.
|
||||||
|
if (
|
||||||
|
self._has_sig_or_bom
|
||||||
|
and self._encoding == "utf_7"
|
||||||
|
and self._string
|
||||||
|
and self._string[0] == "\ufeff"
|
||||||
|
):
|
||||||
|
self._string = self._string[1:]
|
||||||
|
return self._string
|
||||||
|
|
||||||
|
def __repr__(self) -> str:
|
||||||
|
return f"<CharsetMatch '{self.encoding}' fp({self.fingerprint})>"
|
||||||
|
|
||||||
|
def add_submatch(self, other: CharsetMatch) -> None:
|
||||||
|
if not isinstance(other, CharsetMatch) or other == self:
|
||||||
|
raise ValueError(
|
||||||
|
"Unable to add instance <{}> as a submatch of a CharsetMatch".format(
|
||||||
|
other.__class__
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
other._string = None # Unload RAM usage; dirty trick.
|
||||||
|
self._leaves.append(other)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def encoding(self) -> str:
|
||||||
|
return self._encoding
|
||||||
|
|
||||||
|
@property
|
||||||
|
def encoding_aliases(self) -> list[str]:
|
||||||
|
"""
|
||||||
|
Encoding name are known by many name, using this could help when searching for IBM855 when it's listed as CP855.
|
||||||
|
"""
|
||||||
|
also_known_as: list[str] = []
|
||||||
|
for u, p in aliases.items():
|
||||||
|
if self.encoding == u:
|
||||||
|
also_known_as.append(p)
|
||||||
|
elif self.encoding == p:
|
||||||
|
also_known_as.append(u)
|
||||||
|
return also_known_as
|
||||||
|
|
||||||
|
@property
|
||||||
|
def bom(self) -> bool:
|
||||||
|
return self._has_sig_or_bom
|
||||||
|
|
||||||
|
@property
|
||||||
|
def byte_order_mark(self) -> bool:
|
||||||
|
return self._has_sig_or_bom
|
||||||
|
|
||||||
|
@property
|
||||||
|
def languages(self) -> list[str]:
|
||||||
|
"""
|
||||||
|
Return the complete list of possible languages found in decoded sequence.
|
||||||
|
Usually not really useful. Returned list may be empty even if 'language' property return something != 'Unknown'.
|
||||||
|
"""
|
||||||
|
return [e[0] for e in self._languages]
|
||||||
|
|
||||||
|
@property
|
||||||
|
def language(self) -> str:
|
||||||
|
"""
|
||||||
|
Most probable language found in decoded sequence. If none were detected or inferred, the property will return
|
||||||
|
"Unknown".
|
||||||
|
"""
|
||||||
|
if not self._languages:
|
||||||
|
# Trying to infer the language based on the given encoding
|
||||||
|
# Its either English or we should not pronounce ourselves in certain cases.
|
||||||
|
if "ascii" in self.could_be_from_charset:
|
||||||
|
return "English"
|
||||||
|
|
||||||
|
# doing it there to avoid circular import
|
||||||
|
from charset_normalizer.cd import encoding_languages, mb_encoding_languages
|
||||||
|
|
||||||
|
languages = (
|
||||||
|
mb_encoding_languages(self.encoding)
|
||||||
|
if is_multi_byte_encoding(self.encoding)
|
||||||
|
else encoding_languages(self.encoding)
|
||||||
|
)
|
||||||
|
|
||||||
|
if len(languages) == 0 or "Latin Based" in languages:
|
||||||
|
return "Unknown"
|
||||||
|
|
||||||
|
return languages[0]
|
||||||
|
|
||||||
|
return self._languages[0][0]
|
||||||
|
|
||||||
|
@property
|
||||||
|
def chaos(self) -> float:
|
||||||
|
return self._mean_mess_ratio
|
||||||
|
|
||||||
|
@property
|
||||||
|
def coherence(self) -> float:
|
||||||
|
if not self._languages:
|
||||||
|
return 0.0
|
||||||
|
return self._languages[0][1]
|
||||||
|
|
||||||
|
@property
|
||||||
|
def percent_chaos(self) -> float:
|
||||||
|
return round(self.chaos * 100, ndigits=3)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def percent_coherence(self) -> float:
|
||||||
|
return round(self.coherence * 100, ndigits=3)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def raw(self) -> bytes | bytearray:
|
||||||
|
"""
|
||||||
|
Original untouched bytes.
|
||||||
|
"""
|
||||||
|
return self._payload
|
||||||
|
|
||||||
|
@property
|
||||||
|
def submatch(self) -> list[CharsetMatch]:
|
||||||
|
return self._leaves
|
||||||
|
|
||||||
|
@property
|
||||||
|
def has_submatch(self) -> bool:
|
||||||
|
return len(self._leaves) > 0
|
||||||
|
|
||||||
|
@property
|
||||||
|
def alphabets(self) -> list[str]:
|
||||||
|
if self._unicode_ranges is not None:
|
||||||
|
return self._unicode_ranges
|
||||||
|
# list detected ranges
|
||||||
|
detected_ranges: list[str | None] = [unicode_range(char) for char in str(self)]
|
||||||
|
# filter and sort
|
||||||
|
self._unicode_ranges = sorted(list({r for r in detected_ranges if r}))
|
||||||
|
return self._unicode_ranges
|
||||||
|
|
||||||
|
@property
|
||||||
|
def could_be_from_charset(self) -> list[str]:
|
||||||
|
"""
|
||||||
|
The complete list of encoding that output the exact SAME str result and therefore could be the originating
|
||||||
|
encoding.
|
||||||
|
This list does include the encoding available in property 'encoding'.
|
||||||
|
"""
|
||||||
|
return [self._encoding] + [m.encoding for m in self._leaves]
|
||||||
|
|
||||||
|
def output(self, encoding: str = "utf_8") -> bytes:
|
||||||
|
"""
|
||||||
|
Method to get re-encoded bytes payload using given target encoding. Default to UTF-8.
|
||||||
|
Any errors will be simply ignored by the encoder NOT replaced.
|
||||||
|
"""
|
||||||
|
if self._output_encoding is None or self._output_encoding != encoding:
|
||||||
|
self._output_encoding = encoding
|
||||||
|
decoded_string = str(self)
|
||||||
|
if (
|
||||||
|
self._preemptive_declaration is not None
|
||||||
|
and self._preemptive_declaration.lower()
|
||||||
|
not in ["utf-8", "utf8", "utf_8"]
|
||||||
|
):
|
||||||
|
patched_header = sub(
|
||||||
|
RE_POSSIBLE_ENCODING_INDICATION,
|
||||||
|
lambda m: m.string[m.span()[0] : m.span()[1]].replace(
|
||||||
|
m.groups()[0],
|
||||||
|
iana_name(self._output_encoding).replace("_", "-"), # type: ignore[arg-type]
|
||||||
|
),
|
||||||
|
decoded_string[:8192],
|
||||||
|
count=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
decoded_string = patched_header + decoded_string[8192:]
|
||||||
|
|
||||||
|
self._output_payload = decoded_string.encode(encoding, "replace")
|
||||||
|
|
||||||
|
return self._output_payload # type: ignore
|
||||||
|
|
||||||
|
@property
|
||||||
|
def fingerprint(self) -> int:
|
||||||
|
"""
|
||||||
|
Retrieve a hash fingerprint of the decoded payload, used for deduplication.
|
||||||
|
"""
|
||||||
|
return hash(str(self))
|
||||||
|
|
||||||
|
|
||||||
|
class CharsetMatches:
|
||||||
|
"""
|
||||||
|
Container with every CharsetMatch items ordered by default from most probable to the less one.
|
||||||
|
Act like a list(iterable) but does not implements all related methods.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, results: list[CharsetMatch] | None = None):
|
||||||
|
self._results: list[CharsetMatch] = sorted(results) if results else []
|
||||||
|
self._is_sorted: bool = True
|
||||||
|
|
||||||
|
def _ensure_sorted(self) -> None:
|
||||||
|
if not self._is_sorted:
|
||||||
|
self._results.sort()
|
||||||
|
self._is_sorted = True
|
||||||
|
|
||||||
|
def __iter__(self) -> Iterator[CharsetMatch]:
|
||||||
|
self._ensure_sorted()
|
||||||
|
yield from self._results
|
||||||
|
|
||||||
|
def __getitem__(self, item: int | str) -> CharsetMatch:
|
||||||
|
"""
|
||||||
|
Retrieve a single item either by its position or encoding name (alias may be used here).
|
||||||
|
Raise KeyError upon invalid index or encoding not present in results.
|
||||||
|
"""
|
||||||
|
if isinstance(item, int):
|
||||||
|
self._ensure_sorted()
|
||||||
|
return self._results[item]
|
||||||
|
if isinstance(item, str):
|
||||||
|
item = iana_name(item, False)
|
||||||
|
for result in self._results:
|
||||||
|
if item in result.could_be_from_charset:
|
||||||
|
return result
|
||||||
|
raise KeyError
|
||||||
|
|
||||||
|
def __len__(self) -> int:
|
||||||
|
return len(self._results)
|
||||||
|
|
||||||
|
def __bool__(self) -> bool:
|
||||||
|
return len(self._results) > 0
|
||||||
|
|
||||||
|
def append(self, item: CharsetMatch) -> None:
|
||||||
|
"""
|
||||||
|
Insert a single match. Will be inserted accordingly to preserve sort.
|
||||||
|
Can be inserted as a submatch.
|
||||||
|
"""
|
||||||
|
if not isinstance(item, CharsetMatch):
|
||||||
|
raise ValueError(
|
||||||
|
"Cannot append instance '{}' to CharsetMatches".format(
|
||||||
|
str(item.__class__)
|
||||||
|
)
|
||||||
|
)
|
||||||
|
# We should disable the submatch factoring when the input file is too heavy (conserve RAM usage)
|
||||||
|
if len(item.raw) < TOO_BIG_SEQUENCE:
|
||||||
|
for match in self._results:
|
||||||
|
if match.fingerprint == item.fingerprint and match.chaos == item.chaos:
|
||||||
|
match.add_submatch(item)
|
||||||
|
return
|
||||||
|
self._results.append(item)
|
||||||
|
self._is_sorted = False
|
||||||
|
|
||||||
|
def best(self) -> CharsetMatch | None:
|
||||||
|
"""
|
||||||
|
Simply return the first match. Strict equivalent to matches[0].
|
||||||
|
"""
|
||||||
|
if not self._results:
|
||||||
|
return None
|
||||||
|
self._ensure_sorted()
|
||||||
|
return self._results[0]
|
||||||
|
|
||||||
|
def first(self) -> CharsetMatch | None:
|
||||||
|
"""
|
||||||
|
Redundant method, call the method best(). Kept for BC reasons.
|
||||||
|
"""
|
||||||
|
return self.best()
|
||||||
|
|
||||||
|
|
||||||
|
CoherenceMatch = Tuple[str, float]
|
||||||
|
CoherenceMatches = List[CoherenceMatch]
|
||||||
|
|
||||||
|
|
||||||
|
class CliDetectionResult:
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
path: str,
|
||||||
|
encoding: str | None,
|
||||||
|
encoding_aliases: list[str],
|
||||||
|
alternative_encodings: list[str],
|
||||||
|
language: str,
|
||||||
|
alphabets: list[str],
|
||||||
|
has_sig_or_bom: bool,
|
||||||
|
chaos: float,
|
||||||
|
coherence: float,
|
||||||
|
unicode_path: str | None,
|
||||||
|
is_preferred: bool,
|
||||||
|
):
|
||||||
|
self.path: str = path
|
||||||
|
self.unicode_path: str | None = unicode_path
|
||||||
|
self.encoding: str | None = encoding
|
||||||
|
self.encoding_aliases: list[str] = encoding_aliases
|
||||||
|
self.alternative_encodings: list[str] = alternative_encodings
|
||||||
|
self.language: str = language
|
||||||
|
self.alphabets: list[str] = alphabets
|
||||||
|
self.has_sig_or_bom: bool = has_sig_or_bom
|
||||||
|
self.chaos: float = chaos
|
||||||
|
self.coherence: float = coherence
|
||||||
|
self.is_preferred: bool = is_preferred
|
||||||
|
|
||||||
|
@property
|
||||||
|
def __dict__(self) -> dict[str, Any]: # type: ignore
|
||||||
|
return {
|
||||||
|
"path": self.path,
|
||||||
|
"encoding": self.encoding,
|
||||||
|
"encoding_aliases": self.encoding_aliases,
|
||||||
|
"alternative_encodings": self.alternative_encodings,
|
||||||
|
"language": self.language,
|
||||||
|
"alphabets": self.alphabets,
|
||||||
|
"has_sig_or_bom": self.has_sig_or_bom,
|
||||||
|
"chaos": self.chaos,
|
||||||
|
"coherence": self.coherence,
|
||||||
|
"unicode_path": self.unicode_path,
|
||||||
|
"is_preferred": self.is_preferred,
|
||||||
|
}
|
||||||
|
|
||||||
|
def to_json(self) -> str:
|
||||||
|
from json import dumps
|
||||||
|
|
||||||
|
return dumps(self.__dict__, ensure_ascii=True, indent=4)
|
||||||
+452
@@ -0,0 +1,452 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib
|
||||||
|
import logging
|
||||||
|
import unicodedata
|
||||||
|
from bisect import bisect_right
|
||||||
|
from codecs import IncrementalDecoder
|
||||||
|
from functools import lru_cache
|
||||||
|
from typing import Generator
|
||||||
|
|
||||||
|
from .constant import (
|
||||||
|
ENCODING_MARKS,
|
||||||
|
IANA_SUPPORTED_SIMILAR,
|
||||||
|
RE_POSSIBLE_ENCODING_INDICATION,
|
||||||
|
UNICODE_RANGES_COMBINED,
|
||||||
|
_SECONDARY_RANGE_NAMES,
|
||||||
|
COMMON_CJK_CHARACTERS,
|
||||||
|
_LATIN,
|
||||||
|
_CJK,
|
||||||
|
_HANGUL,
|
||||||
|
_KATAKANA,
|
||||||
|
_HIRAGANA,
|
||||||
|
_HALFWIDTH_KATAKANA,
|
||||||
|
_THAI,
|
||||||
|
_ARABIC,
|
||||||
|
_ARABIC_ISOLATED_FORM,
|
||||||
|
_LIGATURE,
|
||||||
|
_SUPERSCRIPT,
|
||||||
|
_SENTENCE_OPEN_PUNCTUATION,
|
||||||
|
_ACCENT_KEYWORDS,
|
||||||
|
_ACCENTUATED,
|
||||||
|
_KNOWN_MB_DECODERS,
|
||||||
|
_KNOWN_MB_CLASSES,
|
||||||
|
_IANA_NAMES,
|
||||||
|
_MULTIBYTE_SEARCH_RADIUS,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _character_flags(character: str) -> int:
|
||||||
|
"""Compute all name-based classification flags with a single unicodedata.name() call."""
|
||||||
|
try:
|
||||||
|
desc: str = unicodedata.name(character)
|
||||||
|
except ValueError:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
flags: int = 0
|
||||||
|
|
||||||
|
if "LATIN" in desc:
|
||||||
|
flags |= _LATIN
|
||||||
|
if "CJK" in desc:
|
||||||
|
flags |= _CJK
|
||||||
|
if "HANGUL" in desc:
|
||||||
|
flags |= _HANGUL
|
||||||
|
if "KATAKANA" in desc:
|
||||||
|
flags |= _KATAKANA
|
||||||
|
if "HALFWIDTH" in desc:
|
||||||
|
flags |= _HALFWIDTH_KATAKANA
|
||||||
|
if "HIRAGANA" in desc:
|
||||||
|
flags |= _HIRAGANA
|
||||||
|
if "THAI" in desc:
|
||||||
|
flags |= _THAI
|
||||||
|
if "ARABIC" in desc:
|
||||||
|
flags |= _ARABIC
|
||||||
|
if "ISOLATED FORM" in desc:
|
||||||
|
flags |= _ARABIC_ISOLATED_FORM
|
||||||
|
if "LIGATURE" in desc or desc.endswith("LETTER AE"):
|
||||||
|
flags |= _LIGATURE
|
||||||
|
if "SUPERSCRIPT" in desc:
|
||||||
|
flags |= _SUPERSCRIPT
|
||||||
|
if desc in {"INVERTED QUESTION MARK", "INVERTED EXCLAMATION MARK"}:
|
||||||
|
flags |= _SENTENCE_OPEN_PUNCTUATION
|
||||||
|
|
||||||
|
for kw in _ACCENT_KEYWORDS:
|
||||||
|
if kw in desc:
|
||||||
|
flags |= _ACCENTUATED
|
||||||
|
break
|
||||||
|
|
||||||
|
return flags
|
||||||
|
|
||||||
|
|
||||||
|
def is_accentuated(character: str) -> bool:
|
||||||
|
return bool(_character_flags(character) & _ACCENTUATED)
|
||||||
|
|
||||||
|
|
||||||
|
def remove_accent(character: str) -> str:
|
||||||
|
decomposed: str = unicodedata.decomposition(character)
|
||||||
|
if not decomposed:
|
||||||
|
return character
|
||||||
|
|
||||||
|
codes: list[str] = decomposed.split(" ")
|
||||||
|
|
||||||
|
return chr(int(codes[0], 16))
|
||||||
|
|
||||||
|
|
||||||
|
# Pre-built sorted lookup table for O(log n) binary search in unicode_range().
|
||||||
|
# Each entry is (range_start, range_end_exclusive, range_name).
|
||||||
|
_UNICODE_RANGES_SORTED: list[tuple[int, int, str]] = sorted(
|
||||||
|
(ord_range.start, ord_range.stop, name)
|
||||||
|
for name, ord_range in UNICODE_RANGES_COMBINED.items()
|
||||||
|
)
|
||||||
|
_UNICODE_RANGE_STARTS: list[int] = [e[0] for e in _UNICODE_RANGES_SORTED]
|
||||||
|
|
||||||
|
|
||||||
|
def unicode_range(character: str) -> str | None:
|
||||||
|
"""
|
||||||
|
Retrieve the Unicode range official name from a single character.
|
||||||
|
"""
|
||||||
|
character_ord: int = ord(character)
|
||||||
|
|
||||||
|
if character_ord < 32:
|
||||||
|
return "Control character"
|
||||||
|
if character_ord < 128:
|
||||||
|
return "Basic Latin"
|
||||||
|
|
||||||
|
# Binary search: find the rightmost range whose start <= character_ord
|
||||||
|
idx = bisect_right(_UNICODE_RANGE_STARTS, character_ord) - 1
|
||||||
|
if idx >= 0:
|
||||||
|
start, stop, name = _UNICODE_RANGES_SORTED[idx]
|
||||||
|
if character_ord < stop:
|
||||||
|
return name
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def is_latin(character: str) -> bool:
|
||||||
|
return bool(_character_flags(character) & _LATIN)
|
||||||
|
|
||||||
|
|
||||||
|
def is_punctuation(character: str) -> bool:
|
||||||
|
character_category: str = unicodedata.category(character)
|
||||||
|
|
||||||
|
if "P" in character_category:
|
||||||
|
return True
|
||||||
|
|
||||||
|
character_range: str | None = unicode_range(character)
|
||||||
|
|
||||||
|
if character_range is None:
|
||||||
|
return False
|
||||||
|
|
||||||
|
return "Punctuation" in character_range
|
||||||
|
|
||||||
|
|
||||||
|
def is_symbol(character: str) -> bool:
|
||||||
|
character_category: str = unicodedata.category(character)
|
||||||
|
|
||||||
|
if "S" in character_category or "N" in character_category:
|
||||||
|
return True
|
||||||
|
|
||||||
|
character_range: str | None = unicode_range(character)
|
||||||
|
|
||||||
|
if character_range is None:
|
||||||
|
return False
|
||||||
|
|
||||||
|
return "Forms" in character_range and character_category != "Lo"
|
||||||
|
|
||||||
|
|
||||||
|
def is_emoticon(character: str) -> bool:
|
||||||
|
character_range: str | None = unicode_range(character)
|
||||||
|
|
||||||
|
if character_range is None:
|
||||||
|
return False
|
||||||
|
|
||||||
|
return "Emoticons" in character_range or "Pictographs" in character_range
|
||||||
|
|
||||||
|
|
||||||
|
def is_separator(character: str) -> bool:
|
||||||
|
if character.isspace() or character in {"|", "+", "<", ">"}:
|
||||||
|
return True
|
||||||
|
|
||||||
|
character_category: str = unicodedata.category(character)
|
||||||
|
|
||||||
|
return "Z" in character_category or character_category in {"Po", "Pd", "Pc"}
|
||||||
|
|
||||||
|
|
||||||
|
def is_case_variable(character: str) -> bool:
|
||||||
|
return character.islower() != character.isupper()
|
||||||
|
|
||||||
|
|
||||||
|
def is_cjk(character: str) -> bool:
|
||||||
|
return bool(_character_flags(character) & _CJK)
|
||||||
|
|
||||||
|
|
||||||
|
def is_hiragana(character: str) -> bool:
|
||||||
|
return bool(_character_flags(character) & _HIRAGANA)
|
||||||
|
|
||||||
|
|
||||||
|
def is_katakana(character: str) -> bool:
|
||||||
|
return bool(_character_flags(character) & _KATAKANA)
|
||||||
|
|
||||||
|
|
||||||
|
def is_hangul(character: str) -> bool:
|
||||||
|
return bool(_character_flags(character) & _HANGUL)
|
||||||
|
|
||||||
|
|
||||||
|
def is_thai(character: str) -> bool:
|
||||||
|
return bool(_character_flags(character) & _THAI)
|
||||||
|
|
||||||
|
|
||||||
|
def is_arabic(character: str) -> bool:
|
||||||
|
return bool(_character_flags(character) & _ARABIC)
|
||||||
|
|
||||||
|
|
||||||
|
def is_arabic_isolated_form(character: str) -> bool:
|
||||||
|
return bool(_character_flags(character) & _ARABIC_ISOLATED_FORM)
|
||||||
|
|
||||||
|
|
||||||
|
def is_cjk_uncommon(character: str) -> bool:
|
||||||
|
return character not in COMMON_CJK_CHARACTERS
|
||||||
|
|
||||||
|
|
||||||
|
def is_unicode_range_secondary(range_name: str) -> bool:
|
||||||
|
return range_name in _SECONDARY_RANGE_NAMES
|
||||||
|
|
||||||
|
|
||||||
|
def is_unprintable(character: str) -> bool:
|
||||||
|
return (
|
||||||
|
not character.isspace() # includes \n \t \r \v
|
||||||
|
and not character.isprintable()
|
||||||
|
and character != "\x1a" # Why? Its the ASCII substitute character.
|
||||||
|
and character != "\ufeff" # bug discovered in Python,
|
||||||
|
# Zero Width No-Break Space located in Arabic Presentation Forms-B, Unicode 1.1 not acknowledged as space.
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def any_specified_encoding(
|
||||||
|
sequence: bytes | bytearray, search_zone: int = 8192
|
||||||
|
) -> str | None:
|
||||||
|
"""
|
||||||
|
Extract using ASCII-only decoder any specified encoding in the first n-bytes.
|
||||||
|
"""
|
||||||
|
if not isinstance(sequence, (bytes, bytearray)):
|
||||||
|
raise TypeError
|
||||||
|
|
||||||
|
seq_len: int = len(sequence)
|
||||||
|
|
||||||
|
# Cheap literal pre-filter.
|
||||||
|
search_bytes = sequence[: min(seq_len, search_zone)]
|
||||||
|
lowered_bytes = search_bytes.lower()
|
||||||
|
if b"coding" not in lowered_bytes and b"charset" not in lowered_bytes:
|
||||||
|
return None
|
||||||
|
|
||||||
|
decoded_zone: str = search_bytes.decode("ascii", errors="ignore")
|
||||||
|
|
||||||
|
for match in RE_POSSIBLE_ENCODING_INDICATION.finditer(decoded_zone):
|
||||||
|
specified_encoding = match.group(1).lower().replace("-", "_")
|
||||||
|
encoding_iana = _IANA_NAMES.get(specified_encoding)
|
||||||
|
if encoding_iana is not None:
|
||||||
|
return encoding_iana
|
||||||
|
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
@lru_cache(maxsize=None)
|
||||||
|
def is_multi_byte_encoding(name: str) -> bool:
|
||||||
|
"""
|
||||||
|
Verify is a specific encoding is a multi byte one based on it IANA name
|
||||||
|
"""
|
||||||
|
if name in _KNOWN_MB_DECODERS:
|
||||||
|
return True
|
||||||
|
|
||||||
|
# Besides the Unicode family above, every multibyte codec shipped with
|
||||||
|
# Python is implemented by _multibytecodec through exactly one of the six
|
||||||
|
# cjkcodecs providers below. Probing those providers directly (getcodec)
|
||||||
|
# classifies a name without importing its "encodings.<name>" module:
|
||||||
|
# classifying the whole IANA_SUPPORTED list would otherwise import many
|
||||||
|
# modules and dominate "import charset_normalizer" wall time.
|
||||||
|
# see https://github.com/jawah/charset_normalizer/issues/742
|
||||||
|
for provider in _KNOWN_MB_CLASSES:
|
||||||
|
try:
|
||||||
|
importlib.import_module(provider).getcodec(name) # type: ignore[attr-defined]
|
||||||
|
except (ImportError, AttributeError, LookupError): # Defensive: edge cases
|
||||||
|
continue
|
||||||
|
return True
|
||||||
|
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def identify_sig_or_bom(sequence: bytes | bytearray) -> tuple[str | None, bytes]:
|
||||||
|
"""
|
||||||
|
Identify and extract SIG/BOM in given sequence.
|
||||||
|
"""
|
||||||
|
|
||||||
|
for iana_encoding in ENCODING_MARKS:
|
||||||
|
marks: bytes | list[bytes] = ENCODING_MARKS[iana_encoding]
|
||||||
|
|
||||||
|
if isinstance(marks, bytes):
|
||||||
|
marks = [marks]
|
||||||
|
|
||||||
|
for mark in marks:
|
||||||
|
if sequence.startswith(mark):
|
||||||
|
return iana_encoding, mark
|
||||||
|
|
||||||
|
return None, b""
|
||||||
|
|
||||||
|
|
||||||
|
def should_strip_sig_or_bom(iana_encoding: str) -> bool:
|
||||||
|
return iana_encoding not in {"utf_16", "utf_32"}
|
||||||
|
|
||||||
|
|
||||||
|
def iana_name(cp_name: str, strict: bool = True) -> str:
|
||||||
|
"""Returns the Python normalized encoding name (Not the IANA official name)."""
|
||||||
|
cp_name = cp_name.lower().replace("-", "_")
|
||||||
|
|
||||||
|
encoding_iana = _IANA_NAMES.get(cp_name)
|
||||||
|
if encoding_iana is not None:
|
||||||
|
return encoding_iana
|
||||||
|
|
||||||
|
if strict:
|
||||||
|
raise ValueError(f"Unable to retrieve IANA for '{cp_name}'")
|
||||||
|
|
||||||
|
return cp_name
|
||||||
|
|
||||||
|
|
||||||
|
def cp_similarity(iana_name_a: str, iana_name_b: str) -> float:
|
||||||
|
if is_multi_byte_encoding(iana_name_a) or is_multi_byte_encoding(iana_name_b):
|
||||||
|
return 0.0
|
||||||
|
|
||||||
|
decoder_a = importlib.import_module(f"encodings.{iana_name_a}").IncrementalDecoder
|
||||||
|
decoder_b = importlib.import_module(f"encodings.{iana_name_b}").IncrementalDecoder
|
||||||
|
|
||||||
|
id_a: IncrementalDecoder = decoder_a(errors="ignore")
|
||||||
|
id_b: IncrementalDecoder = decoder_b(errors="ignore")
|
||||||
|
|
||||||
|
character_match_count: int = 0
|
||||||
|
|
||||||
|
for i in range(256):
|
||||||
|
to_be_decoded: bytes = bytes([i])
|
||||||
|
if id_a.decode(to_be_decoded) == id_b.decode(to_be_decoded):
|
||||||
|
character_match_count += 1
|
||||||
|
|
||||||
|
return character_match_count / 256
|
||||||
|
|
||||||
|
|
||||||
|
def is_cp_similar(iana_name_a: str, iana_name_b: str) -> bool:
|
||||||
|
"""
|
||||||
|
Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using
|
||||||
|
the function cp_similarity.
|
||||||
|
"""
|
||||||
|
return (
|
||||||
|
iana_name_a in IANA_SUPPORTED_SIMILAR
|
||||||
|
and iana_name_b in IANA_SUPPORTED_SIMILAR[iana_name_a]
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def set_logging_handler(
|
||||||
|
name: str = "charset_normalizer",
|
||||||
|
level: int = logging.INFO,
|
||||||
|
format_string: str = "%(asctime)s | %(levelname)s | %(message)s",
|
||||||
|
) -> None:
|
||||||
|
logger = logging.getLogger(name)
|
||||||
|
logger.setLevel(level)
|
||||||
|
|
||||||
|
handler = logging.StreamHandler()
|
||||||
|
handler.setFormatter(logging.Formatter(format_string))
|
||||||
|
logger.addHandler(handler)
|
||||||
|
|
||||||
|
|
||||||
|
def cut_sequence_chunks(
|
||||||
|
sequences: bytes | bytearray,
|
||||||
|
encoding_iana: str,
|
||||||
|
offsets: range,
|
||||||
|
chunk_size: int,
|
||||||
|
bom_or_sig_available: bool,
|
||||||
|
strip_sig_or_bom: bool,
|
||||||
|
sig_payload: bytes,
|
||||||
|
is_multi_byte_decoder: bool,
|
||||||
|
decoded_payload: str | None = None,
|
||||||
|
deferred_decoding: bool = False,
|
||||||
|
) -> Generator[str, None, None]:
|
||||||
|
# iso2022 codec is stateful, generic mb cuter isn't going to cut it!
|
||||||
|
if decoded_payload and encoding_iana.startswith("iso2022_"):
|
||||||
|
decoded_length = len(decoded_payload)
|
||||||
|
sequence_length = len(sequences)
|
||||||
|
for i in offsets:
|
||||||
|
decoded_offset = i * decoded_length // sequence_length
|
||||||
|
chunk = decoded_payload[decoded_offset : decoded_offset + chunk_size]
|
||||||
|
if not chunk:
|
||||||
|
break
|
||||||
|
yield chunk
|
||||||
|
elif decoded_payload and not is_multi_byte_decoder:
|
||||||
|
for i in offsets:
|
||||||
|
chunk = decoded_payload[i : i + chunk_size]
|
||||||
|
if not chunk:
|
||||||
|
break
|
||||||
|
yield chunk
|
||||||
|
elif deferred_decoding:
|
||||||
|
# Deferred single-byte probing: the whole payload is not decoded
|
||||||
|
# yet. Single-byte codecs are stateless (1 byte == 1 char), hence
|
||||||
|
# decode(base)[i:j] == decode(base[i:j]): slicing the raw bytes
|
||||||
|
# yields exactly the chunks the branch above would have produced,
|
||||||
|
# short trailing chunks included, and raises UnicodeDecodeError on
|
||||||
|
# invalid bytes just like the whole-payload decode would.
|
||||||
|
base_bytes = (
|
||||||
|
sequences if not strip_sig_or_bom else sequences[len(sig_payload) :]
|
||||||
|
)
|
||||||
|
for i in offsets:
|
||||||
|
cut_sequence = base_bytes[i : i + chunk_size]
|
||||||
|
if not cut_sequence:
|
||||||
|
break
|
||||||
|
yield str(cut_sequence, encoding_iana)
|
||||||
|
else:
|
||||||
|
for i in offsets:
|
||||||
|
chunk_end = i + chunk_size
|
||||||
|
if chunk_end > len(sequences) + 8:
|
||||||
|
continue
|
||||||
|
|
||||||
|
cut_sequence = sequences[i : i + chunk_size]
|
||||||
|
|
||||||
|
if bom_or_sig_available and not strip_sig_or_bom:
|
||||||
|
cut_sequence = sig_payload + cut_sequence
|
||||||
|
|
||||||
|
chunk = cut_sequence.decode(
|
||||||
|
encoding_iana,
|
||||||
|
errors="ignore" if is_multi_byte_decoder else "strict",
|
||||||
|
)
|
||||||
|
|
||||||
|
# multi-byte bad cutting detector and adjustment
|
||||||
|
# not the cleanest way to perform that fix but clever enough for now.
|
||||||
|
if is_multi_byte_decoder and i > 0:
|
||||||
|
chunk_partial_size_chk: int = min(chunk_size, 16)
|
||||||
|
chunk_prefix = chunk[:chunk_partial_size_chk]
|
||||||
|
found_nearby = False
|
||||||
|
|
||||||
|
if decoded_payload:
|
||||||
|
decoded_length = len(decoded_payload)
|
||||||
|
expected_offset = i * decoded_length // len(sequences)
|
||||||
|
search_start = max(0, expected_offset - _MULTIBYTE_SEARCH_RADIUS)
|
||||||
|
search_end = min(
|
||||||
|
decoded_length, expected_offset + _MULTIBYTE_SEARCH_RADIUS
|
||||||
|
)
|
||||||
|
found_nearby = (
|
||||||
|
decoded_payload.find(chunk_prefix, search_start, search_end)
|
||||||
|
>= 0
|
||||||
|
)
|
||||||
|
|
||||||
|
if (
|
||||||
|
decoded_payload
|
||||||
|
and not found_nearby
|
||||||
|
and chunk_prefix not in decoded_payload
|
||||||
|
):
|
||||||
|
for j in range(i, i - 4, -1):
|
||||||
|
cut_sequence = sequences[j:chunk_end]
|
||||||
|
|
||||||
|
if bom_or_sig_available and not strip_sig_or_bom:
|
||||||
|
cut_sequence = sig_payload + cut_sequence
|
||||||
|
|
||||||
|
chunk = cut_sequence.decode(encoding_iana, errors="ignore")
|
||||||
|
|
||||||
|
if chunk[:chunk_partial_size_chk] in decoded_payload:
|
||||||
|
break
|
||||||
|
|
||||||
|
yield chunk
|
||||||
+8
@@ -0,0 +1,8 @@
|
|||||||
|
"""
|
||||||
|
Expose version
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
__version__ = "3.5.1"
|
||||||
|
VERSION = __version__.split(".")
|
||||||
+154
@@ -0,0 +1,154 @@
|
|||||||
|
# defusedxml
|
||||||
|
#
|
||||||
|
# Copyright (c) 2013 by Christian Heimes <[email protected]>
|
||||||
|
# Licensed to PSF under a Contributor Agreement.
|
||||||
|
# See https://www.python.org/psf/license for licensing details.
|
||||||
|
"""Defused xml.etree.ElementTree facade
|
||||||
|
"""
|
||||||
|
from __future__ import print_function, absolute_import
|
||||||
|
|
||||||
|
import sys
|
||||||
|
import warnings
|
||||||
|
from xml.etree.ElementTree import ParseError
|
||||||
|
from xml.etree.ElementTree import TreeBuilder as _TreeBuilder
|
||||||
|
from xml.etree.ElementTree import parse as _parse
|
||||||
|
from xml.etree.ElementTree import tostring
|
||||||
|
|
||||||
|
from .common import PY3
|
||||||
|
|
||||||
|
if PY3:
|
||||||
|
import importlib
|
||||||
|
else:
|
||||||
|
from xml.etree.ElementTree import XMLParser as _XMLParser
|
||||||
|
from xml.etree.ElementTree import iterparse as _iterparse
|
||||||
|
|
||||||
|
|
||||||
|
from .common import (
|
||||||
|
DTDForbidden,
|
||||||
|
EntitiesForbidden,
|
||||||
|
ExternalReferenceForbidden,
|
||||||
|
_generate_etree_functions,
|
||||||
|
)
|
||||||
|
|
||||||
|
__origin__ = "xml.etree.ElementTree"
|
||||||
|
|
||||||
|
|
||||||
|
def _get_py3_cls():
|
||||||
|
"""Python 3.3 hides the pure Python code but defusedxml requires it.
|
||||||
|
|
||||||
|
The code is based on test.support.import_fresh_module().
|
||||||
|
"""
|
||||||
|
pymodname = "xml.etree.ElementTree"
|
||||||
|
cmodname = "_elementtree"
|
||||||
|
|
||||||
|
pymod = sys.modules.pop(pymodname, None)
|
||||||
|
cmod = sys.modules.pop(cmodname, None)
|
||||||
|
|
||||||
|
sys.modules[cmodname] = None
|
||||||
|
try:
|
||||||
|
pure_pymod = importlib.import_module(pymodname)
|
||||||
|
finally:
|
||||||
|
# restore module
|
||||||
|
sys.modules[pymodname] = pymod
|
||||||
|
if cmod is not None:
|
||||||
|
sys.modules[cmodname] = cmod
|
||||||
|
else:
|
||||||
|
sys.modules.pop(cmodname, None)
|
||||||
|
# restore attribute on original package
|
||||||
|
etree_pkg = sys.modules["xml.etree"]
|
||||||
|
if pymod is not None:
|
||||||
|
etree_pkg.ElementTree = pymod
|
||||||
|
elif hasattr(etree_pkg, "ElementTree"):
|
||||||
|
del etree_pkg.ElementTree
|
||||||
|
|
||||||
|
_XMLParser = pure_pymod.XMLParser
|
||||||
|
_iterparse = pure_pymod.iterparse
|
||||||
|
# patch pure module to use ParseError from C extension
|
||||||
|
pure_pymod.ParseError = ParseError
|
||||||
|
|
||||||
|
return _XMLParser, _iterparse
|
||||||
|
|
||||||
|
|
||||||
|
if PY3:
|
||||||
|
_XMLParser, _iterparse = _get_py3_cls()
|
||||||
|
|
||||||
|
|
||||||
|
_sentinel = object()
|
||||||
|
|
||||||
|
|
||||||
|
class DefusedXMLParser(_XMLParser):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
html=_sentinel,
|
||||||
|
target=None,
|
||||||
|
encoding=None,
|
||||||
|
forbid_dtd=False,
|
||||||
|
forbid_entities=True,
|
||||||
|
forbid_external=True,
|
||||||
|
):
|
||||||
|
# Python 2.x old style class
|
||||||
|
_XMLParser.__init__(self, target=target, encoding=encoding)
|
||||||
|
if html is not _sentinel:
|
||||||
|
# the 'html' argument has been deprecated and ignored in all
|
||||||
|
# supported versions of Python. Python 3.8 finally removed it.
|
||||||
|
if html:
|
||||||
|
raise TypeError("'html=True' is no longer supported.")
|
||||||
|
else:
|
||||||
|
warnings.warn(
|
||||||
|
"'html' keyword argument is no longer supported. Pass "
|
||||||
|
"in arguments as keyword arguments.",
|
||||||
|
category=DeprecationWarning,
|
||||||
|
)
|
||||||
|
|
||||||
|
self.forbid_dtd = forbid_dtd
|
||||||
|
self.forbid_entities = forbid_entities
|
||||||
|
self.forbid_external = forbid_external
|
||||||
|
if PY3:
|
||||||
|
parser = self.parser
|
||||||
|
else:
|
||||||
|
parser = self._parser
|
||||||
|
if self.forbid_dtd:
|
||||||
|
parser.StartDoctypeDeclHandler = self.defused_start_doctype_decl
|
||||||
|
if self.forbid_entities:
|
||||||
|
parser.EntityDeclHandler = self.defused_entity_decl
|
||||||
|
parser.UnparsedEntityDeclHandler = self.defused_unparsed_entity_decl
|
||||||
|
if self.forbid_external:
|
||||||
|
parser.ExternalEntityRefHandler = self.defused_external_entity_ref_handler
|
||||||
|
|
||||||
|
def defused_start_doctype_decl(self, name, sysid, pubid, has_internal_subset):
|
||||||
|
raise DTDForbidden(name, sysid, pubid)
|
||||||
|
|
||||||
|
def defused_entity_decl(
|
||||||
|
self, name, is_parameter_entity, value, base, sysid, pubid, notation_name
|
||||||
|
):
|
||||||
|
raise EntitiesForbidden(name, value, base, sysid, pubid, notation_name)
|
||||||
|
|
||||||
|
def defused_unparsed_entity_decl(self, name, base, sysid, pubid, notation_name):
|
||||||
|
# expat 1.2
|
||||||
|
raise EntitiesForbidden(name, None, base, sysid, pubid, notation_name) # pragma: no cover
|
||||||
|
|
||||||
|
def defused_external_entity_ref_handler(self, context, base, sysid, pubid):
|
||||||
|
raise ExternalReferenceForbidden(context, base, sysid, pubid)
|
||||||
|
|
||||||
|
|
||||||
|
# aliases
|
||||||
|
# XMLParse is a typo, keep it for backwards compatibility
|
||||||
|
XMLTreeBuilder = XMLParse = XMLParser = DefusedXMLParser
|
||||||
|
|
||||||
|
parse, iterparse, fromstring = _generate_etree_functions(
|
||||||
|
DefusedXMLParser, _TreeBuilder, _parse, _iterparse
|
||||||
|
)
|
||||||
|
XML = fromstring
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"ParseError",
|
||||||
|
"XML",
|
||||||
|
"XMLParse",
|
||||||
|
"XMLParser",
|
||||||
|
"XMLTreeBuilder",
|
||||||
|
"fromstring",
|
||||||
|
"iterparse",
|
||||||
|
"parse",
|
||||||
|
"tostring",
|
||||||
|
]
|
||||||
+67
@@ -0,0 +1,67 @@
|
|||||||
|
# defusedxml
|
||||||
|
#
|
||||||
|
# Copyright (c) 2013 by Christian Heimes <[email protected]>
|
||||||
|
# Licensed to PSF under a Contributor Agreement.
|
||||||
|
# See https://www.python.org/psf/license for licensing details.
|
||||||
|
"""Defuse XML bomb denial of service vulnerabilities
|
||||||
|
"""
|
||||||
|
from __future__ import print_function, absolute_import
|
||||||
|
|
||||||
|
import warnings
|
||||||
|
|
||||||
|
from .common import (
|
||||||
|
DefusedXmlException,
|
||||||
|
DTDForbidden,
|
||||||
|
EntitiesForbidden,
|
||||||
|
ExternalReferenceForbidden,
|
||||||
|
NotSupportedError,
|
||||||
|
_apply_defusing,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def defuse_stdlib():
|
||||||
|
"""Monkey patch and defuse all stdlib packages
|
||||||
|
|
||||||
|
:warning: The monkey patch is an EXPERIMETNAL feature.
|
||||||
|
"""
|
||||||
|
defused = {}
|
||||||
|
|
||||||
|
with warnings.catch_warnings():
|
||||||
|
from . import cElementTree
|
||||||
|
from . import ElementTree
|
||||||
|
from . import minidom
|
||||||
|
from . import pulldom
|
||||||
|
from . import sax
|
||||||
|
from . import expatbuilder
|
||||||
|
from . import expatreader
|
||||||
|
from . import xmlrpc
|
||||||
|
|
||||||
|
xmlrpc.monkey_patch()
|
||||||
|
defused[xmlrpc] = None
|
||||||
|
|
||||||
|
defused_mods = [
|
||||||
|
cElementTree,
|
||||||
|
ElementTree,
|
||||||
|
minidom,
|
||||||
|
pulldom,
|
||||||
|
sax,
|
||||||
|
expatbuilder,
|
||||||
|
expatreader,
|
||||||
|
]
|
||||||
|
|
||||||
|
for defused_mod in defused_mods:
|
||||||
|
stdlib_mod = _apply_defusing(defused_mod)
|
||||||
|
defused[defused_mod] = stdlib_mod
|
||||||
|
|
||||||
|
return defused
|
||||||
|
|
||||||
|
|
||||||
|
__version__ = "0.7.1"
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"DefusedXmlException",
|
||||||
|
"DTDForbidden",
|
||||||
|
"EntitiesForbidden",
|
||||||
|
"ExternalReferenceForbidden",
|
||||||
|
"NotSupportedError",
|
||||||
|
]
|
||||||
+62
@@ -0,0 +1,62 @@
|
|||||||
|
# defusedxml
|
||||||
|
#
|
||||||
|
# Copyright (c) 2013 by Christian Heimes <[email protected]>
|
||||||
|
# Licensed to PSF under a Contributor Agreement.
|
||||||
|
# See https://www.python.org/psf/license for licensing details.
|
||||||
|
"""Defused xml.etree.cElementTree
|
||||||
|
"""
|
||||||
|
from __future__ import absolute_import
|
||||||
|
|
||||||
|
import warnings
|
||||||
|
|
||||||
|
from .common import _generate_etree_functions
|
||||||
|
|
||||||
|
from xml.etree.cElementTree import TreeBuilder as _TreeBuilder
|
||||||
|
from xml.etree.cElementTree import parse as _parse
|
||||||
|
from xml.etree.cElementTree import tostring
|
||||||
|
|
||||||
|
# iterparse from ElementTree!
|
||||||
|
from xml.etree.ElementTree import iterparse as _iterparse
|
||||||
|
|
||||||
|
# This module is an alias for ElementTree just like xml.etree.cElementTree
|
||||||
|
from .ElementTree import (
|
||||||
|
XML,
|
||||||
|
XMLParse,
|
||||||
|
XMLParser,
|
||||||
|
XMLTreeBuilder,
|
||||||
|
fromstring,
|
||||||
|
iterparse,
|
||||||
|
parse,
|
||||||
|
tostring,
|
||||||
|
DefusedXMLParser,
|
||||||
|
ParseError,
|
||||||
|
)
|
||||||
|
|
||||||
|
__origin__ = "xml.etree.cElementTree"
|
||||||
|
|
||||||
|
|
||||||
|
warnings.warn(
|
||||||
|
"defusedxml.cElementTree is deprecated, import from defusedxml.ElementTree instead.",
|
||||||
|
category=DeprecationWarning,
|
||||||
|
stacklevel=2,
|
||||||
|
)
|
||||||
|
|
||||||
|
# XMLParse is a typo, keep it for backwards compatibility
|
||||||
|
XMLTreeBuilder = XMLParse = XMLParser = DefusedXMLParser
|
||||||
|
|
||||||
|
parse, iterparse, fromstring = _generate_etree_functions(
|
||||||
|
DefusedXMLParser, _TreeBuilder, _parse, _iterparse
|
||||||
|
)
|
||||||
|
XML = fromstring
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"ParseError",
|
||||||
|
"XML",
|
||||||
|
"XMLParse",
|
||||||
|
"XMLParser",
|
||||||
|
"XMLTreeBuilder",
|
||||||
|
"fromstring",
|
||||||
|
"iterparse",
|
||||||
|
"parse",
|
||||||
|
"tostring",
|
||||||
|
]
|
||||||
@@ -0,0 +1,129 @@
|
|||||||
|
# defusedxml
|
||||||
|
#
|
||||||
|
# Copyright (c) 2013 by Christian Heimes <[email protected]>
|
||||||
|
# Licensed to PSF under a Contributor Agreement.
|
||||||
|
# See https://www.python.org/psf/license for licensing details.
|
||||||
|
"""Common constants, exceptions and helpe functions
|
||||||
|
"""
|
||||||
|
import sys
|
||||||
|
import xml.parsers.expat
|
||||||
|
|
||||||
|
PY3 = sys.version_info[0] == 3
|
||||||
|
|
||||||
|
# Fail early when pyexpat is not installed correctly
|
||||||
|
if not hasattr(xml.parsers.expat, "ParserCreate"):
|
||||||
|
raise ImportError("pyexpat") # pragma: no cover
|
||||||
|
|
||||||
|
|
||||||
|
class DefusedXmlException(ValueError):
|
||||||
|
"""Base exception"""
|
||||||
|
|
||||||
|
def __repr__(self):
|
||||||
|
return str(self)
|
||||||
|
|
||||||
|
|
||||||
|
class DTDForbidden(DefusedXmlException):
|
||||||
|
"""Document type definition is forbidden"""
|
||||||
|
|
||||||
|
def __init__(self, name, sysid, pubid):
|
||||||
|
super(DTDForbidden, self).__init__()
|
||||||
|
self.name = name
|
||||||
|
self.sysid = sysid
|
||||||
|
self.pubid = pubid
|
||||||
|
|
||||||
|
def __str__(self):
|
||||||
|
tpl = "DTDForbidden(name='{}', system_id={!r}, public_id={!r})"
|
||||||
|
return tpl.format(self.name, self.sysid, self.pubid)
|
||||||
|
|
||||||
|
|
||||||
|
class EntitiesForbidden(DefusedXmlException):
|
||||||
|
"""Entity definition is forbidden"""
|
||||||
|
|
||||||
|
def __init__(self, name, value, base, sysid, pubid, notation_name):
|
||||||
|
super(EntitiesForbidden, self).__init__()
|
||||||
|
self.name = name
|
||||||
|
self.value = value
|
||||||
|
self.base = base
|
||||||
|
self.sysid = sysid
|
||||||
|
self.pubid = pubid
|
||||||
|
self.notation_name = notation_name
|
||||||
|
|
||||||
|
def __str__(self):
|
||||||
|
tpl = "EntitiesForbidden(name='{}', system_id={!r}, public_id={!r})"
|
||||||
|
return tpl.format(self.name, self.sysid, self.pubid)
|
||||||
|
|
||||||
|
|
||||||
|
class ExternalReferenceForbidden(DefusedXmlException):
|
||||||
|
"""Resolving an external reference is forbidden"""
|
||||||
|
|
||||||
|
def __init__(self, context, base, sysid, pubid):
|
||||||
|
super(ExternalReferenceForbidden, self).__init__()
|
||||||
|
self.context = context
|
||||||
|
self.base = base
|
||||||
|
self.sysid = sysid
|
||||||
|
self.pubid = pubid
|
||||||
|
|
||||||
|
def __str__(self):
|
||||||
|
tpl = "ExternalReferenceForbidden(system_id='{}', public_id={})"
|
||||||
|
return tpl.format(self.sysid, self.pubid)
|
||||||
|
|
||||||
|
|
||||||
|
class NotSupportedError(DefusedXmlException):
|
||||||
|
"""The operation is not supported"""
|
||||||
|
|
||||||
|
|
||||||
|
def _apply_defusing(defused_mod):
|
||||||
|
assert defused_mod is sys.modules[defused_mod.__name__]
|
||||||
|
stdlib_name = defused_mod.__origin__
|
||||||
|
__import__(stdlib_name, {}, {}, ["*"])
|
||||||
|
stdlib_mod = sys.modules[stdlib_name]
|
||||||
|
stdlib_names = set(dir(stdlib_mod))
|
||||||
|
for name, obj in vars(defused_mod).items():
|
||||||
|
if name.startswith("_") or name not in stdlib_names:
|
||||||
|
continue
|
||||||
|
setattr(stdlib_mod, name, obj)
|
||||||
|
return stdlib_mod
|
||||||
|
|
||||||
|
|
||||||
|
def _generate_etree_functions(DefusedXMLParser, _TreeBuilder, _parse, _iterparse):
|
||||||
|
"""Factory for functions needed by etree, dependent on whether
|
||||||
|
cElementTree or ElementTree is used."""
|
||||||
|
|
||||||
|
def parse(source, parser=None, forbid_dtd=False, forbid_entities=True, forbid_external=True):
|
||||||
|
if parser is None:
|
||||||
|
parser = DefusedXMLParser(
|
||||||
|
target=_TreeBuilder(),
|
||||||
|
forbid_dtd=forbid_dtd,
|
||||||
|
forbid_entities=forbid_entities,
|
||||||
|
forbid_external=forbid_external,
|
||||||
|
)
|
||||||
|
return _parse(source, parser)
|
||||||
|
|
||||||
|
def iterparse(
|
||||||
|
source,
|
||||||
|
events=None,
|
||||||
|
parser=None,
|
||||||
|
forbid_dtd=False,
|
||||||
|
forbid_entities=True,
|
||||||
|
forbid_external=True,
|
||||||
|
):
|
||||||
|
if parser is None:
|
||||||
|
parser = DefusedXMLParser(
|
||||||
|
target=_TreeBuilder(),
|
||||||
|
forbid_dtd=forbid_dtd,
|
||||||
|
forbid_entities=forbid_entities,
|
||||||
|
forbid_external=forbid_external,
|
||||||
|
)
|
||||||
|
return _iterparse(source, events, parser)
|
||||||
|
|
||||||
|
def fromstring(text, forbid_dtd=False, forbid_entities=True, forbid_external=True):
|
||||||
|
parser = DefusedXMLParser(
|
||||||
|
target=_TreeBuilder(),
|
||||||
|
forbid_dtd=forbid_dtd,
|
||||||
|
forbid_entities=forbid_entities,
|
||||||
|
forbid_external=forbid_external,
|
||||||
|
)
|
||||||
|
parser.feed(text)
|
||||||
|
return parser.close()
|
||||||
|
|
||||||
|
return parse, iterparse, fromstring
|
||||||
+107
@@ -0,0 +1,107 @@
|
|||||||
|
# defusedxml
|
||||||
|
#
|
||||||
|
# Copyright (c) 2013 by Christian Heimes <[email protected]>
|
||||||
|
# Licensed to PSF under a Contributor Agreement.
|
||||||
|
# See https://www.python.org/psf/license for licensing details.
|
||||||
|
"""Defused xml.dom.expatbuilder
|
||||||
|
"""
|
||||||
|
from __future__ import print_function, absolute_import
|
||||||
|
|
||||||
|
from xml.dom.expatbuilder import ExpatBuilder as _ExpatBuilder
|
||||||
|
from xml.dom.expatbuilder import Namespaces as _Namespaces
|
||||||
|
|
||||||
|
from .common import DTDForbidden, EntitiesForbidden, ExternalReferenceForbidden
|
||||||
|
|
||||||
|
__origin__ = "xml.dom.expatbuilder"
|
||||||
|
|
||||||
|
|
||||||
|
class DefusedExpatBuilder(_ExpatBuilder):
|
||||||
|
"""Defused document builder"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self, options=None, forbid_dtd=False, forbid_entities=True, forbid_external=True
|
||||||
|
):
|
||||||
|
_ExpatBuilder.__init__(self, options)
|
||||||
|
self.forbid_dtd = forbid_dtd
|
||||||
|
self.forbid_entities = forbid_entities
|
||||||
|
self.forbid_external = forbid_external
|
||||||
|
|
||||||
|
def defused_start_doctype_decl(self, name, sysid, pubid, has_internal_subset):
|
||||||
|
raise DTDForbidden(name, sysid, pubid)
|
||||||
|
|
||||||
|
def defused_entity_decl(
|
||||||
|
self, name, is_parameter_entity, value, base, sysid, pubid, notation_name
|
||||||
|
):
|
||||||
|
raise EntitiesForbidden(name, value, base, sysid, pubid, notation_name)
|
||||||
|
|
||||||
|
def defused_unparsed_entity_decl(self, name, base, sysid, pubid, notation_name):
|
||||||
|
# expat 1.2
|
||||||
|
raise EntitiesForbidden(name, None, base, sysid, pubid, notation_name) # pragma: no cover
|
||||||
|
|
||||||
|
def defused_external_entity_ref_handler(self, context, base, sysid, pubid):
|
||||||
|
raise ExternalReferenceForbidden(context, base, sysid, pubid)
|
||||||
|
|
||||||
|
def install(self, parser):
|
||||||
|
_ExpatBuilder.install(self, parser)
|
||||||
|
|
||||||
|
if self.forbid_dtd:
|
||||||
|
parser.StartDoctypeDeclHandler = self.defused_start_doctype_decl
|
||||||
|
if self.forbid_entities:
|
||||||
|
# if self._options.entities:
|
||||||
|
parser.EntityDeclHandler = self.defused_entity_decl
|
||||||
|
parser.UnparsedEntityDeclHandler = self.defused_unparsed_entity_decl
|
||||||
|
if self.forbid_external:
|
||||||
|
parser.ExternalEntityRefHandler = self.defused_external_entity_ref_handler
|
||||||
|
|
||||||
|
|
||||||
|
class DefusedExpatBuilderNS(_Namespaces, DefusedExpatBuilder):
|
||||||
|
"""Defused document builder that supports namespaces."""
|
||||||
|
|
||||||
|
def install(self, parser):
|
||||||
|
DefusedExpatBuilder.install(self, parser)
|
||||||
|
if self._options.namespace_declarations:
|
||||||
|
parser.StartNamespaceDeclHandler = self.start_namespace_decl_handler
|
||||||
|
|
||||||
|
def reset(self):
|
||||||
|
DefusedExpatBuilder.reset(self)
|
||||||
|
self._initNamespaces()
|
||||||
|
|
||||||
|
|
||||||
|
def parse(file, namespaces=True, forbid_dtd=False, forbid_entities=True, forbid_external=True):
|
||||||
|
"""Parse a document, returning the resulting Document node.
|
||||||
|
|
||||||
|
'file' may be either a file name or an open file object.
|
||||||
|
"""
|
||||||
|
if namespaces:
|
||||||
|
build_builder = DefusedExpatBuilderNS
|
||||||
|
else:
|
||||||
|
build_builder = DefusedExpatBuilder
|
||||||
|
builder = build_builder(
|
||||||
|
forbid_dtd=forbid_dtd, forbid_entities=forbid_entities, forbid_external=forbid_external
|
||||||
|
)
|
||||||
|
|
||||||
|
if isinstance(file, str):
|
||||||
|
fp = open(file, "rb")
|
||||||
|
try:
|
||||||
|
result = builder.parseFile(fp)
|
||||||
|
finally:
|
||||||
|
fp.close()
|
||||||
|
else:
|
||||||
|
result = builder.parseFile(file)
|
||||||
|
return result
|
||||||
|
|
||||||
|
|
||||||
|
def parseString(
|
||||||
|
string, namespaces=True, forbid_dtd=False, forbid_entities=True, forbid_external=True
|
||||||
|
):
|
||||||
|
"""Parse a document from a string, returning the resulting
|
||||||
|
Document node.
|
||||||
|
"""
|
||||||
|
if namespaces:
|
||||||
|
build_builder = DefusedExpatBuilderNS
|
||||||
|
else:
|
||||||
|
build_builder = DefusedExpatBuilder
|
||||||
|
builder = build_builder(
|
||||||
|
forbid_dtd=forbid_dtd, forbid_entities=forbid_entities, forbid_external=forbid_external
|
||||||
|
)
|
||||||
|
return builder.parseString(string)
|
||||||
+61
@@ -0,0 +1,61 @@
|
|||||||
|
# defusedxml
|
||||||
|
#
|
||||||
|
# Copyright (c) 2013 by Christian Heimes <[email protected]>
|
||||||
|
# Licensed to PSF under a Contributor Agreement.
|
||||||
|
# See https://www.python.org/psf/license for licensing details.
|
||||||
|
"""Defused xml.sax.expatreader
|
||||||
|
"""
|
||||||
|
from __future__ import print_function, absolute_import
|
||||||
|
|
||||||
|
from xml.sax.expatreader import ExpatParser as _ExpatParser
|
||||||
|
|
||||||
|
from .common import DTDForbidden, EntitiesForbidden, ExternalReferenceForbidden
|
||||||
|
|
||||||
|
__origin__ = "xml.sax.expatreader"
|
||||||
|
|
||||||
|
|
||||||
|
class DefusedExpatParser(_ExpatParser):
|
||||||
|
"""Defused SAX driver for the pyexpat C module."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
namespaceHandling=0,
|
||||||
|
bufsize=2 ** 16 - 20,
|
||||||
|
forbid_dtd=False,
|
||||||
|
forbid_entities=True,
|
||||||
|
forbid_external=True,
|
||||||
|
):
|
||||||
|
_ExpatParser.__init__(self, namespaceHandling, bufsize)
|
||||||
|
self.forbid_dtd = forbid_dtd
|
||||||
|
self.forbid_entities = forbid_entities
|
||||||
|
self.forbid_external = forbid_external
|
||||||
|
|
||||||
|
def defused_start_doctype_decl(self, name, sysid, pubid, has_internal_subset):
|
||||||
|
raise DTDForbidden(name, sysid, pubid)
|
||||||
|
|
||||||
|
def defused_entity_decl(
|
||||||
|
self, name, is_parameter_entity, value, base, sysid, pubid, notation_name
|
||||||
|
):
|
||||||
|
raise EntitiesForbidden(name, value, base, sysid, pubid, notation_name)
|
||||||
|
|
||||||
|
def defused_unparsed_entity_decl(self, name, base, sysid, pubid, notation_name):
|
||||||
|
# expat 1.2
|
||||||
|
raise EntitiesForbidden(name, None, base, sysid, pubid, notation_name) # pragma: no cover
|
||||||
|
|
||||||
|
def defused_external_entity_ref_handler(self, context, base, sysid, pubid):
|
||||||
|
raise ExternalReferenceForbidden(context, base, sysid, pubid)
|
||||||
|
|
||||||
|
def reset(self):
|
||||||
|
_ExpatParser.reset(self)
|
||||||
|
parser = self._parser
|
||||||
|
if self.forbid_dtd:
|
||||||
|
parser.StartDoctypeDeclHandler = self.defused_start_doctype_decl
|
||||||
|
if self.forbid_entities:
|
||||||
|
parser.EntityDeclHandler = self.defused_entity_decl
|
||||||
|
parser.UnparsedEntityDeclHandler = self.defused_unparsed_entity_decl
|
||||||
|
if self.forbid_external:
|
||||||
|
parser.ExternalEntityRefHandler = self.defused_external_entity_ref_handler
|
||||||
|
|
||||||
|
|
||||||
|
def create_parser(*args, **kwargs):
|
||||||
|
return DefusedExpatParser(*args, **kwargs)
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
# defusedxml
|
||||||
|
#
|
||||||
|
# Copyright (c) 2013 by Christian Heimes <[email protected]>
|
||||||
|
# Licensed to PSF under a Contributor Agreement.
|
||||||
|
# See https://www.python.org/psf/license for licensing details.
|
||||||
|
"""DEPRECATED Example code for lxml.etree protection
|
||||||
|
|
||||||
|
The code has NO protection against decompression bombs.
|
||||||
|
"""
|
||||||
|
from __future__ import print_function, absolute_import
|
||||||
|
|
||||||
|
import threading
|
||||||
|
import warnings
|
||||||
|
|
||||||
|
from lxml import etree as _etree
|
||||||
|
|
||||||
|
from .common import DTDForbidden, EntitiesForbidden, NotSupportedError
|
||||||
|
|
||||||
|
LXML3 = _etree.LXML_VERSION[0] >= 3
|
||||||
|
|
||||||
|
__origin__ = "lxml.etree"
|
||||||
|
|
||||||
|
tostring = _etree.tostring
|
||||||
|
|
||||||
|
|
||||||
|
warnings.warn(
|
||||||
|
"defusedxml.lxml is no longer supported and will be removed in a future release.",
|
||||||
|
category=DeprecationWarning,
|
||||||
|
stacklevel=2,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class RestrictedElement(_etree.ElementBase):
|
||||||
|
"""A restricted Element class that filters out instances of some classes"""
|
||||||
|
|
||||||
|
__slots__ = ()
|
||||||
|
# blacklist = (etree._Entity, etree._ProcessingInstruction, etree._Comment)
|
||||||
|
blacklist = _etree._Entity
|
||||||
|
|
||||||
|
def _filter(self, iterator):
|
||||||
|
blacklist = self.blacklist
|
||||||
|
for child in iterator:
|
||||||
|
if isinstance(child, blacklist):
|
||||||
|
continue
|
||||||
|
yield child
|
||||||
|
|
||||||
|
def __iter__(self):
|
||||||
|
iterator = super(RestrictedElement, self).__iter__()
|
||||||
|
return self._filter(iterator)
|
||||||
|
|
||||||
|
def iterchildren(self, tag=None, reversed=False):
|
||||||
|
iterator = super(RestrictedElement, self).iterchildren(tag=tag, reversed=reversed)
|
||||||
|
return self._filter(iterator)
|
||||||
|
|
||||||
|
def iter(self, tag=None, *tags):
|
||||||
|
iterator = super(RestrictedElement, self).iter(tag=tag, *tags)
|
||||||
|
return self._filter(iterator)
|
||||||
|
|
||||||
|
def iterdescendants(self, tag=None, *tags):
|
||||||
|
iterator = super(RestrictedElement, self).iterdescendants(tag=tag, *tags)
|
||||||
|
return self._filter(iterator)
|
||||||
|
|
||||||
|
def itersiblings(self, tag=None, preceding=False):
|
||||||
|
iterator = super(RestrictedElement, self).itersiblings(tag=tag, preceding=preceding)
|
||||||
|
return self._filter(iterator)
|
||||||
|
|
||||||
|
def getchildren(self):
|
||||||
|
iterator = super(RestrictedElement, self).__iter__()
|
||||||
|
return list(self._filter(iterator))
|
||||||
|
|
||||||
|
def getiterator(self, tag=None):
|
||||||
|
iterator = super(RestrictedElement, self).getiterator(tag)
|
||||||
|
return self._filter(iterator)
|
||||||
|
|
||||||
|
|
||||||
|
class GlobalParserTLS(threading.local):
|
||||||
|
"""Thread local context for custom parser instances"""
|
||||||
|
|
||||||
|
parser_config = {
|
||||||
|
"resolve_entities": False,
|
||||||
|
# 'remove_comments': True,
|
||||||
|
# 'remove_pis': True,
|
||||||
|
}
|
||||||
|
|
||||||
|
element_class = RestrictedElement
|
||||||
|
|
||||||
|
def createDefaultParser(self):
|
||||||
|
parser = _etree.XMLParser(**self.parser_config)
|
||||||
|
element_class = self.element_class
|
||||||
|
if self.element_class is not None:
|
||||||
|
lookup = _etree.ElementDefaultClassLookup(element=element_class)
|
||||||
|
parser.set_element_class_lookup(lookup)
|
||||||
|
return parser
|
||||||
|
|
||||||
|
def setDefaultParser(self, parser):
|
||||||
|
self._default_parser = parser
|
||||||
|
|
||||||
|
def getDefaultParser(self):
|
||||||
|
parser = getattr(self, "_default_parser", None)
|
||||||
|
if parser is None:
|
||||||
|
parser = self.createDefaultParser()
|
||||||
|
self.setDefaultParser(parser)
|
||||||
|
return parser
|
||||||
|
|
||||||
|
|
||||||
|
_parser_tls = GlobalParserTLS()
|
||||||
|
getDefaultParser = _parser_tls.getDefaultParser
|
||||||
|
|
||||||
|
|
||||||
|
def check_docinfo(elementtree, forbid_dtd=False, forbid_entities=True):
|
||||||
|
"""Check docinfo of an element tree for DTD and entity declarations
|
||||||
|
|
||||||
|
The check for entity declarations needs lxml 3 or newer. lxml 2.x does
|
||||||
|
not support dtd.iterentities().
|
||||||
|
"""
|
||||||
|
docinfo = elementtree.docinfo
|
||||||
|
if docinfo.doctype:
|
||||||
|
if forbid_dtd:
|
||||||
|
raise DTDForbidden(docinfo.doctype, docinfo.system_url, docinfo.public_id)
|
||||||
|
if forbid_entities and not LXML3:
|
||||||
|
# lxml < 3 has no iterentities()
|
||||||
|
raise NotSupportedError("Unable to check for entity declarations " "in lxml 2.x")
|
||||||
|
|
||||||
|
if forbid_entities:
|
||||||
|
for dtd in docinfo.internalDTD, docinfo.externalDTD:
|
||||||
|
if dtd is None:
|
||||||
|
continue
|
||||||
|
for entity in dtd.iterentities():
|
||||||
|
raise EntitiesForbidden(entity.name, entity.content, None, None, None, None)
|
||||||
|
|
||||||
|
|
||||||
|
def parse(source, parser=None, base_url=None, forbid_dtd=False, forbid_entities=True):
|
||||||
|
if parser is None:
|
||||||
|
parser = getDefaultParser()
|
||||||
|
elementtree = _etree.parse(source, parser, base_url=base_url)
|
||||||
|
check_docinfo(elementtree, forbid_dtd, forbid_entities)
|
||||||
|
return elementtree
|
||||||
|
|
||||||
|
|
||||||
|
def fromstring(text, parser=None, base_url=None, forbid_dtd=False, forbid_entities=True):
|
||||||
|
if parser is None:
|
||||||
|
parser = getDefaultParser()
|
||||||
|
rootelement = _etree.fromstring(text, parser, base_url=base_url)
|
||||||
|
elementtree = rootelement.getroottree()
|
||||||
|
check_docinfo(elementtree, forbid_dtd, forbid_entities)
|
||||||
|
return rootelement
|
||||||
|
|
||||||
|
|
||||||
|
XML = fromstring
|
||||||
|
|
||||||
|
|
||||||
|
def iterparse(*args, **kwargs):
|
||||||
|
raise NotSupportedError("defused lxml.etree.iterparse not available")
|
||||||
@@ -0,0 +1,63 @@
|
|||||||
|
# defusedxml
|
||||||
|
#
|
||||||
|
# Copyright (c) 2013 by Christian Heimes <[email protected]>
|
||||||
|
# Licensed to PSF under a Contributor Agreement.
|
||||||
|
# See https://www.python.org/psf/license for licensing details.
|
||||||
|
"""Defused xml.dom.minidom
|
||||||
|
"""
|
||||||
|
from __future__ import print_function, absolute_import
|
||||||
|
|
||||||
|
from xml.dom.minidom import _do_pulldom_parse
|
||||||
|
from . import expatbuilder as _expatbuilder
|
||||||
|
from . import pulldom as _pulldom
|
||||||
|
|
||||||
|
__origin__ = "xml.dom.minidom"
|
||||||
|
|
||||||
|
|
||||||
|
def parse(
|
||||||
|
file, parser=None, bufsize=None, forbid_dtd=False, forbid_entities=True, forbid_external=True
|
||||||
|
):
|
||||||
|
"""Parse a file into a DOM by filename or file object."""
|
||||||
|
if parser is None and not bufsize:
|
||||||
|
return _expatbuilder.parse(
|
||||||
|
file,
|
||||||
|
forbid_dtd=forbid_dtd,
|
||||||
|
forbid_entities=forbid_entities,
|
||||||
|
forbid_external=forbid_external,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
return _do_pulldom_parse(
|
||||||
|
_pulldom.parse,
|
||||||
|
(file,),
|
||||||
|
{
|
||||||
|
"parser": parser,
|
||||||
|
"bufsize": bufsize,
|
||||||
|
"forbid_dtd": forbid_dtd,
|
||||||
|
"forbid_entities": forbid_entities,
|
||||||
|
"forbid_external": forbid_external,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def parseString(
|
||||||
|
string, parser=None, forbid_dtd=False, forbid_entities=True, forbid_external=True
|
||||||
|
):
|
||||||
|
"""Parse a file into a DOM from a string."""
|
||||||
|
if parser is None:
|
||||||
|
return _expatbuilder.parseString(
|
||||||
|
string,
|
||||||
|
forbid_dtd=forbid_dtd,
|
||||||
|
forbid_entities=forbid_entities,
|
||||||
|
forbid_external=forbid_external,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
return _do_pulldom_parse(
|
||||||
|
_pulldom.parseString,
|
||||||
|
(string,),
|
||||||
|
{
|
||||||
|
"parser": parser,
|
||||||
|
"forbid_dtd": forbid_dtd,
|
||||||
|
"forbid_entities": forbid_entities,
|
||||||
|
"forbid_external": forbid_external,
|
||||||
|
},
|
||||||
|
)
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# defusedxml
|
||||||
|
#
|
||||||
|
# Copyright (c) 2013 by Christian Heimes <[email protected]>
|
||||||
|
# Licensed to PSF under a Contributor Agreement.
|
||||||
|
# See https://www.python.org/psf/license for licensing details.
|
||||||
|
"""Defused xml.dom.pulldom
|
||||||
|
"""
|
||||||
|
from __future__ import print_function, absolute_import
|
||||||
|
|
||||||
|
from xml.dom.pulldom import parse as _parse
|
||||||
|
from xml.dom.pulldom import parseString as _parseString
|
||||||
|
from .sax import make_parser
|
||||||
|
|
||||||
|
__origin__ = "xml.dom.pulldom"
|
||||||
|
|
||||||
|
|
||||||
|
def parse(
|
||||||
|
stream_or_string,
|
||||||
|
parser=None,
|
||||||
|
bufsize=None,
|
||||||
|
forbid_dtd=False,
|
||||||
|
forbid_entities=True,
|
||||||
|
forbid_external=True,
|
||||||
|
):
|
||||||
|
if parser is None:
|
||||||
|
parser = make_parser()
|
||||||
|
parser.forbid_dtd = forbid_dtd
|
||||||
|
parser.forbid_entities = forbid_entities
|
||||||
|
parser.forbid_external = forbid_external
|
||||||
|
return _parse(stream_or_string, parser, bufsize)
|
||||||
|
|
||||||
|
|
||||||
|
def parseString(
|
||||||
|
string, parser=None, forbid_dtd=False, forbid_entities=True, forbid_external=True
|
||||||
|
):
|
||||||
|
if parser is None:
|
||||||
|
parser = make_parser()
|
||||||
|
parser.forbid_dtd = forbid_dtd
|
||||||
|
parser.forbid_entities = forbid_entities
|
||||||
|
parser.forbid_external = forbid_external
|
||||||
|
return _parseString(string, parser)
|
||||||
@@ -0,0 +1,60 @@
|
|||||||
|
# defusedxml
|
||||||
|
#
|
||||||
|
# Copyright (c) 2013 by Christian Heimes <[email protected]>
|
||||||
|
# Licensed to PSF under a Contributor Agreement.
|
||||||
|
# See https://www.python.org/psf/license for licensing details.
|
||||||
|
"""Defused xml.sax
|
||||||
|
"""
|
||||||
|
from __future__ import print_function, absolute_import
|
||||||
|
|
||||||
|
from xml.sax import InputSource as _InputSource
|
||||||
|
from xml.sax import ErrorHandler as _ErrorHandler
|
||||||
|
|
||||||
|
from . import expatreader
|
||||||
|
|
||||||
|
__origin__ = "xml.sax"
|
||||||
|
|
||||||
|
|
||||||
|
def parse(
|
||||||
|
source,
|
||||||
|
handler,
|
||||||
|
errorHandler=_ErrorHandler(),
|
||||||
|
forbid_dtd=False,
|
||||||
|
forbid_entities=True,
|
||||||
|
forbid_external=True,
|
||||||
|
):
|
||||||
|
parser = make_parser()
|
||||||
|
parser.setContentHandler(handler)
|
||||||
|
parser.setErrorHandler(errorHandler)
|
||||||
|
parser.forbid_dtd = forbid_dtd
|
||||||
|
parser.forbid_entities = forbid_entities
|
||||||
|
parser.forbid_external = forbid_external
|
||||||
|
parser.parse(source)
|
||||||
|
|
||||||
|
|
||||||
|
def parseString(
|
||||||
|
string,
|
||||||
|
handler,
|
||||||
|
errorHandler=_ErrorHandler(),
|
||||||
|
forbid_dtd=False,
|
||||||
|
forbid_entities=True,
|
||||||
|
forbid_external=True,
|
||||||
|
):
|
||||||
|
from io import BytesIO
|
||||||
|
|
||||||
|
if errorHandler is None:
|
||||||
|
errorHandler = _ErrorHandler()
|
||||||
|
parser = make_parser()
|
||||||
|
parser.setContentHandler(handler)
|
||||||
|
parser.setErrorHandler(errorHandler)
|
||||||
|
parser.forbid_dtd = forbid_dtd
|
||||||
|
parser.forbid_entities = forbid_entities
|
||||||
|
parser.forbid_external = forbid_external
|
||||||
|
|
||||||
|
inpsrc = _InputSource()
|
||||||
|
inpsrc.setByteStream(BytesIO(string))
|
||||||
|
parser.parse(inpsrc)
|
||||||
|
|
||||||
|
|
||||||
|
def make_parser(parser_list=[]):
|
||||||
|
return expatreader.create_parser()
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
# defusedxml
|
||||||
|
#
|
||||||
|
# Copyright (c) 2013 by Christian Heimes <[email protected]>
|
||||||
|
# Licensed to PSF under a Contributor Agreement.
|
||||||
|
# See https://www.python.org/psf/license for licensing details.
|
||||||
|
"""Defused xmlrpclib
|
||||||
|
|
||||||
|
Also defuses gzip bomb
|
||||||
|
"""
|
||||||
|
from __future__ import print_function, absolute_import
|
||||||
|
|
||||||
|
import io
|
||||||
|
|
||||||
|
from .common import DTDForbidden, EntitiesForbidden, ExternalReferenceForbidden, PY3
|
||||||
|
|
||||||
|
if PY3:
|
||||||
|
__origin__ = "xmlrpc.client"
|
||||||
|
from xmlrpc.client import ExpatParser
|
||||||
|
from xmlrpc import client as xmlrpc_client
|
||||||
|
from xmlrpc import server as xmlrpc_server
|
||||||
|
from xmlrpc.client import gzip_decode as _orig_gzip_decode
|
||||||
|
from xmlrpc.client import GzipDecodedResponse as _OrigGzipDecodedResponse
|
||||||
|
else:
|
||||||
|
__origin__ = "xmlrpclib"
|
||||||
|
from xmlrpclib import ExpatParser
|
||||||
|
import xmlrpclib as xmlrpc_client
|
||||||
|
|
||||||
|
xmlrpc_server = None
|
||||||
|
from xmlrpclib import gzip_decode as _orig_gzip_decode
|
||||||
|
from xmlrpclib import GzipDecodedResponse as _OrigGzipDecodedResponse
|
||||||
|
|
||||||
|
try:
|
||||||
|
import gzip
|
||||||
|
except ImportError: # pragma: no cover
|
||||||
|
gzip = None
|
||||||
|
|
||||||
|
|
||||||
|
# Limit maximum request size to prevent resource exhaustion DoS
|
||||||
|
# Also used to limit maximum amount of gzip decoded data in order to prevent
|
||||||
|
# decompression bombs
|
||||||
|
# A value of -1 or smaller disables the limit
|
||||||
|
MAX_DATA = 30 * 1024 * 1024 # 30 MB
|
||||||
|
|
||||||
|
|
||||||
|
def defused_gzip_decode(data, limit=None):
|
||||||
|
"""gzip encoded data -> unencoded data
|
||||||
|
|
||||||
|
Decode data using the gzip content encoding as described in RFC 1952
|
||||||
|
"""
|
||||||
|
if not gzip: # pragma: no cover
|
||||||
|
raise NotImplementedError
|
||||||
|
if limit is None:
|
||||||
|
limit = MAX_DATA
|
||||||
|
f = io.BytesIO(data)
|
||||||
|
gzf = gzip.GzipFile(mode="rb", fileobj=f)
|
||||||
|
try:
|
||||||
|
if limit < 0: # no limit
|
||||||
|
decoded = gzf.read()
|
||||||
|
else:
|
||||||
|
decoded = gzf.read(limit + 1)
|
||||||
|
except IOError: # pragma: no cover
|
||||||
|
raise ValueError("invalid data")
|
||||||
|
f.close()
|
||||||
|
gzf.close()
|
||||||
|
if limit >= 0 and len(decoded) > limit:
|
||||||
|
raise ValueError("max gzipped payload length exceeded")
|
||||||
|
return decoded
|
||||||
|
|
||||||
|
|
||||||
|
class DefusedGzipDecodedResponse(gzip.GzipFile if gzip else object):
|
||||||
|
"""a file-like object to decode a response encoded with the gzip
|
||||||
|
method, as described in RFC 1952.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, response, limit=None):
|
||||||
|
# response doesn't support tell() and read(), required by
|
||||||
|
# GzipFile
|
||||||
|
if not gzip: # pragma: no cover
|
||||||
|
raise NotImplementedError
|
||||||
|
self.limit = limit = limit if limit is not None else MAX_DATA
|
||||||
|
if limit < 0: # no limit
|
||||||
|
data = response.read()
|
||||||
|
self.readlength = None
|
||||||
|
else:
|
||||||
|
data = response.read(limit + 1)
|
||||||
|
self.readlength = 0
|
||||||
|
if limit >= 0 and len(data) > limit:
|
||||||
|
raise ValueError("max payload length exceeded")
|
||||||
|
self.stringio = io.BytesIO(data)
|
||||||
|
gzip.GzipFile.__init__(self, mode="rb", fileobj=self.stringio)
|
||||||
|
|
||||||
|
def read(self, n):
|
||||||
|
if self.limit >= 0:
|
||||||
|
left = self.limit - self.readlength
|
||||||
|
n = min(n, left + 1)
|
||||||
|
data = gzip.GzipFile.read(self, n)
|
||||||
|
self.readlength += len(data)
|
||||||
|
if self.readlength > self.limit:
|
||||||
|
raise ValueError("max payload length exceeded")
|
||||||
|
return data
|
||||||
|
else:
|
||||||
|
return gzip.GzipFile.read(self, n)
|
||||||
|
|
||||||
|
def close(self):
|
||||||
|
gzip.GzipFile.close(self)
|
||||||
|
self.stringio.close()
|
||||||
|
|
||||||
|
|
||||||
|
class DefusedExpatParser(ExpatParser):
|
||||||
|
def __init__(self, target, forbid_dtd=False, forbid_entities=True, forbid_external=True):
|
||||||
|
ExpatParser.__init__(self, target)
|
||||||
|
self.forbid_dtd = forbid_dtd
|
||||||
|
self.forbid_entities = forbid_entities
|
||||||
|
self.forbid_external = forbid_external
|
||||||
|
parser = self._parser
|
||||||
|
if self.forbid_dtd:
|
||||||
|
parser.StartDoctypeDeclHandler = self.defused_start_doctype_decl
|
||||||
|
if self.forbid_entities:
|
||||||
|
parser.EntityDeclHandler = self.defused_entity_decl
|
||||||
|
parser.UnparsedEntityDeclHandler = self.defused_unparsed_entity_decl
|
||||||
|
if self.forbid_external:
|
||||||
|
parser.ExternalEntityRefHandler = self.defused_external_entity_ref_handler
|
||||||
|
|
||||||
|
def defused_start_doctype_decl(self, name, sysid, pubid, has_internal_subset):
|
||||||
|
raise DTDForbidden(name, sysid, pubid)
|
||||||
|
|
||||||
|
def defused_entity_decl(
|
||||||
|
self, name, is_parameter_entity, value, base, sysid, pubid, notation_name
|
||||||
|
):
|
||||||
|
raise EntitiesForbidden(name, value, base, sysid, pubid, notation_name)
|
||||||
|
|
||||||
|
def defused_unparsed_entity_decl(self, name, base, sysid, pubid, notation_name):
|
||||||
|
# expat 1.2
|
||||||
|
raise EntitiesForbidden(name, None, base, sysid, pubid, notation_name) # pragma: no cover
|
||||||
|
|
||||||
|
def defused_external_entity_ref_handler(self, context, base, sysid, pubid):
|
||||||
|
raise ExternalReferenceForbidden(context, base, sysid, pubid)
|
||||||
|
|
||||||
|
|
||||||
|
def monkey_patch():
|
||||||
|
xmlrpc_client.FastParser = DefusedExpatParser
|
||||||
|
xmlrpc_client.GzipDecodedResponse = DefusedGzipDecodedResponse
|
||||||
|
xmlrpc_client.gzip_decode = defused_gzip_decode
|
||||||
|
if xmlrpc_server:
|
||||||
|
xmlrpc_server.gzip_decode = defused_gzip_decode
|
||||||
|
|
||||||
|
|
||||||
|
def unmonkey_patch():
|
||||||
|
xmlrpc_client.FastParser = None
|
||||||
|
xmlrpc_client.GzipDecodedResponse = _OrigGzipDecodedResponse
|
||||||
|
xmlrpc_client.gzip_decode = _orig_gzip_decode
|
||||||
|
if xmlrpc_server:
|
||||||
|
xmlrpc_server.gzip_decode = _orig_gzip_decode
|
||||||
@@ -0,0 +1,47 @@
|
|||||||
|
from .core import (
|
||||||
|
IDNABidiError,
|
||||||
|
IDNAError,
|
||||||
|
InvalidCodepoint,
|
||||||
|
InvalidCodepointContext,
|
||||||
|
alabel,
|
||||||
|
check_bidi,
|
||||||
|
check_hyphen_ok,
|
||||||
|
check_initial_combiner,
|
||||||
|
check_label,
|
||||||
|
check_nfc,
|
||||||
|
decode,
|
||||||
|
encode,
|
||||||
|
ulabel,
|
||||||
|
uts46_remap,
|
||||||
|
valid_contextj,
|
||||||
|
valid_contexto,
|
||||||
|
valid_label_length,
|
||||||
|
valid_string_length,
|
||||||
|
)
|
||||||
|
from .idnadata import __version__ as unicode_version
|
||||||
|
from .intranges import intranges_contain
|
||||||
|
from .package_data import __version__
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"__version__",
|
||||||
|
"unicode_version",
|
||||||
|
"IDNABidiError",
|
||||||
|
"IDNAError",
|
||||||
|
"InvalidCodepoint",
|
||||||
|
"InvalidCodepointContext",
|
||||||
|
"alabel",
|
||||||
|
"check_bidi",
|
||||||
|
"check_hyphen_ok",
|
||||||
|
"check_initial_combiner",
|
||||||
|
"check_label",
|
||||||
|
"check_nfc",
|
||||||
|
"decode",
|
||||||
|
"encode",
|
||||||
|
"intranges_contain",
|
||||||
|
"ulabel",
|
||||||
|
"uts46_remap",
|
||||||
|
"valid_contextj",
|
||||||
|
"valid_contexto",
|
||||||
|
"valid_label_length",
|
||||||
|
"valid_string_length",
|
||||||
|
]
|
||||||
@@ -0,0 +1,6 @@
|
|||||||
|
import sys
|
||||||
|
|
||||||
|
from .cli import main
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,132 @@
|
|||||||
|
"""Command-line interface for the :mod:`idna` package.
|
||||||
|
|
||||||
|
Invoked via ``python -m idna``. See :func:`main` for the entry point.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import sys
|
||||||
|
from itertools import chain
|
||||||
|
from typing import IO, TYPE_CHECKING
|
||||||
|
|
||||||
|
from . import IDNAError, decode, encode, unicode_version
|
||||||
|
from .core import _alabel_prefix, _unicode_dots_re
|
||||||
|
from .package_data import __version__
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from collections.abc import Iterable
|
||||||
|
|
||||||
|
|
||||||
|
def _looks_like_alabel(s: str) -> bool:
|
||||||
|
"""Return True if any label in ``s`` carries the ``xn--`` ACE prefix."""
|
||||||
|
prefix = _alabel_prefix.decode("ascii")
|
||||||
|
return any(label.lower().startswith(prefix) for label in _unicode_dots_re.split(s))
|
||||||
|
|
||||||
|
|
||||||
|
def _build_parser() -> argparse.ArgumentParser:
|
||||||
|
parser = argparse.ArgumentParser(
|
||||||
|
prog="python -m idna",
|
||||||
|
description=(
|
||||||
|
"Convert a domain name between its Unicode (U-label) and "
|
||||||
|
"ASCII-compatible (A-label) forms. With no mode flag, the "
|
||||||
|
"direction is chosen from the first input — if it contains "
|
||||||
|
"an xn-- label the stream is decoded, otherwise it is "
|
||||||
|
"encoded — and the same mode is applied to every remaining "
|
||||||
|
"input. UTS #46 mapping is applied by default; pass "
|
||||||
|
"--strict to disable it. When no domains are given on the "
|
||||||
|
"command line and stdin is piped, one domain per line is "
|
||||||
|
"read from stdin."
|
||||||
|
),
|
||||||
|
)
|
||||||
|
mode = parser.add_mutually_exclusive_group()
|
||||||
|
mode.add_argument(
|
||||||
|
"-e",
|
||||||
|
"--encode",
|
||||||
|
dest="mode",
|
||||||
|
action="store_const",
|
||||||
|
const="encode",
|
||||||
|
help="Encode the input to its ASCII A-label form.",
|
||||||
|
)
|
||||||
|
mode.add_argument(
|
||||||
|
"-d",
|
||||||
|
"--decode",
|
||||||
|
dest="mode",
|
||||||
|
action="store_const",
|
||||||
|
const="decode",
|
||||||
|
help="Decode the input from its ASCII A-label form.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--strict",
|
||||||
|
action="store_true",
|
||||||
|
help="Disable the default UTS #46 mapping and apply IDNA 2008 rules verbatim.",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"--version",
|
||||||
|
action="version",
|
||||||
|
version=f"idna {__version__} (Unicode {unicode_version})",
|
||||||
|
)
|
||||||
|
parser.add_argument(
|
||||||
|
"domain",
|
||||||
|
nargs="*",
|
||||||
|
help="One or more domain names to convert. Omit to read from stdin.",
|
||||||
|
)
|
||||||
|
return parser
|
||||||
|
|
||||||
|
|
||||||
|
def _iter_stdin(stream: IO[str]) -> Iterable[str]:
|
||||||
|
"""Yield non-empty stripped lines from ``stream``, ignoring blanks."""
|
||||||
|
for line in stream:
|
||||||
|
stripped = line.strip()
|
||||||
|
if stripped:
|
||||||
|
yield stripped
|
||||||
|
|
||||||
|
|
||||||
|
def _convert_one(domain: str, mode: str, uts46: bool) -> bool:
|
||||||
|
"""Convert ``domain`` and write the result; return ``False`` on failure."""
|
||||||
|
try:
|
||||||
|
if mode == "decode":
|
||||||
|
print(decode(domain, uts46=uts46))
|
||||||
|
else:
|
||||||
|
print(encode(domain, uts46=uts46).decode("ascii"))
|
||||||
|
except IDNAError as err:
|
||||||
|
print(f"idna: {mode} failed for {domain!r}: {err}", file=sys.stderr)
|
||||||
|
return False
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def main(argv: list[str] | None = None) -> int:
|
||||||
|
"""Entry point for ``python -m idna``.
|
||||||
|
|
||||||
|
When more than one domain is supplied (via positional arguments or
|
||||||
|
piped stdin) and no mode flag is given, the first input determines
|
||||||
|
the direction and that mode is applied uniformly to the rest.
|
||||||
|
|
||||||
|
:param argv: Argument list excluding the program name. Defaults to
|
||||||
|
:data:`sys.argv` when ``None``.
|
||||||
|
:returns: ``0`` on success, ``1`` if any conversion fails.
|
||||||
|
"""
|
||||||
|
parser = _build_parser()
|
||||||
|
args = parser.parse_args(argv)
|
||||||
|
uts46 = not args.strict
|
||||||
|
|
||||||
|
if args.domain:
|
||||||
|
domains: Iterable[str] = args.domain
|
||||||
|
elif not sys.stdin.isatty():
|
||||||
|
domains = _iter_stdin(sys.stdin)
|
||||||
|
else:
|
||||||
|
parser.error("a domain argument is required when stdin is a terminal")
|
||||||
|
|
||||||
|
iterator = iter(domains)
|
||||||
|
first = next(iterator, None)
|
||||||
|
if first is None:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
mode = args.mode or ("decode" if _looks_like_alabel(first) else "encode")
|
||||||
|
|
||||||
|
results = [_convert_one(domain, mode, uts46) for domain in chain([first], iterator)]
|
||||||
|
return 0 if all(results) else 1
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
@@ -0,0 +1,163 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import codecs
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .core import IDNAError, _unicode_dots_re, alabel, decode, encode, ulabel
|
||||||
|
|
||||||
|
|
||||||
|
class Codec(codecs.Codec):
|
||||||
|
"""Stateless IDNA 2008 codec.
|
||||||
|
|
||||||
|
Implements the :class:`codecs.Codec` protocol so that the whole-domain
|
||||||
|
encoder (:func:`idna.encode`) and decoder (:func:`idna.decode`) are
|
||||||
|
accessible through the standard codec machinery as ``"idna2008"``.
|
||||||
|
|
||||||
|
Only the ``"strict"`` error handler is supported; any other handler
|
||||||
|
raises :exc:`~idna.IDNAError`.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def encode(self, data: str, errors: str = "strict") -> tuple[bytes, int]: # ty: ignore[invalid-method-override]
|
||||||
|
if errors != "strict":
|
||||||
|
raise IDNAError(f'Unsupported error handling "{errors}"', code="unsupported_errors")
|
||||||
|
|
||||||
|
if not data:
|
||||||
|
return b"", 0
|
||||||
|
|
||||||
|
return encode(data), len(data)
|
||||||
|
|
||||||
|
def decode(self, data: bytes, errors: str = "strict") -> tuple[str, int]: # ty: ignore[invalid-method-override]
|
||||||
|
if errors != "strict":
|
||||||
|
raise IDNAError(f'Unsupported error handling "{errors}"', code="unsupported_errors")
|
||||||
|
|
||||||
|
if not data:
|
||||||
|
return "", 0
|
||||||
|
|
||||||
|
return decode(data), len(data)
|
||||||
|
|
||||||
|
|
||||||
|
class IncrementalEncoder(codecs.BufferedIncrementalEncoder):
|
||||||
|
"""Incremental IDNA 2008 encoder.
|
||||||
|
|
||||||
|
Buffers a partial trailing label across calls until either the next
|
||||||
|
label separator is seen or ``final=True``, so that streamed input is
|
||||||
|
encoded one whole label at a time. Any of the four Unicode label
|
||||||
|
separators (``U+002E``, ``U+3002``, ``U+FF0E``, ``U+FF61``) ends a
|
||||||
|
label; the result always uses ``U+002E`` as the separator.
|
||||||
|
|
||||||
|
Only the ``"strict"`` error handler is supported.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def _buffer_encode(self, data: str, errors: str, final: bool) -> tuple[bytes, int]: # ty: ignore[invalid-method-override]
|
||||||
|
if errors != "strict":
|
||||||
|
raise IDNAError(f'Unsupported error handling "{errors}"', code="unsupported_errors")
|
||||||
|
|
||||||
|
if not data:
|
||||||
|
return b"", 0
|
||||||
|
|
||||||
|
labels = _unicode_dots_re.split(data)
|
||||||
|
trailing_dot = b""
|
||||||
|
if labels:
|
||||||
|
if not labels[-1]:
|
||||||
|
trailing_dot = b"."
|
||||||
|
del labels[-1]
|
||||||
|
elif not final:
|
||||||
|
# Keep potentially unfinished label until the next call
|
||||||
|
del labels[-1]
|
||||||
|
if labels:
|
||||||
|
trailing_dot = b"."
|
||||||
|
|
||||||
|
result = []
|
||||||
|
size = 0
|
||||||
|
for label in labels:
|
||||||
|
result.append(alabel(label))
|
||||||
|
if size:
|
||||||
|
size += 1
|
||||||
|
size += len(label)
|
||||||
|
|
||||||
|
result_bytes = b".".join(result) + trailing_dot
|
||||||
|
size += len(trailing_dot)
|
||||||
|
return result_bytes, size
|
||||||
|
|
||||||
|
|
||||||
|
class IncrementalDecoder(codecs.BufferedIncrementalDecoder):
|
||||||
|
"""Incremental IDNA 2008 decoder.
|
||||||
|
|
||||||
|
Buffers a partial trailing label across calls until either the next
|
||||||
|
label separator is seen or ``final=True``, so that streamed input is
|
||||||
|
decoded one whole label at a time.
|
||||||
|
|
||||||
|
Only the ``"strict"`` error handler is supported.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def _buffer_decode(self, data: Any, errors: str, final: bool) -> tuple[str, int]: # ty: ignore[invalid-method-override]
|
||||||
|
if errors != "strict":
|
||||||
|
raise IDNAError(f'Unsupported error handling "{errors}"', code="unsupported_errors")
|
||||||
|
|
||||||
|
if not data:
|
||||||
|
return ("", 0)
|
||||||
|
|
||||||
|
if not isinstance(data, str):
|
||||||
|
try:
|
||||||
|
data = str(data, "ascii")
|
||||||
|
except UnicodeDecodeError as err:
|
||||||
|
raise IDNAError("Invalid ASCII in A-label", code="invalid_ascii") from err
|
||||||
|
|
||||||
|
labels = _unicode_dots_re.split(data)
|
||||||
|
trailing_dot = ""
|
||||||
|
if labels:
|
||||||
|
if not labels[-1]:
|
||||||
|
trailing_dot = "."
|
||||||
|
del labels[-1]
|
||||||
|
elif not final:
|
||||||
|
# Keep potentially unfinished label until the next call
|
||||||
|
del labels[-1]
|
||||||
|
if labels:
|
||||||
|
trailing_dot = "."
|
||||||
|
|
||||||
|
result = []
|
||||||
|
size = 0
|
||||||
|
for label in labels:
|
||||||
|
result.append(ulabel(label))
|
||||||
|
if size:
|
||||||
|
size += 1
|
||||||
|
size += len(label)
|
||||||
|
|
||||||
|
result_str = ".".join(result) + trailing_dot
|
||||||
|
size += len(trailing_dot)
|
||||||
|
return (result_str, size)
|
||||||
|
|
||||||
|
|
||||||
|
class StreamWriter(Codec, codecs.StreamWriter):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class StreamReader(Codec, codecs.StreamReader):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def search_function(name: str) -> codecs.CodecInfo | None:
|
||||||
|
"""Codec search function registered with :mod:`codecs`.
|
||||||
|
|
||||||
|
Returns a :class:`codecs.CodecInfo` for the ``"idna2008"`` codec name
|
||||||
|
so that ``str.encode("idna2008")`` and ``bytes.decode("idna2008")``
|
||||||
|
invoke the IDNA 2008 codec defined in this module.
|
||||||
|
|
||||||
|
:param name: The codec name being looked up.
|
||||||
|
:returns: A :class:`codecs.CodecInfo` instance if ``name`` is
|
||||||
|
``"idna2008"``, otherwise ``None``.
|
||||||
|
"""
|
||||||
|
if name != "idna2008":
|
||||||
|
return None
|
||||||
|
return codecs.CodecInfo(
|
||||||
|
name=name,
|
||||||
|
encode=Codec().encode,
|
||||||
|
decode=Codec().decode, # type: ignore
|
||||||
|
incrementalencoder=IncrementalEncoder,
|
||||||
|
incrementaldecoder=IncrementalDecoder,
|
||||||
|
streamwriter=StreamWriter,
|
||||||
|
streamreader=StreamReader,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
codecs.register(search_function)
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .core import decode, encode
|
||||||
|
|
||||||
|
|
||||||
|
def ToASCII(label: str) -> bytes:
|
||||||
|
"""Compatibility shim for :rfc:`3490` ``ToASCII``.
|
||||||
|
|
||||||
|
Delegates to :func:`idna.encode` (IDNA 2008). Provided to ease porting
|
||||||
|
of code written against the legacy :mod:`encodings.idna` API; new code
|
||||||
|
should call :func:`idna.encode` directly.
|
||||||
|
|
||||||
|
:param label: The label or domain to encode.
|
||||||
|
:returns: The encoded form as ASCII :class:`bytes`.
|
||||||
|
"""
|
||||||
|
return encode(label)
|
||||||
|
|
||||||
|
|
||||||
|
def ToUnicode(label: bytes | bytearray) -> str:
|
||||||
|
"""Compatibility shim for :rfc:`3490` ``ToUnicode``.
|
||||||
|
|
||||||
|
Delegates to :func:`idna.decode` (IDNA 2008). Provided to ease porting
|
||||||
|
of code written against the legacy :mod:`encodings.idna` API; new code
|
||||||
|
should call :func:`idna.decode` directly.
|
||||||
|
|
||||||
|
:param label: The label or domain to decode.
|
||||||
|
:returns: The decoded Unicode form.
|
||||||
|
"""
|
||||||
|
return decode(label)
|
||||||
|
|
||||||
|
|
||||||
|
def nameprep(s: Any) -> None:
|
||||||
|
"""Stub for :rfc:`3491` Nameprep, which is not used by IDNA 2008.
|
||||||
|
|
||||||
|
IDNA 2008 (:rfc:`5891`) replaces Nameprep with the per-codepoint
|
||||||
|
validity classes from :rfc:`5892`; this function exists only to
|
||||||
|
return a clear error if legacy code attempts to call it.
|
||||||
|
|
||||||
|
:raises NotImplementedError: Always.
|
||||||
|
"""
|
||||||
|
raise NotImplementedError("IDNA 2008 does not utilise nameprep protocol")
|
||||||
@@ -0,0 +1,862 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import bisect
|
||||||
|
import re
|
||||||
|
import unicodedata
|
||||||
|
import warnings
|
||||||
|
from typing import Literal
|
||||||
|
|
||||||
|
from . import idnadata
|
||||||
|
from .intranges import intranges_contain
|
||||||
|
|
||||||
|
_virama_combining_class = 9
|
||||||
|
_alabel_prefix = b"xn--"
|
||||||
|
_max_input_length = 1024
|
||||||
|
_STATUS_VALID, _STATUS_MAPPED, _STATUS_DEVIATION, _STATUS_IGNORED = b"VMDI"
|
||||||
|
_unicode_dots_re = re.compile("[\u002e\u3002\uff0e\uff61]")
|
||||||
|
_std3_disallowed_re = re.compile("[\x00-\x2c\x2f\x3a-\x40A-Z\x5b-\x60\x7b-\x7f]")
|
||||||
|
_bidi_rtl_first = frozenset({"R", "AL"})
|
||||||
|
_bidi_rtl_categories = frozenset({"R", "AL", "AN"})
|
||||||
|
_bidi_rtl_allowed = frozenset({"R", "AL", "AN", "EN", "ES", "CS", "ET", "ON", "BN", "NSM"})
|
||||||
|
_bidi_rtl_valid_ending = frozenset({"R", "AL", "EN", "AN"})
|
||||||
|
_bidi_rtl_numeric = frozenset({"AN", "EN"})
|
||||||
|
_bidi_ltr_allowed = frozenset({"L", "EN", "ES", "CS", "ET", "ON", "BN", "NSM"})
|
||||||
|
_bidi_ltr_valid_ending = frozenset({"L", "EN"})
|
||||||
|
_bidi_joiner_l_or_d = frozenset({"L", "D"})
|
||||||
|
_bidi_joiner_r_or_d = frozenset({"R", "D"})
|
||||||
|
|
||||||
|
|
||||||
|
def _joining_type(cp: int) -> str | None:
|
||||||
|
for jt, ranges in idnadata.joining_types.items():
|
||||||
|
if intranges_contain(cp, ranges):
|
||||||
|
return jt
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
# Machine-readable identifiers for the rule an :class:`IDNAError` reports.
|
||||||
|
# These strings are stable and documented; exception message wording is not.
|
||||||
|
_ErrorCode = Literal[
|
||||||
|
"input_too_long",
|
||||||
|
"label_too_long",
|
||||||
|
"domain_too_long",
|
||||||
|
"empty_label",
|
||||||
|
"empty_domain",
|
||||||
|
"not_nfc",
|
||||||
|
"hyphen_3_4",
|
||||||
|
"hyphen_start_end",
|
||||||
|
"leading_combiner",
|
||||||
|
"disallowed_codepoint",
|
||||||
|
"contextj",
|
||||||
|
"contexto",
|
||||||
|
"unknown_codepoint",
|
||||||
|
"bidi_rule_1",
|
||||||
|
"bidi_rule_2",
|
||||||
|
"bidi_rule_3",
|
||||||
|
"bidi_rule_4",
|
||||||
|
"bidi_rule_5",
|
||||||
|
"bidi_rule_6",
|
||||||
|
"bidi_unknown_direction",
|
||||||
|
"invalid_alabel",
|
||||||
|
"non_canonical_alabel",
|
||||||
|
"invalid_ascii",
|
||||||
|
"invalid_utf8",
|
||||||
|
"uts46_disallowed",
|
||||||
|
"uts46_std3",
|
||||||
|
"unsupported_errors",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class IDNAError(UnicodeError):
|
||||||
|
"""Base exception for all IDNA-encoding related problems.
|
||||||
|
|
||||||
|
``str(err)`` is a human-readable description of the failure. The
|
||||||
|
exception also carries machine-readable attributes so callers do not
|
||||||
|
need to parse the message:
|
||||||
|
|
||||||
|
* ``code`` -- a short, stable identifier for the rule that failed, such
|
||||||
|
as ``"disallowed_codepoint"`` or ``"bidi_rule_2"``; the full list is
|
||||||
|
documented in the README. Message wording, by contrast, may change
|
||||||
|
between releases.
|
||||||
|
* ``text`` -- the label (or, for UTS #46 processing, the domain) that
|
||||||
|
was being validated;
|
||||||
|
* ``codepoint`` -- the offending codepoint, as an ``int``;
|
||||||
|
* ``position`` -- the 1-based index of the offending character within
|
||||||
|
``text``, matching the position quoted in the message.
|
||||||
|
|
||||||
|
Each is ``None`` when it does not apply.
|
||||||
|
"""
|
||||||
|
|
||||||
|
code: str | None
|
||||||
|
text: str | None
|
||||||
|
codepoint: int | None
|
||||||
|
position: int | None
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
*args: object,
|
||||||
|
code: _ErrorCode | None = None,
|
||||||
|
text: str | None = None,
|
||||||
|
codepoint: int | None = None,
|
||||||
|
position: int | None = None,
|
||||||
|
) -> None:
|
||||||
|
super().__init__(*args)
|
||||||
|
self.code = code
|
||||||
|
self.text = text
|
||||||
|
self.codepoint = codepoint
|
||||||
|
self.position = position
|
||||||
|
|
||||||
|
|
||||||
|
class IDNABidiError(IDNAError):
|
||||||
|
"""Exception when bidirectional requirements are not satisfied"""
|
||||||
|
|
||||||
|
|
||||||
|
class InvalidCodepoint(IDNAError):
|
||||||
|
"""Exception when a disallowed or unallocated codepoint is used"""
|
||||||
|
|
||||||
|
|
||||||
|
class InvalidCodepointContext(IDNAError):
|
||||||
|
"""Exception when the codepoint is not valid in the context it is used"""
|
||||||
|
|
||||||
|
|
||||||
|
def _combining_class(cp: int) -> int:
|
||||||
|
v = unicodedata.combining(chr(cp))
|
||||||
|
if v == 0 and not unicodedata.name(chr(cp)):
|
||||||
|
raise ValueError("Unknown character in unicodedata")
|
||||||
|
return v
|
||||||
|
|
||||||
|
|
||||||
|
def _is_script(cp: str, script: str) -> bool:
|
||||||
|
return intranges_contain(ord(cp), idnadata.scripts[script])
|
||||||
|
|
||||||
|
|
||||||
|
def _punycode(s: str) -> bytes:
|
||||||
|
return s.encode("punycode")
|
||||||
|
|
||||||
|
|
||||||
|
def _unot(s: int) -> str:
|
||||||
|
return f"U+{s:04X}"
|
||||||
|
|
||||||
|
|
||||||
|
def valid_label_length(label: bytes | str) -> bool:
|
||||||
|
"""Check that a label does not exceed the maximum permitted length.
|
||||||
|
|
||||||
|
Per :rfc:`1035` (and :rfc:`5891` §4.2.4) a DNS label must not exceed
|
||||||
|
63 octets. The argument may be either a :class:`str` (a U-label, where
|
||||||
|
length is measured in characters) or :class:`bytes` (an A-label, where
|
||||||
|
length is measured in octets).
|
||||||
|
|
||||||
|
:param label: The label to check.
|
||||||
|
:returns: ``True`` if the label is within the length limit, otherwise
|
||||||
|
``False``.
|
||||||
|
"""
|
||||||
|
return len(label) <= 63
|
||||||
|
|
||||||
|
|
||||||
|
def valid_string_length(domain: bytes | str, trailing_dot: bool) -> bool:
|
||||||
|
"""Check that a full domain name does not exceed the maximum length.
|
||||||
|
|
||||||
|
Per :rfc:`1035`, a domain name is limited to 253 octets when no trailing
|
||||||
|
dot is present, or 254 octets when one is included.
|
||||||
|
|
||||||
|
:param domain: The full (possibly multi-label) domain name.
|
||||||
|
:param trailing_dot: ``True`` if ``domain`` includes a trailing ``.``.
|
||||||
|
:returns: ``True`` if the domain is within the length limit, otherwise
|
||||||
|
``False``.
|
||||||
|
"""
|
||||||
|
return len(domain) <= (254 if trailing_dot else 253)
|
||||||
|
|
||||||
|
|
||||||
|
def check_bidi(label: str, check_ltr: bool = False) -> bool:
|
||||||
|
"""Validate the Bidi Rule from :rfc:`5893` for a single label.
|
||||||
|
|
||||||
|
The Bidi Rule constrains how bidirectional characters (Hebrew, Arabic,
|
||||||
|
etc.) may appear within a label. By default the check is only applied
|
||||||
|
when the label contains at least one right-to-left character (Unicode
|
||||||
|
bidirectional categories ``R``, ``AL``, or ``AN``); set ``check_ltr``
|
||||||
|
to ``True`` to apply it to LTR-only labels as well.
|
||||||
|
|
||||||
|
:param label: The label to validate, as a Unicode string.
|
||||||
|
:param check_ltr: If ``True``, apply the rules even when the label
|
||||||
|
contains no RTL characters.
|
||||||
|
:returns: ``True`` if the label satisfies the Bidi Rule.
|
||||||
|
:raises IDNABidiError: If any of Bidi Rule conditions 1-6 are violated,
|
||||||
|
or if the directional category of a codepoint cannot be determined.
|
||||||
|
"""
|
||||||
|
if len(label) > _max_input_length:
|
||||||
|
raise IDNAError("Label too long", code="input_too_long")
|
||||||
|
# Bidi rules should only be applied if string contains RTL characters
|
||||||
|
bidi_label = False
|
||||||
|
for idx, cp in enumerate(label, 1):
|
||||||
|
direction = unicodedata.bidirectional(cp)
|
||||||
|
if direction == "":
|
||||||
|
# String likely comes from a newer version of Unicode
|
||||||
|
raise IDNABidiError(
|
||||||
|
f"Unknown directionality in label {label!r} at position {idx}",
|
||||||
|
code="bidi_unknown_direction",
|
||||||
|
text=label,
|
||||||
|
codepoint=ord(cp),
|
||||||
|
position=idx,
|
||||||
|
)
|
||||||
|
if direction in _bidi_rtl_categories:
|
||||||
|
bidi_label = True
|
||||||
|
if not bidi_label and not check_ltr:
|
||||||
|
return True
|
||||||
|
|
||||||
|
# Bidi rule 1
|
||||||
|
direction = unicodedata.bidirectional(label[0])
|
||||||
|
if direction in _bidi_rtl_first:
|
||||||
|
rtl = True
|
||||||
|
elif direction == "L":
|
||||||
|
rtl = False
|
||||||
|
else:
|
||||||
|
raise IDNABidiError(
|
||||||
|
f"First codepoint in label {label!r} must be directionality L, R or AL",
|
||||||
|
code="bidi_rule_1",
|
||||||
|
text=label,
|
||||||
|
codepoint=ord(label[0]),
|
||||||
|
position=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
valid_ending = False
|
||||||
|
ending_idx = 1
|
||||||
|
number_type: str | None = None
|
||||||
|
for idx, cp in enumerate(label, 1):
|
||||||
|
direction = unicodedata.bidirectional(cp)
|
||||||
|
|
||||||
|
if rtl:
|
||||||
|
# Bidi rule 2
|
||||||
|
if direction not in _bidi_rtl_allowed:
|
||||||
|
raise IDNABidiError(
|
||||||
|
f"Invalid direction for codepoint at position {idx} in a right-to-left label",
|
||||||
|
code="bidi_rule_2",
|
||||||
|
text=label,
|
||||||
|
codepoint=ord(cp),
|
||||||
|
position=idx,
|
||||||
|
)
|
||||||
|
# Bidi rule 3
|
||||||
|
if direction in _bidi_rtl_valid_ending:
|
||||||
|
valid_ending = True
|
||||||
|
ending_idx = idx
|
||||||
|
elif direction != "NSM":
|
||||||
|
valid_ending = False
|
||||||
|
ending_idx = idx
|
||||||
|
# Bidi rule 4
|
||||||
|
if direction in _bidi_rtl_numeric:
|
||||||
|
if not number_type:
|
||||||
|
number_type = direction
|
||||||
|
elif number_type != direction:
|
||||||
|
raise IDNABidiError(
|
||||||
|
"Can not mix numeral types in a right-to-left label",
|
||||||
|
code="bidi_rule_4",
|
||||||
|
text=label,
|
||||||
|
codepoint=ord(cp),
|
||||||
|
position=idx,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Bidi rule 5
|
||||||
|
if direction not in _bidi_ltr_allowed:
|
||||||
|
raise IDNABidiError(
|
||||||
|
f"Invalid direction for codepoint at position {idx} in a left-to-right label",
|
||||||
|
code="bidi_rule_5",
|
||||||
|
text=label,
|
||||||
|
codepoint=ord(cp),
|
||||||
|
position=idx,
|
||||||
|
)
|
||||||
|
# Bidi rule 6
|
||||||
|
if direction in _bidi_ltr_valid_ending:
|
||||||
|
valid_ending = True
|
||||||
|
ending_idx = idx
|
||||||
|
elif direction != "NSM":
|
||||||
|
valid_ending = False
|
||||||
|
ending_idx = idx
|
||||||
|
|
||||||
|
if not valid_ending:
|
||||||
|
# Rules 3 and 6 concern the last character that is not a
|
||||||
|
# non-spacing mark, which is what ``ending_idx`` tracks.
|
||||||
|
raise IDNABidiError(
|
||||||
|
"Label ends with illegal codepoint directionality",
|
||||||
|
code="bidi_rule_3" if rtl else "bidi_rule_6",
|
||||||
|
text=label,
|
||||||
|
codepoint=ord(label[ending_idx - 1]),
|
||||||
|
position=ending_idx,
|
||||||
|
)
|
||||||
|
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def check_initial_combiner(label: str) -> bool:
|
||||||
|
"""Reject labels that begin with a combining mark.
|
||||||
|
|
||||||
|
Per :rfc:`5891` §4.2.3.2 a label must not start with a character of
|
||||||
|
Unicode general category ``M`` (Mark).
|
||||||
|
|
||||||
|
:param label: The label to check.
|
||||||
|
:returns: ``True`` if the first character is not a combining mark.
|
||||||
|
:raises IDNAError: If the label begins with a combining character.
|
||||||
|
"""
|
||||||
|
if label and unicodedata.category(label[0])[0] == "M":
|
||||||
|
raise IDNAError(
|
||||||
|
"Label begins with an illegal combining character",
|
||||||
|
code="leading_combiner",
|
||||||
|
text=label,
|
||||||
|
codepoint=ord(label[0]),
|
||||||
|
position=1,
|
||||||
|
)
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def check_hyphen_ok(label: str) -> bool:
|
||||||
|
"""Validate the hyphen restrictions for a label.
|
||||||
|
|
||||||
|
Per :rfc:`5891` §4.2.3.1 a label must not start or end with a hyphen
|
||||||
|
(``U+002D``), and must not have hyphens in both the third and fourth
|
||||||
|
positions (the prefix reserved for A-labels).
|
||||||
|
|
||||||
|
:param label: The label to check.
|
||||||
|
:returns: ``True`` if the hyphen restrictions are satisfied.
|
||||||
|
:raises IDNAError: If any of the hyphen restrictions are violated.
|
||||||
|
"""
|
||||||
|
if label[2:4] == "--":
|
||||||
|
raise IDNAError("Label has disallowed hyphens in 3rd and 4th position", code="hyphen_3_4")
|
||||||
|
if label.startswith("-") or label.endswith("-"):
|
||||||
|
raise IDNAError("Label must not start or end with a hyphen", code="hyphen_start_end")
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
def check_nfc(label: str) -> None:
|
||||||
|
"""Require that a label is in Unicode Normalization Form C.
|
||||||
|
|
||||||
|
:param label: The label to check.
|
||||||
|
:raises IDNAError: If ``label`` differs from its NFC normalisation.
|
||||||
|
"""
|
||||||
|
if len(label) > _max_input_length:
|
||||||
|
raise IDNAError("Label too long", code="input_too_long")
|
||||||
|
if unicodedata.normalize("NFC", label) != label:
|
||||||
|
raise IDNAError("Label must be in Normalization Form C", code="not_nfc")
|
||||||
|
|
||||||
|
|
||||||
|
def valid_contextj(label: str, pos: int) -> bool:
|
||||||
|
"""Validate the CONTEXTJ rules from :rfc:`5892` Appendix A.
|
||||||
|
|
||||||
|
These rules govern the contextual use of the joiner codepoints
|
||||||
|
``U+200C`` (ZERO WIDTH NON-JOINER, Appendix A.1) and ``U+200D``
|
||||||
|
(ZERO WIDTH JOINER, Appendix A.2) within a label.
|
||||||
|
|
||||||
|
:param label: The label containing the codepoint.
|
||||||
|
:param pos: Index of the joiner codepoint within ``label``.
|
||||||
|
:returns: ``True`` if the codepoint at ``pos`` satisfies its CONTEXTJ
|
||||||
|
rule, ``False`` otherwise (including when the codepoint at
|
||||||
|
``pos`` is not a recognised joiner).
|
||||||
|
:raises ValueError: If an adjacent codepoint has no Unicode name when
|
||||||
|
determining its combining class.
|
||||||
|
:raises IDNAError: If ``label`` exceeds the defensive input length limit.
|
||||||
|
"""
|
||||||
|
if len(label) > _max_input_length:
|
||||||
|
raise IDNAError("Label too long", code="input_too_long")
|
||||||
|
cp_value = ord(label[pos])
|
||||||
|
|
||||||
|
if cp_value == 0x200C:
|
||||||
|
if pos > 0 and _combining_class(ord(label[pos - 1])) == _virama_combining_class:
|
||||||
|
return True
|
||||||
|
|
||||||
|
ok = False
|
||||||
|
for i in range(pos - 1, -1, -1):
|
||||||
|
joining_type = _joining_type(ord(label[i]))
|
||||||
|
if joining_type == "T":
|
||||||
|
continue
|
||||||
|
if joining_type in _bidi_joiner_l_or_d:
|
||||||
|
ok = True
|
||||||
|
break
|
||||||
|
break
|
||||||
|
|
||||||
|
if not ok:
|
||||||
|
return False
|
||||||
|
|
||||||
|
ok = False
|
||||||
|
for i in range(pos + 1, len(label)):
|
||||||
|
joining_type = _joining_type(ord(label[i]))
|
||||||
|
if joining_type == "T":
|
||||||
|
continue
|
||||||
|
if joining_type in _bidi_joiner_r_or_d:
|
||||||
|
ok = True
|
||||||
|
break
|
||||||
|
break
|
||||||
|
return ok
|
||||||
|
|
||||||
|
if cp_value == 0x200D:
|
||||||
|
return pos > 0 and _combining_class(ord(label[pos - 1])) == _virama_combining_class
|
||||||
|
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def valid_contexto(label: str, pos: int, exception: bool = False) -> bool:
|
||||||
|
"""Validate the CONTEXTO rules from :rfc:`5892` Appendix A.
|
||||||
|
|
||||||
|
Covers the contextual rules for codepoints such as MIDDLE DOT
|
||||||
|
(``U+00B7``), Greek lower numeral sign, Hebrew punctuation, Katakana
|
||||||
|
middle dot, and the Arabic-Indic / Extended Arabic-Indic digit ranges.
|
||||||
|
|
||||||
|
:param label: The label containing the codepoint.
|
||||||
|
:param pos: Index of the codepoint within ``label``.
|
||||||
|
:param exception: Reserved for forward compatibility; currently unused.
|
||||||
|
:returns: ``True`` if the codepoint at ``pos`` satisfies its CONTEXTO
|
||||||
|
rule, ``False`` otherwise (including when the codepoint is not a
|
||||||
|
recognised CONTEXTO codepoint).
|
||||||
|
:raises IDNAError: If ``label`` exceeds the defensive input length limit.
|
||||||
|
"""
|
||||||
|
if len(label) > _max_input_length:
|
||||||
|
raise IDNAError("Label too long", code="input_too_long")
|
||||||
|
cp_value = ord(label[pos])
|
||||||
|
|
||||||
|
if cp_value == 0x00B7:
|
||||||
|
return 0 < pos < len(label) - 1 and ord(label[pos - 1]) == 0x006C and ord(label[pos + 1]) == 0x006C
|
||||||
|
|
||||||
|
if cp_value == 0x0375:
|
||||||
|
if pos < len(label) - 1 and len(label) > 1:
|
||||||
|
return _is_script(label[pos + 1], "Greek")
|
||||||
|
return False
|
||||||
|
|
||||||
|
if cp_value in {0x05F3, 0x05F4}:
|
||||||
|
if pos > 0:
|
||||||
|
return _is_script(label[pos - 1], "Hebrew")
|
||||||
|
return False
|
||||||
|
|
||||||
|
if cp_value == 0x30FB:
|
||||||
|
for cp in label:
|
||||||
|
if cp == "\u30fb":
|
||||||
|
continue
|
||||||
|
if _is_script(cp, "Hiragana") or _is_script(cp, "Katakana") or _is_script(cp, "Han"):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
|
||||||
|
if 0x660 <= cp_value <= 0x669:
|
||||||
|
return not any(0x6F0 <= ord(cp) <= 0x06F9 for cp in label)
|
||||||
|
|
||||||
|
if 0x6F0 <= cp_value <= 0x6F9:
|
||||||
|
return not any(0x660 <= ord(cp) <= 0x0669 for cp in label)
|
||||||
|
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def check_label(label: str | bytes | bytearray) -> None:
|
||||||
|
"""Run the full set of IDNA 2008 validity checks on a single label.
|
||||||
|
|
||||||
|
Applies, in order: NFC normalisation (:func:`check_nfc`), hyphen
|
||||||
|
restrictions (:func:`check_hyphen_ok`), the no-leading-combiner rule
|
||||||
|
(:func:`check_initial_combiner`), per-codepoint validity (PVALID,
|
||||||
|
CONTEXTJ, CONTEXTO classes from :rfc:`5892`), and the Bidi Rule
|
||||||
|
(:func:`check_bidi`).
|
||||||
|
|
||||||
|
:param label: The label to validate. ``bytes`` or ``bytearray`` input
|
||||||
|
is decoded as UTF-8 first.
|
||||||
|
:raises IDNAError: If the label is empty or fails a structural rule.
|
||||||
|
:raises InvalidCodepoint: If the label contains a DISALLOWED or
|
||||||
|
UNASSIGNED codepoint.
|
||||||
|
:raises InvalidCodepointContext: If a CONTEXTJ or CONTEXTO codepoint
|
||||||
|
is not valid in its context.
|
||||||
|
:raises IDNABidiError: If the Bidi Rule is violated.
|
||||||
|
"""
|
||||||
|
if len(label) > _max_input_length:
|
||||||
|
raise IDNAError("Label too long", code="input_too_long")
|
||||||
|
if isinstance(label, (bytes, bytearray)):
|
||||||
|
try:
|
||||||
|
label = label.decode("utf-8")
|
||||||
|
except UnicodeDecodeError as err:
|
||||||
|
raise IDNAError("Invalid UTF-8 in label", code="invalid_utf8") from err
|
||||||
|
if len(label) == 0:
|
||||||
|
raise IDNAError("Empty Label", code="empty_label")
|
||||||
|
|
||||||
|
# Check against the domain length rather than the label length to
|
||||||
|
# support some UTS #46 use cases, while still bounding the work done
|
||||||
|
# by the label contextual rules below.
|
||||||
|
if not valid_string_length(label, trailing_dot=True):
|
||||||
|
raise IDNAError("Label too long", code="label_too_long")
|
||||||
|
|
||||||
|
check_nfc(label)
|
||||||
|
check_hyphen_ok(label)
|
||||||
|
check_initial_combiner(label)
|
||||||
|
|
||||||
|
for pos, cp in enumerate(label):
|
||||||
|
cp_value = ord(cp)
|
||||||
|
if intranges_contain(cp_value, idnadata.codepoint_classes["PVALID"]):
|
||||||
|
continue
|
||||||
|
if intranges_contain(cp_value, idnadata.codepoint_classes["CONTEXTJ"]):
|
||||||
|
try:
|
||||||
|
contextj_ok = valid_contextj(label, pos)
|
||||||
|
except ValueError as err:
|
||||||
|
raise IDNAError(
|
||||||
|
f"Unknown codepoint adjacent to joiner {_unot(cp_value)} at position {pos + 1} in {label!r}",
|
||||||
|
code="unknown_codepoint",
|
||||||
|
text=label,
|
||||||
|
codepoint=cp_value,
|
||||||
|
position=pos + 1,
|
||||||
|
) from err
|
||||||
|
if not contextj_ok:
|
||||||
|
raise InvalidCodepointContext(
|
||||||
|
f"Joiner {_unot(cp_value)} not allowed at position {pos + 1} in {label!r}",
|
||||||
|
code="contextj",
|
||||||
|
text=label,
|
||||||
|
codepoint=cp_value,
|
||||||
|
position=pos + 1,
|
||||||
|
)
|
||||||
|
elif intranges_contain(cp_value, idnadata.codepoint_classes["CONTEXTO"]):
|
||||||
|
if not valid_contexto(label, pos):
|
||||||
|
raise InvalidCodepointContext(
|
||||||
|
f"Codepoint {_unot(cp_value)} not allowed at position {pos + 1} in {label!r}",
|
||||||
|
code="contexto",
|
||||||
|
text=label,
|
||||||
|
codepoint=cp_value,
|
||||||
|
position=pos + 1,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
raise InvalidCodepoint(
|
||||||
|
f"Codepoint {_unot(cp_value)} at position {pos + 1} of {label!r} not allowed",
|
||||||
|
code="disallowed_codepoint",
|
||||||
|
text=label,
|
||||||
|
codepoint=cp_value,
|
||||||
|
position=pos + 1,
|
||||||
|
)
|
||||||
|
|
||||||
|
check_bidi(label)
|
||||||
|
|
||||||
|
|
||||||
|
def alabel(label: str) -> bytes:
|
||||||
|
"""Convert a single U-label into its A-label form.
|
||||||
|
|
||||||
|
The result is the ASCII-Compatible Encoding (ACE) form per :rfc:`5891`
|
||||||
|
§4: the label is validated, Punycode-encoded, and prefixed with
|
||||||
|
``xn--``. Pure ASCII labels that are already valid IDNA labels are
|
||||||
|
returned unchanged (as :class:`bytes`).
|
||||||
|
|
||||||
|
:param label: The label to convert, as a Unicode string.
|
||||||
|
:returns: The A-label as ASCII-encoded :class:`bytes`.
|
||||||
|
:raises IDNAError: If the label is invalid or the resulting A-label
|
||||||
|
exceeds 63 octets.
|
||||||
|
"""
|
||||||
|
if len(label) > _max_input_length:
|
||||||
|
raise IDNAError("Label too long", code="input_too_long")
|
||||||
|
try:
|
||||||
|
label_bytes = label.encode("ascii")
|
||||||
|
except UnicodeEncodeError:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
ulabel(label_bytes)
|
||||||
|
if not valid_label_length(label_bytes):
|
||||||
|
raise IDNAError("Label too long", code="label_too_long")
|
||||||
|
return label_bytes
|
||||||
|
|
||||||
|
check_label(label)
|
||||||
|
label_bytes = _alabel_prefix + _punycode(label)
|
||||||
|
|
||||||
|
if not valid_label_length(label_bytes):
|
||||||
|
raise IDNAError("Label too long", code="label_too_long")
|
||||||
|
|
||||||
|
return label_bytes
|
||||||
|
|
||||||
|
|
||||||
|
def ulabel(label: str | bytes | bytearray) -> str:
|
||||||
|
"""Convert a single A-label into its U-label form.
|
||||||
|
|
||||||
|
Performs the inverse of :func:`alabel`: an ``xn--``-prefixed label is
|
||||||
|
Punycode-decoded and validated, and is rejected unless it is the
|
||||||
|
canonical A-label for the decoded U-label (:rfc:`5891` §5.3). Labels
|
||||||
|
that are already Unicode (or plain ASCII without the ACE prefix) are
|
||||||
|
validated and returned as a Unicode string.
|
||||||
|
|
||||||
|
:param label: The label to convert. ``bytes`` or ``bytearray`` input
|
||||||
|
is treated as ASCII.
|
||||||
|
:returns: The U-label as a Unicode string.
|
||||||
|
:raises IDNAError: If the label is malformed or fails validation.
|
||||||
|
"""
|
||||||
|
if len(label) > _max_input_length:
|
||||||
|
raise IDNAError("Label too long", code="input_too_long")
|
||||||
|
if not isinstance(label, (bytes, bytearray)):
|
||||||
|
try:
|
||||||
|
label_bytes = label.encode("ascii")
|
||||||
|
except UnicodeEncodeError:
|
||||||
|
check_label(label)
|
||||||
|
return label
|
||||||
|
else:
|
||||||
|
label_bytes = bytes(label)
|
||||||
|
if not label_bytes.isascii():
|
||||||
|
raise IDNAError("Invalid ASCII in A-label", code="invalid_ascii")
|
||||||
|
|
||||||
|
label_bytes = label_bytes.lower()
|
||||||
|
if label_bytes.startswith(_alabel_prefix):
|
||||||
|
label_bytes = label_bytes[len(_alabel_prefix) :]
|
||||||
|
if not label_bytes:
|
||||||
|
raise IDNAError("Malformed A-label, no Punycode eligible content found", code="invalid_alabel")
|
||||||
|
if label_bytes.endswith(b"-"):
|
||||||
|
raise IDNAError("A-label must not end with a hyphen", code="invalid_alabel")
|
||||||
|
else:
|
||||||
|
check_label(label_bytes)
|
||||||
|
return label_bytes.decode("ascii")
|
||||||
|
|
||||||
|
try:
|
||||||
|
label = label_bytes.decode("punycode")
|
||||||
|
except UnicodeError as err:
|
||||||
|
raise IDNAError("Invalid A-label", code="invalid_alabel") from err
|
||||||
|
# RFC 5891 §5.3: the label is rejected unless re-encoding the decoded
|
||||||
|
# form reproduces the (lowercased) input. This catches "fake A-labels"
|
||||||
|
# (RFC 5890 §2.3.2.1) such as ``xn---bbk``, a non-canonical Punycode
|
||||||
|
# spelling of ``xn--bbk`` that would otherwise decode to the same
|
||||||
|
# U-label and so display identically to a different wire-format name.
|
||||||
|
if _punycode(label) != label_bytes:
|
||||||
|
raise IDNAError("A-label is not the canonical Punycode encoding of its U-label", code="non_canonical_alabel")
|
||||||
|
check_label(label)
|
||||||
|
return label
|
||||||
|
|
||||||
|
|
||||||
|
def _check_std3(text: str, domain: str, offset: int) -> None:
|
||||||
|
"""Raise if ``text``, a slice of ``domain`` starting at ``offset`` that
|
||||||
|
UTS #46 mapping left unchanged, contains an ASCII character disallowed
|
||||||
|
under ``UseSTD3ASCIIRules``."""
|
||||||
|
match = _std3_disallowed_re.search(text)
|
||||||
|
if match:
|
||||||
|
codepoint = ord(match.group())
|
||||||
|
position = offset + match.start() + 1
|
||||||
|
raise InvalidCodepoint(
|
||||||
|
f"Codepoint {_unot(codepoint)} not allowed at position {position} in {domain!r}",
|
||||||
|
code="uts46_std3",
|
||||||
|
text=domain,
|
||||||
|
codepoint=codepoint,
|
||||||
|
position=position,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _warn_transitional() -> None:
|
||||||
|
warnings.warn(
|
||||||
|
"Transitional processing is deprecated in UTS #46 and has no effect. "
|
||||||
|
"The transitional argument will be removed in a future version.",
|
||||||
|
DeprecationWarning,
|
||||||
|
stacklevel=3,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def uts46_remap(domain: str, std3_rules: bool = True, transitional: bool = False) -> str:
|
||||||
|
"""Apply the UTS #46 character mapping to a domain string.
|
||||||
|
|
||||||
|
Implements the mapping table from `UTS #46 §4
|
||||||
|
<https://www.unicode.org/reports/tr46/>`_: each character is kept,
|
||||||
|
replaced, or rejected based on its status (``V``, ``M``, ``D``,
|
||||||
|
``I``, ``X``). The result is returned in Normalisation Form C.
|
||||||
|
|
||||||
|
:param domain: The full domain name to remap.
|
||||||
|
:param std3_rules: If ``True``, apply UTS #46's ``UseSTD3ASCIIRules``:
|
||||||
|
after mapping, any ASCII character other than a lowercase letter,
|
||||||
|
digit, hyphen or the label separator ``.`` is rejected, whether it
|
||||||
|
appeared in the input or was produced by a mapping (e.g. U+FF01
|
||||||
|
FULLWIDTH EXCLAMATION MARK maps to ``!``). If ``False``, such
|
||||||
|
characters are passed through.
|
||||||
|
:param transitional: Deprecated and ignored. UTS #46 deprecated
|
||||||
|
transitional processing in Unicode 15.1 and deviation (status
|
||||||
|
``D``) codepoints are now always kept, so this has no effect
|
||||||
|
beyond emitting a :class:`DeprecationWarning`. It will be removed
|
||||||
|
in a future version.
|
||||||
|
:returns: The remapped domain, in Normalisation Form C.
|
||||||
|
:raises InvalidCodepoint: If the domain contains a disallowed
|
||||||
|
codepoint under the chosen rules.
|
||||||
|
:raises IDNAError: If ``domain`` exceeds the defensive input length limit.
|
||||||
|
"""
|
||||||
|
if transitional:
|
||||||
|
_warn_transitional()
|
||||||
|
if len(domain) > _max_input_length:
|
||||||
|
raise IDNAError("Domain too long", code="input_too_long")
|
||||||
|
if domain.isascii():
|
||||||
|
# The only ASCII mapping in UTS #46 is upper- to lowercase, and
|
||||||
|
# ASCII is invariant under NFC, so lowercasing is the whole job.
|
||||||
|
result = domain.lower()
|
||||||
|
if std3_rules:
|
||||||
|
_check_std3(result, domain, 0)
|
||||||
|
return result
|
||||||
|
|
||||||
|
from .uts46data import uts46_replacements, uts46_starts, uts46_statuses
|
||||||
|
|
||||||
|
# ``start`` marks the run of unchanged input not yet copied; a run is
|
||||||
|
# only sliced out when a character must be replaced or dropped, so the
|
||||||
|
# common no-change case makes no copy. STD3 is checked per output piece
|
||||||
|
# to report a violation at its input position.
|
||||||
|
output: list[str] = []
|
||||||
|
start = 0
|
||||||
|
for pos, char in enumerate(domain):
|
||||||
|
code_point = ord(char)
|
||||||
|
i = code_point if code_point < 256 else bisect.bisect_right(uts46_starts, code_point) - 1
|
||||||
|
status = uts46_statuses[i]
|
||||||
|
# UTS #46 §4: V valid, D deviation (kept), M mapped, I ignored,
|
||||||
|
# anything else disallowed.
|
||||||
|
if status == _STATUS_VALID:
|
||||||
|
continue
|
||||||
|
if status == _STATUS_MAPPED:
|
||||||
|
replacement = uts46_replacements[i]
|
||||||
|
elif status == _STATUS_DEVIATION:
|
||||||
|
continue
|
||||||
|
elif status == _STATUS_IGNORED:
|
||||||
|
replacement = None
|
||||||
|
else:
|
||||||
|
raise InvalidCodepoint(
|
||||||
|
f"Codepoint {_unot(code_point)} not allowed at position {pos + 1} in {domain!r}",
|
||||||
|
code="uts46_disallowed",
|
||||||
|
text=domain,
|
||||||
|
codepoint=code_point,
|
||||||
|
position=pos + 1,
|
||||||
|
)
|
||||||
|
if start < pos:
|
||||||
|
run = domain[start:pos]
|
||||||
|
if std3_rules:
|
||||||
|
_check_std3(run, domain, start)
|
||||||
|
output.append(run)
|
||||||
|
if replacement:
|
||||||
|
if std3_rules and _std3_disallowed_re.search(replacement):
|
||||||
|
raise InvalidCodepoint(
|
||||||
|
f"Codepoint {_unot(code_point)} not allowed at position {pos + 1} in {domain!r}",
|
||||||
|
code="uts46_std3",
|
||||||
|
text=domain,
|
||||||
|
codepoint=code_point,
|
||||||
|
position=pos + 1,
|
||||||
|
)
|
||||||
|
output.append(replacement)
|
||||||
|
start = pos + 1
|
||||||
|
|
||||||
|
if start == 0:
|
||||||
|
if std3_rules:
|
||||||
|
_check_std3(domain, domain, 0)
|
||||||
|
return unicodedata.normalize("NFC", domain)
|
||||||
|
tail = domain[start:]
|
||||||
|
if std3_rules:
|
||||||
|
_check_std3(tail, domain, start)
|
||||||
|
output.append(tail)
|
||||||
|
return unicodedata.normalize("NFC", "".join(output))
|
||||||
|
|
||||||
|
|
||||||
|
def encode(
|
||||||
|
s: str | bytes | bytearray,
|
||||||
|
strict: bool = False,
|
||||||
|
uts46: bool = False,
|
||||||
|
std3_rules: bool = False,
|
||||||
|
transitional: bool = False,
|
||||||
|
) -> bytes:
|
||||||
|
"""Encode a Unicode domain name into its ASCII (A-label) form.
|
||||||
|
|
||||||
|
Splits the input on label separators (only ``U+002E`` if ``strict`` is
|
||||||
|
set; otherwise also IDEOGRAPHIC FULL STOP ``U+3002``, FULLWIDTH FULL
|
||||||
|
STOP ``U+FF0E``, and HALFWIDTH IDEOGRAPHIC FULL STOP ``U+FF61``),
|
||||||
|
encodes each label with :func:`alabel`, and rejoins them with ``.``.
|
||||||
|
Optionally pre-processes the input through :func:`uts46_remap`.
|
||||||
|
|
||||||
|
:param s: The domain name to encode.
|
||||||
|
:param strict: If ``True``, only ``U+002E`` is recognised as a label
|
||||||
|
separator.
|
||||||
|
:param uts46: If ``True``, apply UTS #46 mapping before encoding.
|
||||||
|
:param std3_rules: Forwarded to :func:`uts46_remap` when ``uts46`` is
|
||||||
|
``True``.
|
||||||
|
:param transitional: Deprecated and ignored (see :func:`uts46_remap`):
|
||||||
|
emits a :class:`DeprecationWarning` and will be removed in a
|
||||||
|
future version.
|
||||||
|
:returns: The encoded domain as ASCII :class:`bytes`.
|
||||||
|
:raises IDNAError: If the domain is empty, contains an invalid label,
|
||||||
|
or exceeds the maximum domain length.
|
||||||
|
"""
|
||||||
|
if transitional:
|
||||||
|
_warn_transitional()
|
||||||
|
if not isinstance(s, str):
|
||||||
|
try:
|
||||||
|
s = str(s, "ascii")
|
||||||
|
except (UnicodeDecodeError, TypeError) as err:
|
||||||
|
raise IDNAError(
|
||||||
|
"should pass a unicode string to the function rather than a byte string.", code="invalid_ascii"
|
||||||
|
) from err
|
||||||
|
if len(s) > _max_input_length:
|
||||||
|
raise IDNAError("Domain too long", code="input_too_long")
|
||||||
|
if uts46:
|
||||||
|
s = uts46_remap(s, std3_rules)
|
||||||
|
|
||||||
|
if not valid_string_length(s, trailing_dot=True):
|
||||||
|
raise IDNAError("Domain too long", code="domain_too_long")
|
||||||
|
|
||||||
|
trailing_dot = False
|
||||||
|
result = []
|
||||||
|
labels = s.split(".") if strict else _unicode_dots_re.split(s)
|
||||||
|
if not labels or labels == [""]:
|
||||||
|
raise IDNAError("Empty domain", code="empty_domain")
|
||||||
|
if labels[-1] == "":
|
||||||
|
del labels[-1]
|
||||||
|
trailing_dot = True
|
||||||
|
for label in labels:
|
||||||
|
s = alabel(label)
|
||||||
|
if s:
|
||||||
|
result.append(s)
|
||||||
|
else:
|
||||||
|
raise IDNAError("Empty label", code="empty_label")
|
||||||
|
if trailing_dot:
|
||||||
|
result.append(b"")
|
||||||
|
s = b".".join(result)
|
||||||
|
if not valid_string_length(s, trailing_dot):
|
||||||
|
raise IDNAError("Domain too long", code="domain_too_long")
|
||||||
|
return s
|
||||||
|
|
||||||
|
|
||||||
|
def decode(
|
||||||
|
s: str | bytes | bytearray,
|
||||||
|
strict: bool = False,
|
||||||
|
uts46: bool = False,
|
||||||
|
std3_rules: bool = False,
|
||||||
|
display: bool = False,
|
||||||
|
) -> str:
|
||||||
|
"""Decode an A-label-encoded domain name back to Unicode.
|
||||||
|
|
||||||
|
Splits the input on label separators (see :func:`encode` for the
|
||||||
|
rules), decodes each label with :func:`ulabel`, and rejoins them
|
||||||
|
with ``.``. Optionally pre-processes the input through
|
||||||
|
:func:`uts46_remap`.
|
||||||
|
|
||||||
|
:param s: The domain name to decode.
|
||||||
|
:param strict: If ``True``, only ``U+002E`` is recognised as a label
|
||||||
|
separator.
|
||||||
|
:param uts46: If ``True``, apply UTS #46 mapping before decoding.
|
||||||
|
:param std3_rules: Forwarded to :func:`uts46_remap` when ``uts46`` is
|
||||||
|
``True``.
|
||||||
|
:param display: If ``True``, any ``xn--`` label that fails IDNA
|
||||||
|
validation is passed through unchanged (lowercased) rather than
|
||||||
|
aborting the whole call. Intended for "decode for display"
|
||||||
|
consumers (e.g. URL libraries, HTTP clients) that want to show
|
||||||
|
the user the label as it appears on the wire when it cannot be
|
||||||
|
rendered as Unicode. Matches the per-label recovery prescribed
|
||||||
|
by UTS #46 §4 and the WHATWG URL "domain to Unicode" algorithm.
|
||||||
|
:returns: The decoded domain as a Unicode string.
|
||||||
|
:raises IDNAError: If the input is not valid ASCII, contains an
|
||||||
|
invalid label, or is empty.
|
||||||
|
"""
|
||||||
|
if not isinstance(s, str):
|
||||||
|
try:
|
||||||
|
s = str(s, "ascii")
|
||||||
|
except (UnicodeDecodeError, TypeError) as err:
|
||||||
|
raise IDNAError("Invalid ASCII in A-label", code="invalid_ascii") from err
|
||||||
|
if len(s) > _max_input_length:
|
||||||
|
raise IDNAError("Domain too long", code="input_too_long")
|
||||||
|
if uts46:
|
||||||
|
s = uts46_remap(s, std3_rules, False)
|
||||||
|
if not valid_string_length(s, trailing_dot=True):
|
||||||
|
raise IDNAError("Domain too long", code="domain_too_long")
|
||||||
|
trailing_dot = False
|
||||||
|
result = []
|
||||||
|
labels = s.split(".") if strict else _unicode_dots_re.split(s)
|
||||||
|
if not labels or labels == [""]:
|
||||||
|
raise IDNAError("Empty domain", code="empty_domain")
|
||||||
|
if not labels[-1]:
|
||||||
|
del labels[-1]
|
||||||
|
trailing_dot = True
|
||||||
|
for label in labels:
|
||||||
|
try:
|
||||||
|
u = ulabel(label)
|
||||||
|
except IDNAError:
|
||||||
|
if display and label[:4].lower() == "xn--":
|
||||||
|
u = label.lower()
|
||||||
|
else:
|
||||||
|
raise
|
||||||
|
if u:
|
||||||
|
result.append(u)
|
||||||
|
else:
|
||||||
|
raise IDNAError("Empty label", code="empty_label")
|
||||||
|
if trailing_dot:
|
||||||
|
result.append("")
|
||||||
|
return ".".join(result)
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,55 @@
|
|||||||
|
"""
|
||||||
|
Given a list of integers, made up of (hopefully) a small number of long runs
|
||||||
|
of consecutive integers, compute a representation of the form
|
||||||
|
((start1, end1), (start2, end2) ...). Then answer the question "was x present
|
||||||
|
in the original list?" in time O(log(# runs)).
|
||||||
|
"""
|
||||||
|
|
||||||
|
import bisect
|
||||||
|
|
||||||
|
|
||||||
|
def intranges_from_list(list_: list[int]) -> tuple[int, ...]:
|
||||||
|
"""Represent a list of integers as a sequence of ranges:
|
||||||
|
((start_0, end_0), (start_1, end_1), ...), such that the original
|
||||||
|
integers are exactly those x such that start_i <= x < end_i for some i.
|
||||||
|
|
||||||
|
Ranges are encoded as single integers (start << 32 | end), not as tuples.
|
||||||
|
"""
|
||||||
|
|
||||||
|
sorted_list = sorted(list_)
|
||||||
|
ranges = []
|
||||||
|
last_write = -1
|
||||||
|
for i in range(len(sorted_list)):
|
||||||
|
if i + 1 < len(sorted_list) and sorted_list[i] == sorted_list[i + 1] - 1:
|
||||||
|
continue
|
||||||
|
current_range = sorted_list[last_write + 1 : i + 1]
|
||||||
|
ranges.append(_encode_range(current_range[0], current_range[-1] + 1))
|
||||||
|
last_write = i
|
||||||
|
|
||||||
|
return tuple(ranges)
|
||||||
|
|
||||||
|
|
||||||
|
def _encode_range(start: int, end: int) -> int:
|
||||||
|
return (start << 32) | end
|
||||||
|
|
||||||
|
|
||||||
|
def _decode_range(r: int) -> tuple[int, int]:
|
||||||
|
return (r >> 32), (r & ((1 << 32) - 1))
|
||||||
|
|
||||||
|
|
||||||
|
def intranges_contain(int_: int, ranges: tuple[int, ...]) -> bool:
|
||||||
|
"""Determine if `int_` falls into one of the ranges in `ranges`."""
|
||||||
|
tuple_ = _encode_range(int_, 0)
|
||||||
|
pos = bisect.bisect_left(ranges, tuple_)
|
||||||
|
# we could be immediately ahead of a tuple (start, end)
|
||||||
|
# with start < int_ <= end
|
||||||
|
if pos > 0:
|
||||||
|
left, right = _decode_range(ranges[pos - 1])
|
||||||
|
if left <= int_ < right:
|
||||||
|
return True
|
||||||
|
# or we could be immediately behind a tuple (int_, end)
|
||||||
|
if pos < len(ranges):
|
||||||
|
left, _ = _decode_range(ranges[pos])
|
||||||
|
if left == int_:
|
||||||
|
return True
|
||||||
|
return False
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
__version__ = "3.19"
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,219 @@
|
|||||||
|
# __
|
||||||
|
# /__) _ _ _ _ _/ _
|
||||||
|
# / ( (- (/ (/ (- _) / _)
|
||||||
|
# /
|
||||||
|
|
||||||
|
"""
|
||||||
|
Requests HTTP Library
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Requests is an HTTP library, written in Python, for human beings.
|
||||||
|
Basic GET usage:
|
||||||
|
|
||||||
|
>>> import requests
|
||||||
|
>>> r = requests.get('https://www.python.org')
|
||||||
|
>>> r.status_code
|
||||||
|
200
|
||||||
|
>>> b'Python is a programming language' in r.content
|
||||||
|
True
|
||||||
|
|
||||||
|
... or POST:
|
||||||
|
|
||||||
|
>>> payload = dict(key1='value1', key2='value2')
|
||||||
|
>>> r = requests.post('https://httpbin.org/post', data=payload)
|
||||||
|
>>> print(r.text)
|
||||||
|
{
|
||||||
|
...
|
||||||
|
"form": {
|
||||||
|
"key1": "value1",
|
||||||
|
"key2": "value2"
|
||||||
|
},
|
||||||
|
...
|
||||||
|
}
|
||||||
|
|
||||||
|
The other HTTP methods are supported - see `requests.api`. Full documentation
|
||||||
|
is at <https://requests.readthedocs.io>.
|
||||||
|
|
||||||
|
:copyright: (c) 2017 by Kenneth Reitz.
|
||||||
|
:license: Apache 2.0, see LICENSE for more details.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import warnings
|
||||||
|
|
||||||
|
import urllib3
|
||||||
|
|
||||||
|
from .exceptions import RequestsDependencyWarning
|
||||||
|
|
||||||
|
try:
|
||||||
|
from charset_normalizer import __version__ as charset_normalizer_version
|
||||||
|
except ImportError:
|
||||||
|
charset_normalizer_version = None
|
||||||
|
|
||||||
|
try:
|
||||||
|
from chardet import __version__ as chardet_version # type: ignore[import-not-found]
|
||||||
|
except ImportError:
|
||||||
|
chardet_version = None
|
||||||
|
|
||||||
|
|
||||||
|
def check_compatibility(
|
||||||
|
urllib3_version: str,
|
||||||
|
chardet_version: str | None,
|
||||||
|
charset_normalizer_version: str | None,
|
||||||
|
) -> None:
|
||||||
|
urllib3_version_list = urllib3_version.split(".")[:3]
|
||||||
|
assert urllib3_version_list != ["dev"] # Verify urllib3 isn't installed from git.
|
||||||
|
|
||||||
|
# Sometimes, urllib3 only reports its version as 16.1.
|
||||||
|
if len(urllib3_version_list) == 2:
|
||||||
|
urllib3_version_list.append("0")
|
||||||
|
|
||||||
|
# Check urllib3 for compatibility.
|
||||||
|
major, minor, patch = urllib3_version_list # noqa: F811
|
||||||
|
major, minor, patch = int(major), int(minor), int(patch)
|
||||||
|
# urllib3 >= 1.21.1
|
||||||
|
assert major >= 1
|
||||||
|
if major == 1:
|
||||||
|
assert minor >= 21
|
||||||
|
|
||||||
|
# Check charset_normalizer for compatibility.
|
||||||
|
if chardet_version:
|
||||||
|
major, minor, patch = chardet_version.split(".")[:3]
|
||||||
|
major, minor, patch = int(major), int(minor), int(patch)
|
||||||
|
# chardet_version >= 3.0.2, < 8.0.0
|
||||||
|
assert (3, 0, 2) <= (major, minor, patch) < (8, 0, 0)
|
||||||
|
elif charset_normalizer_version:
|
||||||
|
major, minor, patch = charset_normalizer_version.split(".")[:3]
|
||||||
|
major, minor, patch = int(major), int(minor), int(patch)
|
||||||
|
# charset_normalizer >= 2.0.0 < 4.0.0
|
||||||
|
assert (2, 0, 0) <= (major, minor, patch) < (4, 0, 0)
|
||||||
|
else:
|
||||||
|
warnings.warn(
|
||||||
|
"Unable to find acceptable character detection dependency "
|
||||||
|
"(chardet or charset_normalizer).",
|
||||||
|
RequestsDependencyWarning,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _check_cryptography(cryptography_version: str) -> None:
|
||||||
|
# cryptography < 1.3.4
|
||||||
|
try:
|
||||||
|
cryptography_version_list = list(map(int, cryptography_version.split(".")))
|
||||||
|
except ValueError:
|
||||||
|
return
|
||||||
|
|
||||||
|
if cryptography_version_list < [1, 3, 4]:
|
||||||
|
warning = f"Old version of cryptography ({cryptography_version_list}) may cause slowdown."
|
||||||
|
warnings.warn(warning, RequestsDependencyWarning)
|
||||||
|
|
||||||
|
|
||||||
|
# Check imported dependencies for compatibility.
|
||||||
|
try:
|
||||||
|
check_compatibility(
|
||||||
|
urllib3.__version__, # type: ignore[reportPrivateImportUsage]
|
||||||
|
chardet_version, # type: ignore[reportUnknownArgumentType]
|
||||||
|
charset_normalizer_version,
|
||||||
|
)
|
||||||
|
except (AssertionError, ValueError):
|
||||||
|
warnings.warn(
|
||||||
|
f"urllib3 ({urllib3.__version__}) or chardet " # type: ignore[reportPrivateImportUsage]
|
||||||
|
f"({chardet_version})/charset_normalizer ({charset_normalizer_version}) "
|
||||||
|
"doesn't match a supported version!",
|
||||||
|
RequestsDependencyWarning,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Attempt to enable urllib3's fallback for SNI support
|
||||||
|
# if the standard library doesn't support SNI or the
|
||||||
|
# 'ssl' library isn't available.
|
||||||
|
try:
|
||||||
|
try:
|
||||||
|
import ssl
|
||||||
|
except ImportError:
|
||||||
|
ssl = None
|
||||||
|
|
||||||
|
if not getattr(ssl, "HAS_SNI", False):
|
||||||
|
from urllib3.contrib import pyopenssl
|
||||||
|
|
||||||
|
pyopenssl.inject_into_urllib3()
|
||||||
|
|
||||||
|
# Check cryptography version
|
||||||
|
from cryptography import ( # type: ignore[reportMissingImports]
|
||||||
|
__version__ as cryptography_version, # type: ignore[reportUnknownVariableType]
|
||||||
|
)
|
||||||
|
|
||||||
|
_check_cryptography(cryptography_version) # type: ignore[reportUnknownArgumentType]
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# urllib3's DependencyWarnings should be silenced.
|
||||||
|
from urllib3.exceptions import DependencyWarning
|
||||||
|
|
||||||
|
warnings.simplefilter("ignore", DependencyWarning)
|
||||||
|
|
||||||
|
# Set default logging handler to avoid "No handler found" warnings.
|
||||||
|
import logging
|
||||||
|
from logging import NullHandler
|
||||||
|
|
||||||
|
from . import packages, utils
|
||||||
|
from .__version__ import (
|
||||||
|
__author__,
|
||||||
|
__author_email__,
|
||||||
|
__build__,
|
||||||
|
__cake__,
|
||||||
|
__copyright__,
|
||||||
|
__description__,
|
||||||
|
__license__,
|
||||||
|
__title__,
|
||||||
|
__url__,
|
||||||
|
__version__,
|
||||||
|
)
|
||||||
|
from .api import delete, get, head, options, patch, post, put, request
|
||||||
|
from .exceptions import (
|
||||||
|
ConnectionError,
|
||||||
|
ConnectTimeout,
|
||||||
|
FileModeWarning,
|
||||||
|
HTTPError,
|
||||||
|
JSONDecodeError,
|
||||||
|
ReadTimeout,
|
||||||
|
RequestException,
|
||||||
|
Timeout,
|
||||||
|
TooManyRedirects,
|
||||||
|
URLRequired,
|
||||||
|
)
|
||||||
|
from .models import PreparedRequest, Request, Response
|
||||||
|
from .sessions import Session, session
|
||||||
|
from .status_codes import codes
|
||||||
|
|
||||||
|
__all__ = (
|
||||||
|
"ConnectionError",
|
||||||
|
"ConnectTimeout",
|
||||||
|
"HTTPError",
|
||||||
|
"JSONDecodeError",
|
||||||
|
"PreparedRequest",
|
||||||
|
"ReadTimeout",
|
||||||
|
"Request",
|
||||||
|
"RequestException",
|
||||||
|
"Response",
|
||||||
|
"Session",
|
||||||
|
"Timeout",
|
||||||
|
"TooManyRedirects",
|
||||||
|
"URLRequired",
|
||||||
|
"codes",
|
||||||
|
"delete",
|
||||||
|
"get",
|
||||||
|
"head",
|
||||||
|
"options",
|
||||||
|
"packages",
|
||||||
|
"patch",
|
||||||
|
"post",
|
||||||
|
"put",
|
||||||
|
"request",
|
||||||
|
"session",
|
||||||
|
"utils",
|
||||||
|
)
|
||||||
|
|
||||||
|
logging.getLogger(__name__).addHandler(NullHandler())
|
||||||
|
|
||||||
|
# FileModeWarnings go off per the default.
|
||||||
|
warnings.simplefilter("default", FileModeWarning, append=True)
|
||||||
+14
@@ -0,0 +1,14 @@
|
|||||||
|
# .-. .-. .-. . . .-. .-. .-. .-.
|
||||||
|
# |( |- |.| | | |- `-. | `-.
|
||||||
|
# ' ' `-' `-`.`-' `-' `-' ' `-'
|
||||||
|
|
||||||
|
__title__ = "requests"
|
||||||
|
__description__ = "Python HTTP for Humans."
|
||||||
|
__url__ = "https://requests.readthedocs.io"
|
||||||
|
__version__ = "2.34.2"
|
||||||
|
__build__ = 0x023402
|
||||||
|
__author__ = "Kenneth Reitz"
|
||||||
|
__author_email__ = "[email protected]"
|
||||||
|
__license__ = "Apache-2.0"
|
||||||
|
__copyright__ = "Copyright Kenneth Reitz"
|
||||||
|
__cake__ = "\u2728 \U0001f370 \u2728"
|
||||||
+51
@@ -0,0 +1,51 @@
|
|||||||
|
"""
|
||||||
|
requests._internal_utils
|
||||||
|
~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Provides utility functions that are consumed internally by Requests
|
||||||
|
which depend on extremely few external helpers (such as compat)
|
||||||
|
"""
|
||||||
|
|
||||||
|
import re
|
||||||
|
|
||||||
|
from .compat import builtin_str
|
||||||
|
|
||||||
|
_VALID_HEADER_NAME_RE_BYTE = re.compile(rb"^[^:\s][^:\r\n]*\Z")
|
||||||
|
_VALID_HEADER_NAME_RE_STR = re.compile(r"^[^:\s][^:\r\n]*\Z")
|
||||||
|
_VALID_HEADER_VALUE_RE_BYTE = re.compile(rb"^\S[^\r\n]*\Z|^\Z")
|
||||||
|
_VALID_HEADER_VALUE_RE_STR = re.compile(r"^\S[^\r\n]*\Z|^\Z")
|
||||||
|
|
||||||
|
_HEADER_VALIDATORS_STR = (_VALID_HEADER_NAME_RE_STR, _VALID_HEADER_VALUE_RE_STR)
|
||||||
|
_HEADER_VALIDATORS_BYTE = (_VALID_HEADER_NAME_RE_BYTE, _VALID_HEADER_VALUE_RE_BYTE)
|
||||||
|
HEADER_VALIDATORS = {
|
||||||
|
bytes: _HEADER_VALIDATORS_BYTE,
|
||||||
|
str: _HEADER_VALIDATORS_STR,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def to_native_string(string: str | bytes, encoding: str = "ascii") -> str:
|
||||||
|
"""Given a string object, regardless of type, returns a representation of
|
||||||
|
that string in the native string type, encoding and decoding where
|
||||||
|
necessary. This assumes ASCII unless told otherwise.
|
||||||
|
"""
|
||||||
|
if isinstance(string, builtin_str):
|
||||||
|
out = string
|
||||||
|
else:
|
||||||
|
out = string.decode(encoding)
|
||||||
|
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def unicode_is_ascii(u_string: str) -> bool:
|
||||||
|
"""Determine if unicode string only contains ASCII characters.
|
||||||
|
|
||||||
|
:param str u_string: unicode string to check. Must be unicode
|
||||||
|
and not Python 2 `str`.
|
||||||
|
:rtype: bool
|
||||||
|
"""
|
||||||
|
assert isinstance(u_string, str)
|
||||||
|
try:
|
||||||
|
u_string.encode("ascii")
|
||||||
|
return True
|
||||||
|
except UnicodeEncodeError:
|
||||||
|
return False
|
||||||
@@ -0,0 +1,183 @@
|
|||||||
|
"""
|
||||||
|
requests._types
|
||||||
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This module contains type aliases used internally by the Requests library.
|
||||||
|
These types are not part of the public API and must not be relied upon
|
||||||
|
by external code.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable, Iterable, Mapping, MutableMapping, Sequence
|
||||||
|
from typing import (
|
||||||
|
TYPE_CHECKING,
|
||||||
|
Any,
|
||||||
|
Protocol,
|
||||||
|
TypeAlias,
|
||||||
|
TypeVar,
|
||||||
|
runtime_checkable,
|
||||||
|
)
|
||||||
|
|
||||||
|
_T_co = TypeVar("_T_co", covariant=True)
|
||||||
|
_KT_co = TypeVar("_KT_co", covariant=True)
|
||||||
|
_VT_co = TypeVar("_VT_co", covariant=True)
|
||||||
|
|
||||||
|
|
||||||
|
@runtime_checkable
|
||||||
|
class SupportsRead(Protocol[_T_co]):
|
||||||
|
def read(self, length: int = ..., /) -> _T_co: ...
|
||||||
|
|
||||||
|
|
||||||
|
@runtime_checkable
|
||||||
|
class SupportsItems(Protocol[_KT_co, _VT_co]):
|
||||||
|
def items(self) -> Iterable[tuple[_KT_co, _VT_co]]: ...
|
||||||
|
|
||||||
|
|
||||||
|
# These are needed at runtime for default_hooks() return type
|
||||||
|
HookType: TypeAlias = Callable[["Response"], Any]
|
||||||
|
HooksInputType: TypeAlias = Mapping[str, Iterable[HookType] | HookType]
|
||||||
|
|
||||||
|
|
||||||
|
def is_prepared(request: PreparedRequest) -> TypeIs[_ValidatedRequest]:
|
||||||
|
"""Verify a PreparedRequest has been fully prepared."""
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
return request.url is not None and request.method is not None
|
||||||
|
# noop at runtime to avoid AssertionError
|
||||||
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from http.cookiejar import CookieJar
|
||||||
|
from typing import TypeAlias, TypedDict
|
||||||
|
|
||||||
|
from typing_extensions import (
|
||||||
|
Buffer, # TODO: move to collections.abc when Python >= 3.12
|
||||||
|
TypeIs, # TODO: move to typing when Python >= 3.13
|
||||||
|
)
|
||||||
|
|
||||||
|
from .auth import AuthBase
|
||||||
|
from .cookies import RequestsCookieJar
|
||||||
|
from .models import PreparedRequest, Response
|
||||||
|
from .structures import CaseInsensitiveDict
|
||||||
|
|
||||||
|
class _ValidatedRequest(PreparedRequest):
|
||||||
|
"""Subtype asserting a PreparedRequest has been fully prepared before calling.
|
||||||
|
|
||||||
|
The override suppression is required because mutable attribute types are
|
||||||
|
invariant (Liskov), but we only narrow after preparation is complete. This
|
||||||
|
is the explicit contract for Requests but Python's typing doesn't have a
|
||||||
|
better way to represent the requirement.
|
||||||
|
"""
|
||||||
|
|
||||||
|
url: str # type: ignore[reportIncompatibleVariableOverride]
|
||||||
|
method: str # type: ignore[reportIncompatibleVariableOverride]
|
||||||
|
|
||||||
|
# Type aliases for core API concepts (ordered by request() signature)
|
||||||
|
UriType: TypeAlias = str | bytes
|
||||||
|
|
||||||
|
_ParamsMappingKeyType: TypeAlias = str | bytes | int | float
|
||||||
|
_ParamsMappingValueType: TypeAlias = (
|
||||||
|
str | bytes | int | float | Iterable[str | bytes | int | float] | None
|
||||||
|
)
|
||||||
|
ParamsType: TypeAlias = (
|
||||||
|
SupportsItems[_ParamsMappingKeyType, _ParamsMappingValueType]
|
||||||
|
| tuple[tuple[_ParamsMappingKeyType, _ParamsMappingValueType], ...]
|
||||||
|
| Iterable[tuple[_ParamsMappingKeyType, _ParamsMappingValueType]]
|
||||||
|
| str
|
||||||
|
| bytes
|
||||||
|
| None
|
||||||
|
)
|
||||||
|
|
||||||
|
KVDataType: TypeAlias = Iterable[tuple[Any, Any]] | SupportsItems[Any, Any]
|
||||||
|
|
||||||
|
RawDataType: TypeAlias = KVDataType | str | bytes
|
||||||
|
StreamDataType: TypeAlias = SupportsRead[str | bytes]
|
||||||
|
EncodableDataType: TypeAlias = RawDataType | StreamDataType
|
||||||
|
|
||||||
|
DataType: TypeAlias = (
|
||||||
|
KVDataType
|
||||||
|
| Iterable[bytes | str]
|
||||||
|
| str
|
||||||
|
| bytes
|
||||||
|
| Buffer
|
||||||
|
| SupportsRead[str | bytes]
|
||||||
|
| None
|
||||||
|
)
|
||||||
|
|
||||||
|
BodyType: TypeAlias = (
|
||||||
|
bytes | str | Iterable[bytes | str] | SupportsRead[bytes | str] | None
|
||||||
|
)
|
||||||
|
|
||||||
|
HeadersType: TypeAlias = Mapping[str, str | bytes] | None
|
||||||
|
|
||||||
|
CookiesType: TypeAlias = RequestsCookieJar | Mapping[str, str]
|
||||||
|
|
||||||
|
# Building blocks for FilesType
|
||||||
|
_FileName: TypeAlias = str | None
|
||||||
|
_FileContent: TypeAlias = SupportsRead[str | bytes] | str | bytes
|
||||||
|
_FileSpecBasic: TypeAlias = tuple[_FileName, _FileContent]
|
||||||
|
_FileSpecWithContentType: TypeAlias = tuple[_FileName, _FileContent, str]
|
||||||
|
_FileSpecWithHeaders: TypeAlias = tuple[
|
||||||
|
_FileName, _FileContent, str, CaseInsensitiveDict[str] | Mapping[str, str]
|
||||||
|
]
|
||||||
|
_FileSpec: TypeAlias = (
|
||||||
|
_FileContent | _FileSpecBasic | _FileSpecWithContentType | _FileSpecWithHeaders
|
||||||
|
)
|
||||||
|
FilesType: TypeAlias = (
|
||||||
|
Mapping[str, _FileSpec] | Iterable[tuple[str, _FileSpec]] | None
|
||||||
|
)
|
||||||
|
|
||||||
|
AuthType: TypeAlias = (
|
||||||
|
tuple[str, str] | AuthBase | Callable[[PreparedRequest], PreparedRequest] | None
|
||||||
|
)
|
||||||
|
|
||||||
|
TimeoutType: TypeAlias = float | tuple[float | None, float | None] | None
|
||||||
|
ProxiesType: TypeAlias = MutableMapping[str, str]
|
||||||
|
HooksType: TypeAlias = dict[str, list[HookType]] | None
|
||||||
|
VerifyType: TypeAlias = bool | str
|
||||||
|
CertType: TypeAlias = str | tuple[str, str] | None
|
||||||
|
JsonType: TypeAlias = (
|
||||||
|
None
|
||||||
|
| bool
|
||||||
|
| int
|
||||||
|
| float
|
||||||
|
| str
|
||||||
|
| Sequence["JsonType"]
|
||||||
|
| Mapping[str, "JsonType"]
|
||||||
|
)
|
||||||
|
|
||||||
|
# TypedDicts for Unpack kwargs (PEP 692)
|
||||||
|
|
||||||
|
class BaseRequestKwargs(TypedDict, total=False):
|
||||||
|
headers: HeadersType
|
||||||
|
cookies: RequestsCookieJar | CookieJar | dict[str, str] | None
|
||||||
|
files: FilesType
|
||||||
|
auth: AuthType
|
||||||
|
timeout: TimeoutType
|
||||||
|
allow_redirects: bool
|
||||||
|
proxies: dict[str, str] | None
|
||||||
|
hooks: HooksInputType | None
|
||||||
|
stream: bool | None
|
||||||
|
verify: VerifyType | None
|
||||||
|
cert: CertType
|
||||||
|
|
||||||
|
class RequestKwargs(BaseRequestKwargs, total=False):
|
||||||
|
"""kwargs for request(), options(), head(), delete()."""
|
||||||
|
|
||||||
|
params: ParamsType
|
||||||
|
data: DataType
|
||||||
|
json: JsonType
|
||||||
|
|
||||||
|
class GetKwargs(BaseRequestKwargs, total=False):
|
||||||
|
data: DataType
|
||||||
|
json: JsonType
|
||||||
|
|
||||||
|
class PostKwargs(BaseRequestKwargs, total=False):
|
||||||
|
params: ParamsType
|
||||||
|
|
||||||
|
class DataKwargs(BaseRequestKwargs, total=False):
|
||||||
|
"""kwargs for put(), patch()."""
|
||||||
|
|
||||||
|
params: ParamsType
|
||||||
|
json: JsonType
|
||||||
@@ -0,0 +1,748 @@
|
|||||||
|
"""
|
||||||
|
requests.adapters
|
||||||
|
~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This module contains the transport adapters that Requests uses to define
|
||||||
|
and maintain connections.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os.path
|
||||||
|
import socket # noqa: F401 # type: ignore[reportUnusedImport]
|
||||||
|
import typing
|
||||||
|
import warnings
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from urllib3.exceptions import (
|
||||||
|
ClosedPoolError,
|
||||||
|
ConnectTimeoutError,
|
||||||
|
LocationValueError,
|
||||||
|
MaxRetryError,
|
||||||
|
NewConnectionError,
|
||||||
|
ProtocolError,
|
||||||
|
ReadTimeoutError,
|
||||||
|
ResponseError,
|
||||||
|
)
|
||||||
|
from urllib3.exceptions import HTTPError as _HTTPError
|
||||||
|
from urllib3.exceptions import InvalidHeader as _InvalidHeader
|
||||||
|
from urllib3.exceptions import ProxyError as _ProxyError
|
||||||
|
from urllib3.exceptions import SSLError as _SSLError
|
||||||
|
from urllib3.poolmanager import PoolManager, proxy_from_url
|
||||||
|
from urllib3.util import Timeout as TimeoutSauce
|
||||||
|
from urllib3.util import parse_url
|
||||||
|
from urllib3.util.retry import Retry
|
||||||
|
|
||||||
|
from .auth import _basic_auth_str # type: ignore[reportPrivateUsage]
|
||||||
|
from .compat import basestring, urlparse
|
||||||
|
from .cookies import extract_cookies_to_jar
|
||||||
|
from .exceptions import (
|
||||||
|
ConnectionError,
|
||||||
|
ConnectTimeout,
|
||||||
|
InvalidHeader,
|
||||||
|
InvalidProxyURL,
|
||||||
|
InvalidSchema,
|
||||||
|
InvalidURL,
|
||||||
|
ProxyError,
|
||||||
|
ReadTimeout,
|
||||||
|
RetryError,
|
||||||
|
SSLError,
|
||||||
|
)
|
||||||
|
from .models import Response
|
||||||
|
from .structures import CaseInsensitiveDict
|
||||||
|
from .utils import (
|
||||||
|
DEFAULT_CA_BUNDLE_PATH,
|
||||||
|
get_auth_from_url,
|
||||||
|
get_encoding_from_headers,
|
||||||
|
prepend_scheme_if_needed,
|
||||||
|
select_proxy,
|
||||||
|
urldefragauth,
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
from urllib3.contrib.socks import SOCKSProxyManager # type: ignore[assignment]
|
||||||
|
except ImportError:
|
||||||
|
|
||||||
|
def SOCKSProxyManager(*args: Any, **kwargs: Any) -> None:
|
||||||
|
raise InvalidSchema("Missing dependencies for SOCKS support.")
|
||||||
|
|
||||||
|
|
||||||
|
if typing.TYPE_CHECKING:
|
||||||
|
from urllib3.connectionpool import HTTPConnectionPool
|
||||||
|
from urllib3.poolmanager import PoolManager as _PoolManager
|
||||||
|
|
||||||
|
from . import _types as _t
|
||||||
|
from .models import PreparedRequest
|
||||||
|
|
||||||
|
from ._types import is_prepared as _is_prepared
|
||||||
|
|
||||||
|
DEFAULT_POOLBLOCK = False
|
||||||
|
DEFAULT_POOLSIZE = 10
|
||||||
|
DEFAULT_RETRIES = 0
|
||||||
|
DEFAULT_POOL_TIMEOUT = None
|
||||||
|
|
||||||
|
|
||||||
|
def _urllib3_request_context(
|
||||||
|
request: PreparedRequest,
|
||||||
|
verify: bool | str | None,
|
||||||
|
client_cert: tuple[str, str] | str | None,
|
||||||
|
poolmanager: PoolManager,
|
||||||
|
) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||||
|
host_params: dict[str, Any] = {}
|
||||||
|
pool_kwargs: dict[str, Any] = {}
|
||||||
|
parsed_request_url = urlparse(request.url)
|
||||||
|
scheme = parsed_request_url.scheme.lower()
|
||||||
|
port = parsed_request_url.port
|
||||||
|
|
||||||
|
cert_reqs = "CERT_REQUIRED"
|
||||||
|
if verify is False:
|
||||||
|
cert_reqs = "CERT_NONE"
|
||||||
|
elif isinstance(verify, str):
|
||||||
|
if not os.path.isdir(verify):
|
||||||
|
pool_kwargs["ca_certs"] = verify
|
||||||
|
else:
|
||||||
|
pool_kwargs["ca_cert_dir"] = verify
|
||||||
|
pool_kwargs["cert_reqs"] = cert_reqs
|
||||||
|
if client_cert is not None:
|
||||||
|
if isinstance(client_cert, tuple) and len(client_cert) == 2:
|
||||||
|
pool_kwargs["cert_file"] = client_cert[0]
|
||||||
|
pool_kwargs["key_file"] = client_cert[1]
|
||||||
|
else:
|
||||||
|
# According to our docs, we allow users to specify just the client
|
||||||
|
# cert path
|
||||||
|
pool_kwargs["cert_file"] = client_cert
|
||||||
|
host_params = {
|
||||||
|
"scheme": scheme,
|
||||||
|
"host": parsed_request_url.hostname,
|
||||||
|
"port": port,
|
||||||
|
}
|
||||||
|
return host_params, pool_kwargs
|
||||||
|
|
||||||
|
|
||||||
|
class BaseAdapter:
|
||||||
|
"""The Base Transport Adapter"""
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
def send(
|
||||||
|
self,
|
||||||
|
request: PreparedRequest,
|
||||||
|
stream: bool = False,
|
||||||
|
timeout: _t.TimeoutType = None,
|
||||||
|
verify: _t.VerifyType = True,
|
||||||
|
cert: _t.CertType = None,
|
||||||
|
proxies: dict[str, str] | None = None,
|
||||||
|
) -> Response:
|
||||||
|
"""Sends PreparedRequest object. Returns Response object.
|
||||||
|
|
||||||
|
:param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
|
||||||
|
:param stream: (optional) Whether to stream the request content.
|
||||||
|
:param timeout: (optional) How long to wait for the server to send
|
||||||
|
data before giving up, as a float, or a :ref:`(connect timeout,
|
||||||
|
read timeout) <timeouts>` tuple.
|
||||||
|
:type timeout: float or tuple
|
||||||
|
:param verify: (optional) Either a boolean, in which case it controls whether we verify
|
||||||
|
the server's TLS certificate, or a string, in which case it must be a path
|
||||||
|
to a CA bundle to use
|
||||||
|
:param cert: (optional) Any user-provided SSL certificate to be trusted.
|
||||||
|
:param proxies: (optional) The proxies dictionary to apply to the request.
|
||||||
|
"""
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
"""Cleans up adapter specific items."""
|
||||||
|
raise NotImplementedError
|
||||||
|
|
||||||
|
|
||||||
|
class HTTPAdapter(BaseAdapter):
|
||||||
|
"""The built-in HTTP Adapter for urllib3.
|
||||||
|
|
||||||
|
Provides a general-case interface for Requests sessions to contact HTTP and
|
||||||
|
HTTPS urls by implementing the Transport Adapter interface. This class will
|
||||||
|
usually be created by the :class:`Session <Session>` class under the
|
||||||
|
covers.
|
||||||
|
|
||||||
|
:param pool_connections: The number of urllib3 connection pools to cache.
|
||||||
|
:param pool_maxsize: The maximum number of connections to save in the pool.
|
||||||
|
:param max_retries: The maximum number of retries each connection
|
||||||
|
should attempt. Note, this applies only to failed DNS lookups, socket
|
||||||
|
connections and connection timeouts, never to requests where data has
|
||||||
|
made it to the server. By default, Requests does not retry failed
|
||||||
|
connections. If you need granular control over the conditions under
|
||||||
|
which we retry a request, import urllib3's ``Retry`` class and pass
|
||||||
|
that instead.
|
||||||
|
:param pool_block: Whether the connection pool should block for connections.
|
||||||
|
|
||||||
|
Usage::
|
||||||
|
|
||||||
|
>>> import requests
|
||||||
|
>>> s = requests.Session()
|
||||||
|
>>> a = requests.adapters.HTTPAdapter(max_retries=3)
|
||||||
|
>>> s.mount('http://', a)
|
||||||
|
"""
|
||||||
|
|
||||||
|
__attrs__: list[str] = [
|
||||||
|
"max_retries",
|
||||||
|
"config",
|
||||||
|
"_pool_connections",
|
||||||
|
"_pool_maxsize",
|
||||||
|
"_pool_block",
|
||||||
|
]
|
||||||
|
|
||||||
|
max_retries: Retry
|
||||||
|
config: dict[str, Any]
|
||||||
|
proxy_manager: dict[str, Any]
|
||||||
|
_pool_connections: int
|
||||||
|
_pool_maxsize: int
|
||||||
|
_pool_block: bool
|
||||||
|
poolmanager: _PoolManager
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
pool_connections: int = DEFAULT_POOLSIZE,
|
||||||
|
pool_maxsize: int = DEFAULT_POOLSIZE,
|
||||||
|
max_retries: int | Retry = DEFAULT_RETRIES,
|
||||||
|
pool_block: bool = DEFAULT_POOLBLOCK,
|
||||||
|
) -> None:
|
||||||
|
if max_retries == DEFAULT_RETRIES:
|
||||||
|
self.max_retries = Retry(0, read=False)
|
||||||
|
else:
|
||||||
|
self.max_retries = Retry.from_int(max_retries)
|
||||||
|
self.config = {}
|
||||||
|
self.proxy_manager = {}
|
||||||
|
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
self._pool_connections = pool_connections
|
||||||
|
self._pool_maxsize = pool_maxsize
|
||||||
|
self._pool_block = pool_block
|
||||||
|
|
||||||
|
self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block)
|
||||||
|
|
||||||
|
def __getstate__(self) -> dict[str, Any]:
|
||||||
|
return {attr: getattr(self, attr, None) for attr in self.__attrs__}
|
||||||
|
|
||||||
|
def __setstate__(self, state: dict[str, Any]) -> None:
|
||||||
|
# Can't handle by adding 'proxy_manager' to self.__attrs__ because
|
||||||
|
# self.poolmanager uses a lambda function, which isn't pickleable.
|
||||||
|
self.proxy_manager = {}
|
||||||
|
self.config = {}
|
||||||
|
|
||||||
|
for attr, value in state.items():
|
||||||
|
setattr(self, attr, value)
|
||||||
|
|
||||||
|
self.init_poolmanager(
|
||||||
|
self._pool_connections, self._pool_maxsize, block=self._pool_block
|
||||||
|
)
|
||||||
|
|
||||||
|
def init_poolmanager(
|
||||||
|
self,
|
||||||
|
connections: int,
|
||||||
|
maxsize: int,
|
||||||
|
block: bool = DEFAULT_POOLBLOCK,
|
||||||
|
**pool_kwargs: Any,
|
||||||
|
) -> None:
|
||||||
|
"""Initializes a urllib3 PoolManager.
|
||||||
|
|
||||||
|
This method should not be called from user code, and is only
|
||||||
|
exposed for use when subclassing the
|
||||||
|
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
|
||||||
|
|
||||||
|
:param connections: The number of urllib3 connection pools to cache.
|
||||||
|
:param maxsize: The maximum number of connections to save in the pool.
|
||||||
|
:param block: Block when no free connections are available.
|
||||||
|
:param pool_kwargs: Extra keyword arguments used to initialize the Pool Manager.
|
||||||
|
"""
|
||||||
|
# save these values for pickling
|
||||||
|
self._pool_connections = connections
|
||||||
|
self._pool_maxsize = maxsize
|
||||||
|
self._pool_block = block
|
||||||
|
|
||||||
|
self.poolmanager = PoolManager(
|
||||||
|
num_pools=connections,
|
||||||
|
maxsize=maxsize,
|
||||||
|
block=block,
|
||||||
|
**pool_kwargs,
|
||||||
|
)
|
||||||
|
|
||||||
|
def proxy_manager_for(self, proxy: str, **proxy_kwargs: Any) -> Any:
|
||||||
|
"""Return urllib3 ProxyManager for the given proxy.
|
||||||
|
|
||||||
|
This method should not be called from user code, and is only
|
||||||
|
exposed for use when subclassing the
|
||||||
|
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
|
||||||
|
|
||||||
|
:param proxy: The proxy to return a urllib3 ProxyManager for.
|
||||||
|
:param proxy_kwargs: Extra keyword arguments used to configure the Proxy Manager.
|
||||||
|
:returns: ProxyManager
|
||||||
|
:rtype: urllib3.ProxyManager
|
||||||
|
"""
|
||||||
|
if proxy in self.proxy_manager:
|
||||||
|
manager = self.proxy_manager[proxy]
|
||||||
|
elif proxy.lower().startswith("socks"):
|
||||||
|
username, password = get_auth_from_url(proxy)
|
||||||
|
manager = self.proxy_manager[proxy] = SOCKSProxyManager(
|
||||||
|
proxy,
|
||||||
|
username=username,
|
||||||
|
password=password,
|
||||||
|
num_pools=self._pool_connections,
|
||||||
|
maxsize=self._pool_maxsize,
|
||||||
|
block=self._pool_block,
|
||||||
|
**proxy_kwargs,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
proxy_headers = self.proxy_headers(proxy)
|
||||||
|
manager = self.proxy_manager[proxy] = proxy_from_url(
|
||||||
|
proxy,
|
||||||
|
proxy_headers=proxy_headers,
|
||||||
|
num_pools=self._pool_connections,
|
||||||
|
maxsize=self._pool_maxsize,
|
||||||
|
block=self._pool_block,
|
||||||
|
**proxy_kwargs,
|
||||||
|
)
|
||||||
|
|
||||||
|
return manager
|
||||||
|
|
||||||
|
def cert_verify(
|
||||||
|
self, conn: Any, url: str, verify: _t.VerifyType, cert: _t.CertType
|
||||||
|
) -> None:
|
||||||
|
"""Verify a SSL certificate. This method should not be called from user
|
||||||
|
code, and is only exposed for use when subclassing the
|
||||||
|
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
|
||||||
|
|
||||||
|
:param conn: The urllib3 connection object associated with the cert.
|
||||||
|
:param url: The requested URL.
|
||||||
|
:param verify: Either a boolean, in which case it controls whether we verify
|
||||||
|
the server's TLS certificate, or a string, in which case it must be a path
|
||||||
|
to a CA bundle to use
|
||||||
|
:param cert: The SSL certificate to verify.
|
||||||
|
"""
|
||||||
|
if url.lower().startswith("https") and verify:
|
||||||
|
cert_loc = None
|
||||||
|
|
||||||
|
# Allow self-specified cert location.
|
||||||
|
if verify is not True:
|
||||||
|
cert_loc = verify
|
||||||
|
|
||||||
|
if not cert_loc:
|
||||||
|
cert_loc = DEFAULT_CA_BUNDLE_PATH
|
||||||
|
|
||||||
|
if not cert_loc or not os.path.exists(cert_loc):
|
||||||
|
raise OSError(
|
||||||
|
f"Could not find a suitable TLS CA certificate bundle, "
|
||||||
|
f"invalid path: {cert_loc}"
|
||||||
|
)
|
||||||
|
|
||||||
|
conn.cert_reqs = "CERT_REQUIRED"
|
||||||
|
|
||||||
|
if not os.path.isdir(cert_loc):
|
||||||
|
conn.ca_certs = cert_loc
|
||||||
|
else:
|
||||||
|
conn.ca_cert_dir = cert_loc
|
||||||
|
else:
|
||||||
|
conn.cert_reqs = "CERT_NONE"
|
||||||
|
conn.ca_certs = None
|
||||||
|
conn.ca_cert_dir = None
|
||||||
|
|
||||||
|
if cert:
|
||||||
|
if not isinstance(cert, basestring):
|
||||||
|
conn.cert_file = cert[0]
|
||||||
|
conn.key_file = cert[1]
|
||||||
|
else:
|
||||||
|
conn.cert_file = cert
|
||||||
|
conn.key_file = None
|
||||||
|
if conn.cert_file and not os.path.exists(conn.cert_file):
|
||||||
|
raise OSError(
|
||||||
|
f"Could not find the TLS certificate file, "
|
||||||
|
f"invalid path: {conn.cert_file}"
|
||||||
|
)
|
||||||
|
if conn.key_file and not os.path.exists(conn.key_file):
|
||||||
|
raise OSError(
|
||||||
|
f"Could not find the TLS key file, invalid path: {conn.key_file}"
|
||||||
|
)
|
||||||
|
|
||||||
|
def build_response(self, req: PreparedRequest, resp: Any) -> Response:
|
||||||
|
"""Builds a :class:`Response <requests.Response>` object from a urllib3
|
||||||
|
response. This should not be called from user code, and is only exposed
|
||||||
|
for use when subclassing the
|
||||||
|
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`
|
||||||
|
|
||||||
|
:param req: The :class:`PreparedRequest <PreparedRequest>` used to generate the response.
|
||||||
|
:param resp: The urllib3 response object.
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
assert _is_prepared(req)
|
||||||
|
response = Response()
|
||||||
|
|
||||||
|
# Fallback to None if there's no status_code, for whatever reason.
|
||||||
|
response.status_code = getattr(resp, "status", None) # type: ignore[assignment]
|
||||||
|
|
||||||
|
# Make headers case-insensitive.
|
||||||
|
response.headers = CaseInsensitiveDict(getattr(resp, "headers", {}))
|
||||||
|
|
||||||
|
# Set encoding.
|
||||||
|
response.encoding = get_encoding_from_headers(response.headers)
|
||||||
|
response.raw = resp
|
||||||
|
response.reason = response.raw.reason
|
||||||
|
|
||||||
|
if isinstance(req.url, bytes):
|
||||||
|
response.url = req.url.decode("utf-8")
|
||||||
|
else:
|
||||||
|
response.url = req.url
|
||||||
|
|
||||||
|
# Add new cookies from the server.
|
||||||
|
extract_cookies_to_jar(response.cookies, req, resp)
|
||||||
|
|
||||||
|
# Give the Response some context.
|
||||||
|
response.request = req
|
||||||
|
response.connection = self
|
||||||
|
|
||||||
|
return response
|
||||||
|
|
||||||
|
def build_connection_pool_key_attributes(
|
||||||
|
self, request: PreparedRequest, verify: _t.VerifyType, cert: _t.CertType = None
|
||||||
|
) -> tuple[dict[str, Any], dict[str, Any]]:
|
||||||
|
"""Build the PoolKey attributes used by urllib3 to return a connection.
|
||||||
|
|
||||||
|
This looks at the PreparedRequest, the user-specified verify value,
|
||||||
|
and the value of the cert parameter to determine what PoolKey values
|
||||||
|
to use to select a connection from a given urllib3 Connection Pool.
|
||||||
|
|
||||||
|
The SSL related pool key arguments are not consistently set. As of
|
||||||
|
this writing, use the following to determine what keys may be in that
|
||||||
|
dictionary:
|
||||||
|
|
||||||
|
* If ``verify`` is ``True``, ``"ssl_context"`` will be set and will be the
|
||||||
|
default Requests SSL Context
|
||||||
|
* If ``verify`` is ``False``, ``"ssl_context"`` will not be set but
|
||||||
|
``"cert_reqs"`` will be set
|
||||||
|
* If ``verify`` is a string, (i.e., it is a user-specified trust bundle)
|
||||||
|
``"ca_certs"`` will be set if the string is not a directory recognized
|
||||||
|
by :py:func:`os.path.isdir`, otherwise ``"ca_cert_dir"`` will be
|
||||||
|
set.
|
||||||
|
* If ``"cert"`` is specified, ``"cert_file"`` will always be set. If
|
||||||
|
``"cert"`` is a tuple with a second item, ``"key_file"`` will also
|
||||||
|
be present
|
||||||
|
|
||||||
|
To override these settings, one may subclass this class, call this
|
||||||
|
method and use the above logic to change parameters as desired. For
|
||||||
|
example, if one wishes to use a custom :py:class:`ssl.SSLContext` one
|
||||||
|
must both set ``"ssl_context"`` and based on what else they require,
|
||||||
|
alter the other keys to ensure the desired behaviour.
|
||||||
|
|
||||||
|
:param request:
|
||||||
|
The PreparedRequest being sent over the connection.
|
||||||
|
:type request:
|
||||||
|
:class:`~requests.models.PreparedRequest`
|
||||||
|
:param verify:
|
||||||
|
Either a boolean, in which case it controls whether
|
||||||
|
we verify the server's TLS certificate, or a string, in which case it
|
||||||
|
must be a path to a CA bundle to use.
|
||||||
|
:param cert:
|
||||||
|
(optional) Any user-provided SSL certificate for client
|
||||||
|
authentication (a.k.a., mTLS). This may be a string (i.e., just
|
||||||
|
the path to a file which holds both certificate and key) or a
|
||||||
|
tuple of length 2 with the certificate file path and key file
|
||||||
|
path.
|
||||||
|
:returns:
|
||||||
|
A tuple of two dictionaries. The first is the "host parameters"
|
||||||
|
portion of the Pool Key including scheme, hostname, and port. The
|
||||||
|
second is a dictionary of SSLContext related parameters.
|
||||||
|
"""
|
||||||
|
return _urllib3_request_context(request, verify, cert, self.poolmanager)
|
||||||
|
|
||||||
|
def get_connection_with_tls_context(
|
||||||
|
self,
|
||||||
|
request: PreparedRequest,
|
||||||
|
verify: _t.VerifyType,
|
||||||
|
proxies: dict[str, str] | None = None,
|
||||||
|
cert: _t.CertType = None,
|
||||||
|
) -> HTTPConnectionPool:
|
||||||
|
"""Returns a urllib3 connection for the given request and TLS settings.
|
||||||
|
This should not be called from user code, and is only exposed for use
|
||||||
|
when subclassing the :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
|
||||||
|
|
||||||
|
:param request:
|
||||||
|
The :class:`PreparedRequest <PreparedRequest>` object to be sent
|
||||||
|
over the connection.
|
||||||
|
:param verify:
|
||||||
|
Either a boolean, in which case it controls whether we verify the
|
||||||
|
server's TLS certificate, or a string, in which case it must be a
|
||||||
|
path to a CA bundle to use.
|
||||||
|
:param proxies:
|
||||||
|
(optional) The proxies dictionary to apply to the request.
|
||||||
|
:param cert:
|
||||||
|
(optional) Any user-provided SSL certificate to be used for client
|
||||||
|
authentication (a.k.a., mTLS).
|
||||||
|
:rtype:
|
||||||
|
urllib3.HTTPConnectionPool
|
||||||
|
"""
|
||||||
|
assert _is_prepared(request)
|
||||||
|
|
||||||
|
proxy = select_proxy(request.url, proxies)
|
||||||
|
try:
|
||||||
|
host_params, pool_kwargs = self.build_connection_pool_key_attributes(
|
||||||
|
request,
|
||||||
|
verify,
|
||||||
|
cert,
|
||||||
|
)
|
||||||
|
except ValueError as e:
|
||||||
|
raise InvalidURL(e, request=request)
|
||||||
|
if proxy:
|
||||||
|
proxy = prepend_scheme_if_needed(proxy, "http")
|
||||||
|
proxy_url = parse_url(proxy)
|
||||||
|
if not proxy_url.host:
|
||||||
|
raise InvalidProxyURL(
|
||||||
|
"Please check proxy URL. It is malformed "
|
||||||
|
"and could be missing the host."
|
||||||
|
)
|
||||||
|
proxy_manager = self.proxy_manager_for(proxy)
|
||||||
|
conn = proxy_manager.connection_from_host(
|
||||||
|
**host_params, pool_kwargs=pool_kwargs
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Only scheme should be lower case
|
||||||
|
conn = self.poolmanager.connection_from_host(
|
||||||
|
**host_params, pool_kwargs=pool_kwargs
|
||||||
|
)
|
||||||
|
|
||||||
|
return conn
|
||||||
|
|
||||||
|
def get_connection(
|
||||||
|
self, url: str, proxies: dict[str, str] | None = None
|
||||||
|
) -> HTTPConnectionPool:
|
||||||
|
"""DEPRECATED: Users should move to `get_connection_with_tls_context`
|
||||||
|
for all subclasses of HTTPAdapter using Requests>=2.32.2.
|
||||||
|
|
||||||
|
Returns a urllib3 connection for the given URL. This should not be
|
||||||
|
called from user code, and is only exposed for use when subclassing the
|
||||||
|
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
|
||||||
|
|
||||||
|
:param url: The URL to connect to.
|
||||||
|
:param proxies: (optional) A Requests-style dictionary of proxies used on this request.
|
||||||
|
:rtype: urllib3.HTTPConnectionPool
|
||||||
|
"""
|
||||||
|
warnings.warn(
|
||||||
|
(
|
||||||
|
"`get_connection` has been deprecated in favor of "
|
||||||
|
"`get_connection_with_tls_context`. Custom HTTPAdapter subclasses "
|
||||||
|
"will need to migrate for Requests>=2.32.2. Please see "
|
||||||
|
"https://github.com/psf/requests/pull/6710 for more details."
|
||||||
|
),
|
||||||
|
DeprecationWarning,
|
||||||
|
)
|
||||||
|
proxy = select_proxy(url, proxies)
|
||||||
|
|
||||||
|
if proxy:
|
||||||
|
proxy = prepend_scheme_if_needed(proxy, "http")
|
||||||
|
proxy_url = parse_url(proxy)
|
||||||
|
if not proxy_url.host:
|
||||||
|
raise InvalidProxyURL(
|
||||||
|
"Please check proxy URL. It is malformed "
|
||||||
|
"and could be missing the host."
|
||||||
|
)
|
||||||
|
proxy_manager = self.proxy_manager_for(proxy)
|
||||||
|
conn = proxy_manager.connection_from_url(url)
|
||||||
|
else:
|
||||||
|
# Only scheme should be lower case
|
||||||
|
parsed = urlparse(url)
|
||||||
|
url = parsed.geturl()
|
||||||
|
conn = self.poolmanager.connection_from_url(url)
|
||||||
|
|
||||||
|
return conn
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
"""Disposes of any internal state.
|
||||||
|
|
||||||
|
Currently, this closes the PoolManager and any active ProxyManager,
|
||||||
|
which closes any pooled connections.
|
||||||
|
"""
|
||||||
|
self.poolmanager.clear()
|
||||||
|
for proxy in self.proxy_manager.values():
|
||||||
|
proxy.clear()
|
||||||
|
|
||||||
|
def request_url(
|
||||||
|
self, request: PreparedRequest, proxies: dict[str, str] | None
|
||||||
|
) -> str:
|
||||||
|
"""Obtain the url to use when making the final request.
|
||||||
|
|
||||||
|
If the message is being sent through a HTTP proxy, the full URL has to
|
||||||
|
be used. Otherwise, we should only use the path portion of the URL.
|
||||||
|
|
||||||
|
This should not be called from user code, and is only exposed for use
|
||||||
|
when subclassing the
|
||||||
|
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
|
||||||
|
|
||||||
|
:param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
|
||||||
|
:param proxies: A dictionary of schemes or schemes and hosts to proxy URLs.
|
||||||
|
:rtype: str
|
||||||
|
"""
|
||||||
|
assert _is_prepared(request)
|
||||||
|
|
||||||
|
proxy = select_proxy(request.url, proxies)
|
||||||
|
scheme = urlparse(request.url).scheme
|
||||||
|
|
||||||
|
is_proxied_http_request = proxy and scheme != "https"
|
||||||
|
using_socks_proxy = False
|
||||||
|
if proxy:
|
||||||
|
proxy_scheme = urlparse(proxy).scheme.lower()
|
||||||
|
using_socks_proxy = proxy_scheme.startswith("socks")
|
||||||
|
|
||||||
|
url = request.path_url
|
||||||
|
|
||||||
|
if is_proxied_http_request and not using_socks_proxy:
|
||||||
|
url = urldefragauth(request.url)
|
||||||
|
|
||||||
|
return url
|
||||||
|
|
||||||
|
def add_headers(self, request: PreparedRequest, **kwargs: Any) -> None:
|
||||||
|
"""Add any headers needed by the connection. As of v2.0 this does
|
||||||
|
nothing by default, but is left for overriding by users that subclass
|
||||||
|
the :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
|
||||||
|
|
||||||
|
This should not be called from user code, and is only exposed for use
|
||||||
|
when subclassing the
|
||||||
|
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
|
||||||
|
|
||||||
|
:param request: The :class:`PreparedRequest <PreparedRequest>` to add headers to.
|
||||||
|
:param kwargs: The keyword arguments from the call to send().
|
||||||
|
"""
|
||||||
|
pass
|
||||||
|
|
||||||
|
def proxy_headers(self, proxy: str) -> dict[str, str]:
|
||||||
|
"""Returns a dictionary of the headers to add to any request sent
|
||||||
|
through a proxy. This works with urllib3 magic to ensure that they are
|
||||||
|
correctly sent to the proxy, rather than in a tunnelled request if
|
||||||
|
CONNECT is being used.
|
||||||
|
|
||||||
|
This should not be called from user code, and is only exposed for use
|
||||||
|
when subclassing the
|
||||||
|
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
|
||||||
|
|
||||||
|
:param proxy: The url of the proxy being used for this request.
|
||||||
|
:rtype: dict
|
||||||
|
"""
|
||||||
|
headers: dict[str, str] = {}
|
||||||
|
username, password = get_auth_from_url(proxy)
|
||||||
|
|
||||||
|
if username:
|
||||||
|
headers["Proxy-Authorization"] = _basic_auth_str(username, password)
|
||||||
|
|
||||||
|
return headers
|
||||||
|
|
||||||
|
def send(
|
||||||
|
self,
|
||||||
|
request: PreparedRequest,
|
||||||
|
stream: bool = False,
|
||||||
|
timeout: _t.TimeoutType = None,
|
||||||
|
verify: _t.VerifyType = True,
|
||||||
|
cert: _t.CertType = None,
|
||||||
|
proxies: dict[str, str] | None = None,
|
||||||
|
) -> Response:
|
||||||
|
"""Sends PreparedRequest object. Returns Response object.
|
||||||
|
|
||||||
|
:param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
|
||||||
|
:param stream: (optional) Whether to stream the request content.
|
||||||
|
:param timeout: (optional) How long to wait for the server to send
|
||||||
|
data before giving up, as a float, or a :ref:`(connect timeout,
|
||||||
|
read timeout) <timeouts>` tuple.
|
||||||
|
:type timeout: float or tuple or urllib3 Timeout object
|
||||||
|
:param verify: (optional) Either a boolean, in which case it controls whether
|
||||||
|
we verify the server's TLS certificate, or a string, in which case it
|
||||||
|
must be a path to a CA bundle to use
|
||||||
|
:param cert: (optional) Any user-provided SSL certificate to be trusted.
|
||||||
|
:param proxies: (optional) The proxies dictionary to apply to the request.
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
assert _is_prepared(request)
|
||||||
|
|
||||||
|
try:
|
||||||
|
conn = self.get_connection_with_tls_context(
|
||||||
|
request, verify, proxies=proxies, cert=cert
|
||||||
|
)
|
||||||
|
except LocationValueError as e:
|
||||||
|
raise InvalidURL(e, request=request)
|
||||||
|
|
||||||
|
self.cert_verify(conn, request.url, verify, cert)
|
||||||
|
url = self.request_url(request, proxies)
|
||||||
|
self.add_headers(
|
||||||
|
request,
|
||||||
|
stream=stream,
|
||||||
|
timeout=timeout,
|
||||||
|
verify=verify,
|
||||||
|
cert=cert,
|
||||||
|
proxies=proxies,
|
||||||
|
)
|
||||||
|
|
||||||
|
chunked = not (request.body is None or "Content-Length" in request.headers)
|
||||||
|
|
||||||
|
if isinstance(timeout, tuple):
|
||||||
|
try:
|
||||||
|
connect, read = timeout
|
||||||
|
resolved_timeout = TimeoutSauce(connect=connect, read=read)
|
||||||
|
except ValueError:
|
||||||
|
raise ValueError(
|
||||||
|
f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, "
|
||||||
|
f"or a single float to set both timeouts to the same value."
|
||||||
|
)
|
||||||
|
elif isinstance(timeout, TimeoutSauce):
|
||||||
|
resolved_timeout = timeout
|
||||||
|
else:
|
||||||
|
resolved_timeout = TimeoutSauce(connect=timeout, read=timeout)
|
||||||
|
|
||||||
|
try:
|
||||||
|
resp = conn.urlopen(
|
||||||
|
method=request.method,
|
||||||
|
url=url,
|
||||||
|
body=request.body, # type: ignore[arg-type] # urllib3 stubs don't accept Iterable[bytes | str]
|
||||||
|
headers=request.headers, # type: ignore[arg-type] # urllib3#3072
|
||||||
|
redirect=False,
|
||||||
|
assert_same_host=False,
|
||||||
|
preload_content=False,
|
||||||
|
decode_content=False,
|
||||||
|
retries=self.max_retries,
|
||||||
|
timeout=resolved_timeout,
|
||||||
|
chunked=chunked,
|
||||||
|
)
|
||||||
|
|
||||||
|
except (ProtocolError, OSError) as err:
|
||||||
|
raise ConnectionError(err, request=request)
|
||||||
|
|
||||||
|
except MaxRetryError as e:
|
||||||
|
if isinstance(e.reason, ConnectTimeoutError):
|
||||||
|
# TODO: Remove this in 3.0.0: see #2811
|
||||||
|
if not isinstance(e.reason, NewConnectionError):
|
||||||
|
raise ConnectTimeout(e, request=request)
|
||||||
|
|
||||||
|
if isinstance(e.reason, ResponseError):
|
||||||
|
raise RetryError(e, request=request)
|
||||||
|
|
||||||
|
if isinstance(e.reason, _ProxyError):
|
||||||
|
raise ProxyError(e, request=request)
|
||||||
|
|
||||||
|
if isinstance(e.reason, _SSLError):
|
||||||
|
# This branch is for urllib3 v1.22 and later.
|
||||||
|
raise SSLError(e, request=request)
|
||||||
|
|
||||||
|
raise ConnectionError(e, request=request)
|
||||||
|
|
||||||
|
except ClosedPoolError as e:
|
||||||
|
raise ConnectionError(e, request=request)
|
||||||
|
|
||||||
|
except _ProxyError as e:
|
||||||
|
raise ProxyError(e)
|
||||||
|
|
||||||
|
except (_SSLError, _HTTPError) as e:
|
||||||
|
if isinstance(e, _SSLError):
|
||||||
|
# This branch is for urllib3 versions earlier than v1.22
|
||||||
|
raise SSLError(e, request=request)
|
||||||
|
elif isinstance(e, ReadTimeoutError):
|
||||||
|
raise ReadTimeout(e, request=request)
|
||||||
|
elif isinstance(e, _InvalidHeader):
|
||||||
|
raise InvalidHeader(e, request=request)
|
||||||
|
else:
|
||||||
|
raise
|
||||||
|
|
||||||
|
return self.build_response(request, resp)
|
||||||
@@ -0,0 +1,180 @@
|
|||||||
|
"""
|
||||||
|
requests.api
|
||||||
|
~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This module implements the Requests API.
|
||||||
|
|
||||||
|
:copyright: (c) 2012 by Kenneth Reitz.
|
||||||
|
:license: Apache2, see LICENSE for more details.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import TYPE_CHECKING
|
||||||
|
|
||||||
|
from . import sessions
|
||||||
|
from .models import Response
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from typing_extensions import Unpack
|
||||||
|
|
||||||
|
from . import _types as _t
|
||||||
|
|
||||||
|
|
||||||
|
def request(
|
||||||
|
method: str, url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]
|
||||||
|
) -> Response:
|
||||||
|
"""Constructs and sends a :class:`Request <Request>`.
|
||||||
|
|
||||||
|
:param method: method for the new :class:`Request` object: ``GET``, ``OPTIONS``, ``HEAD``, ``POST``, ``PUT``, ``PATCH``, or ``DELETE``.
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param params: (optional) Dictionary, list of tuples or bytes to send
|
||||||
|
in the query string for the :class:`Request`.
|
||||||
|
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
|
||||||
|
object to send in the body of the :class:`Request`.
|
||||||
|
:param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`.
|
||||||
|
:param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
|
||||||
|
:param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
|
||||||
|
:param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': file-tuple}``) for multipart encoding upload.
|
||||||
|
``file-tuple`` can be a 2-tuple ``('filename', fileobj)``, 3-tuple ``('filename', fileobj, 'content_type')``
|
||||||
|
or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``, where ``'content_type'`` is a string
|
||||||
|
defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers
|
||||||
|
to add for the file.
|
||||||
|
:param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
|
||||||
|
:param timeout: (optional) How many seconds to wait for the server to send data
|
||||||
|
before giving up, as a float, or a :ref:`(connect timeout, read
|
||||||
|
timeout) <timeouts>` tuple.
|
||||||
|
:type timeout: float or tuple
|
||||||
|
:param allow_redirects: (optional) Boolean. Enable/disable GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD redirection. Defaults to ``True``.
|
||||||
|
:type allow_redirects: bool
|
||||||
|
:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
|
||||||
|
:param verify: (optional) Either a boolean, in which case it controls whether we verify
|
||||||
|
the server's TLS certificate, or a string, in which case it must be a path
|
||||||
|
to a CA bundle to use. Defaults to ``True``.
|
||||||
|
:param stream: (optional) if ``False``, the response content will be immediately downloaded.
|
||||||
|
:param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
|
||||||
|
:return: :class:`Response <Response>` object
|
||||||
|
:rtype: requests.Response
|
||||||
|
|
||||||
|
Usage::
|
||||||
|
|
||||||
|
>>> import requests
|
||||||
|
>>> req = requests.request('GET', 'https://httpbin.org/get')
|
||||||
|
>>> req
|
||||||
|
<Response [200]>
|
||||||
|
"""
|
||||||
|
|
||||||
|
# By using the 'with' statement we are sure the session is closed, thus we
|
||||||
|
# avoid leaving sockets open which can trigger a ResourceWarning in some
|
||||||
|
# cases, and look like a memory leak in others.
|
||||||
|
with sessions.Session() as session:
|
||||||
|
return session.request(method=method, url=url, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def get(
|
||||||
|
url: _t.UriType, params: _t.ParamsType = None, **kwargs: Unpack[_t.GetKwargs]
|
||||||
|
) -> Response:
|
||||||
|
r"""Sends a GET request.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param params: (optional) Dictionary, list of tuples or bytes to send
|
||||||
|
in the query string for the :class:`Request`.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes.
|
||||||
|
:return: :class:`Response <Response>` object
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
return request("get", url, params=params, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def options(url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]) -> Response:
|
||||||
|
r"""Sends an OPTIONS request.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes.
|
||||||
|
:return: :class:`Response <Response>` object
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
return request("options", url, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def head(url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]) -> Response:
|
||||||
|
r"""Sends a HEAD request.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes. If
|
||||||
|
`allow_redirects` is not provided, it will be set to `False` (as
|
||||||
|
opposed to the default :meth:`request` behavior).
|
||||||
|
:return: :class:`Response <Response>` object
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
kwargs.setdefault("allow_redirects", False)
|
||||||
|
return request("head", url, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def post(
|
||||||
|
url: _t.UriType,
|
||||||
|
data: _t.DataType = None,
|
||||||
|
json: _t.JsonType = None,
|
||||||
|
**kwargs: Unpack[_t.PostKwargs],
|
||||||
|
) -> Response:
|
||||||
|
r"""Sends a POST request.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
|
||||||
|
object to send in the body of the :class:`Request`.
|
||||||
|
:param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes.
|
||||||
|
:return: :class:`Response <Response>` object
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
return request("post", url, data=data, json=json, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def put(
|
||||||
|
url: _t.UriType, data: _t.DataType = None, **kwargs: Unpack[_t.DataKwargs]
|
||||||
|
) -> Response:
|
||||||
|
r"""Sends a PUT request.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
|
||||||
|
object to send in the body of the :class:`Request`.
|
||||||
|
:param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes.
|
||||||
|
:return: :class:`Response <Response>` object
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
return request("put", url, data=data, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def patch(
|
||||||
|
url: _t.UriType, data: _t.DataType = None, **kwargs: Unpack[_t.DataKwargs]
|
||||||
|
) -> Response:
|
||||||
|
r"""Sends a PATCH request.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
|
||||||
|
object to send in the body of the :class:`Request`.
|
||||||
|
:param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes.
|
||||||
|
:return: :class:`Response <Response>` object
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
return request("patch", url, data=data, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
def delete(url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]) -> Response:
|
||||||
|
r"""Sends a DELETE request.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes.
|
||||||
|
:return: :class:`Response <Response>` object
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
return request("delete", url, **kwargs)
|
||||||
@@ -0,0 +1,354 @@
|
|||||||
|
"""
|
||||||
|
requests.auth
|
||||||
|
~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This module contains the authentication handlers for Requests.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import hashlib
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import threading
|
||||||
|
import time
|
||||||
|
import warnings
|
||||||
|
from base64 import b64encode
|
||||||
|
from typing import TYPE_CHECKING, Any, Final, cast, overload
|
||||||
|
|
||||||
|
from ._internal_utils import to_native_string
|
||||||
|
from .compat import basestring, str, urlparse
|
||||||
|
from .cookies import extract_cookies_to_jar
|
||||||
|
from .utils import parse_dict_header
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from http.cookiejar import CookieJar
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
from .models import PreparedRequest, Response
|
||||||
|
|
||||||
|
CONTENT_TYPE_FORM_URLENCODED: Final = "application/x-www-form-urlencoded"
|
||||||
|
CONTENT_TYPE_MULTI_PART: Final = "multipart/form-data"
|
||||||
|
|
||||||
|
|
||||||
|
def _basic_auth_str(username: bytes | str, password: bytes | str) -> str:
|
||||||
|
"""Returns a Basic Auth string."""
|
||||||
|
|
||||||
|
# "I want us to put a big-ol' comment on top of it that
|
||||||
|
# says that this behaviour is dumb but we need to preserve
|
||||||
|
# it because people are relying on it."
|
||||||
|
# - Lukasa
|
||||||
|
#
|
||||||
|
# These are here solely to maintain backwards compatibility
|
||||||
|
# for things like ints. This will be removed in 3.0.0.
|
||||||
|
if not isinstance(username, basestring): # type: ignore[reportUnnecessaryIsInstance] # runtime guard for non-str/bytes
|
||||||
|
warnings.warn(
|
||||||
|
"Non-string usernames will no longer be supported in Requests "
|
||||||
|
f"3.0.0. Please convert the object you've passed in ({username!r}) to "
|
||||||
|
"a string or bytes object in the near future to avoid "
|
||||||
|
"problems.",
|
||||||
|
category=DeprecationWarning,
|
||||||
|
)
|
||||||
|
username = str(username)
|
||||||
|
|
||||||
|
if not isinstance(password, basestring): # type: ignore[reportUnnecessaryIsInstance] # runtime guard for non-str/bytes
|
||||||
|
warnings.warn(
|
||||||
|
"Non-string passwords will no longer be supported in Requests "
|
||||||
|
f"3.0.0. Please convert the object you've passed in ({type(password)!r}) to "
|
||||||
|
"a string or bytes object in the near future to avoid "
|
||||||
|
"problems.",
|
||||||
|
category=DeprecationWarning,
|
||||||
|
)
|
||||||
|
password = str(password)
|
||||||
|
# -- End Removal --
|
||||||
|
|
||||||
|
if isinstance(username, str):
|
||||||
|
username = username.encode("latin1")
|
||||||
|
|
||||||
|
if isinstance(password, str):
|
||||||
|
password = password.encode("latin1")
|
||||||
|
|
||||||
|
authstr = "Basic " + to_native_string(
|
||||||
|
b64encode(b":".join((username, password))).strip()
|
||||||
|
)
|
||||||
|
|
||||||
|
return authstr
|
||||||
|
|
||||||
|
|
||||||
|
class AuthBase:
|
||||||
|
"""Base class that all auth implementations derive from"""
|
||||||
|
|
||||||
|
def __call__(self, r: PreparedRequest) -> PreparedRequest:
|
||||||
|
raise NotImplementedError("Auth hooks must be callable.")
|
||||||
|
|
||||||
|
|
||||||
|
class HTTPBasicAuth(AuthBase):
|
||||||
|
"""Attaches HTTP Basic Authentication to the given Request object."""
|
||||||
|
|
||||||
|
username: bytes | str
|
||||||
|
password: bytes | str
|
||||||
|
|
||||||
|
@overload
|
||||||
|
def __init__(self, username: str, password: str) -> None: ...
|
||||||
|
@overload
|
||||||
|
def __init__(self, username: bytes, password: bytes) -> None: ...
|
||||||
|
|
||||||
|
def __init__(self, username: bytes | str, password: bytes | str) -> None:
|
||||||
|
self.username = username
|
||||||
|
self.password = password
|
||||||
|
|
||||||
|
def __eq__(self, other: object) -> bool:
|
||||||
|
return all(
|
||||||
|
[
|
||||||
|
self.username == getattr(other, "username", None),
|
||||||
|
self.password == getattr(other, "password", None),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
def __ne__(self, other: Any) -> bool:
|
||||||
|
return not self == other
|
||||||
|
|
||||||
|
def __call__(self, r: PreparedRequest) -> PreparedRequest:
|
||||||
|
r.headers["Authorization"] = _basic_auth_str(self.username, self.password)
|
||||||
|
return r
|
||||||
|
|
||||||
|
|
||||||
|
class HTTPProxyAuth(HTTPBasicAuth):
|
||||||
|
"""Attaches HTTP Proxy Authentication to a given Request object."""
|
||||||
|
|
||||||
|
def __call__(self, r: PreparedRequest) -> PreparedRequest:
|
||||||
|
r.headers["Proxy-Authorization"] = _basic_auth_str(self.username, self.password)
|
||||||
|
return r
|
||||||
|
|
||||||
|
|
||||||
|
class HTTPDigestAuth(AuthBase):
|
||||||
|
"""Attaches HTTP Digest Authentication to the given Request object."""
|
||||||
|
|
||||||
|
username: bytes | str
|
||||||
|
password: bytes | str
|
||||||
|
_thread_local: threading.local
|
||||||
|
last_nonce: str
|
||||||
|
nonce_count: int
|
||||||
|
chal: dict[str, str]
|
||||||
|
pos: int | None
|
||||||
|
num_401_calls: int | None
|
||||||
|
|
||||||
|
@overload
|
||||||
|
def __init__(self, username: str, password: str) -> None: ...
|
||||||
|
@overload
|
||||||
|
def __init__(self, username: bytes, password: bytes) -> None: ...
|
||||||
|
|
||||||
|
def __init__(self, username: bytes | str, password: bytes | str) -> None:
|
||||||
|
self.username = username
|
||||||
|
self.password = password
|
||||||
|
# Keep state in per-thread local storage
|
||||||
|
self._thread_local = threading.local()
|
||||||
|
|
||||||
|
def init_per_thread_state(self) -> None:
|
||||||
|
# Ensure state is initialized just once per-thread
|
||||||
|
if not hasattr(self._thread_local, "init"):
|
||||||
|
self._thread_local.init = True
|
||||||
|
self._thread_local.last_nonce = ""
|
||||||
|
self._thread_local.nonce_count = 0
|
||||||
|
self._thread_local.chal = {}
|
||||||
|
self._thread_local.pos = None
|
||||||
|
self._thread_local.num_401_calls = None
|
||||||
|
|
||||||
|
def build_digest_header(self, method: str, url: str) -> str | None:
|
||||||
|
"""
|
||||||
|
:rtype: str
|
||||||
|
"""
|
||||||
|
|
||||||
|
realm = self._thread_local.chal["realm"]
|
||||||
|
nonce = self._thread_local.chal["nonce"]
|
||||||
|
qop = self._thread_local.chal.get("qop")
|
||||||
|
algorithm = self._thread_local.chal.get("algorithm")
|
||||||
|
opaque = self._thread_local.chal.get("opaque")
|
||||||
|
hash_utf8 = None
|
||||||
|
|
||||||
|
if algorithm is None:
|
||||||
|
_algorithm = "MD5"
|
||||||
|
else:
|
||||||
|
_algorithm = algorithm.upper()
|
||||||
|
# lambdas assume digest modules are imported at the top level
|
||||||
|
if _algorithm == "MD5" or _algorithm == "MD5-SESS":
|
||||||
|
|
||||||
|
def md5_utf8(x: str | bytes) -> str:
|
||||||
|
if isinstance(x, str):
|
||||||
|
x = x.encode("utf-8")
|
||||||
|
return hashlib.md5(x, usedforsecurity=False).hexdigest()
|
||||||
|
|
||||||
|
hash_utf8 = md5_utf8
|
||||||
|
elif _algorithm == "SHA":
|
||||||
|
|
||||||
|
def sha_utf8(x: str | bytes) -> str:
|
||||||
|
if isinstance(x, str):
|
||||||
|
x = x.encode("utf-8")
|
||||||
|
return hashlib.sha1(x, usedforsecurity=False).hexdigest()
|
||||||
|
|
||||||
|
hash_utf8 = sha_utf8
|
||||||
|
elif _algorithm == "SHA-256":
|
||||||
|
|
||||||
|
def sha256_utf8(x: str | bytes) -> str:
|
||||||
|
if isinstance(x, str):
|
||||||
|
x = x.encode("utf-8")
|
||||||
|
return hashlib.sha256(x, usedforsecurity=False).hexdigest()
|
||||||
|
|
||||||
|
hash_utf8 = sha256_utf8
|
||||||
|
elif _algorithm == "SHA-512":
|
||||||
|
|
||||||
|
def sha512_utf8(x: str | bytes) -> str:
|
||||||
|
if isinstance(x, str):
|
||||||
|
x = x.encode("utf-8")
|
||||||
|
return hashlib.sha512(x, usedforsecurity=False).hexdigest()
|
||||||
|
|
||||||
|
hash_utf8 = sha512_utf8
|
||||||
|
|
||||||
|
if hash_utf8 is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
def KD(s: str, d: str) -> str:
|
||||||
|
return hash_utf8(f"{s}:{d}")
|
||||||
|
|
||||||
|
# XXX not implemented yet
|
||||||
|
entdig = None
|
||||||
|
p_parsed = urlparse(url)
|
||||||
|
#: path is request-uri defined in RFC 2616 which should not be empty
|
||||||
|
path = p_parsed.path or "/"
|
||||||
|
if p_parsed.query:
|
||||||
|
path += f"?{p_parsed.query}"
|
||||||
|
|
||||||
|
A1 = f"{self.username}:{realm}:{self.password}"
|
||||||
|
A2 = f"{method}:{path}"
|
||||||
|
|
||||||
|
HA1 = hash_utf8(A1)
|
||||||
|
HA2 = hash_utf8(A2)
|
||||||
|
|
||||||
|
if nonce == self._thread_local.last_nonce:
|
||||||
|
self._thread_local.nonce_count += 1
|
||||||
|
else:
|
||||||
|
self._thread_local.nonce_count = 1
|
||||||
|
ncvalue = f"{self._thread_local.nonce_count:08x}"
|
||||||
|
s = str(self._thread_local.nonce_count).encode("utf-8")
|
||||||
|
s += nonce.encode("utf-8")
|
||||||
|
s += time.ctime().encode("utf-8")
|
||||||
|
s += os.urandom(8)
|
||||||
|
|
||||||
|
cnonce = hashlib.sha1(s, usedforsecurity=False).hexdigest()[:16]
|
||||||
|
if _algorithm == "MD5-SESS":
|
||||||
|
HA1 = hash_utf8(f"{HA1}:{nonce}:{cnonce}") # type: ignore[reportConstantRedefinition] # RFC 2617 terminology
|
||||||
|
|
||||||
|
if not qop:
|
||||||
|
respdig = KD(HA1, f"{nonce}:{HA2}")
|
||||||
|
elif qop == "auth" or "auth" in qop.split(","):
|
||||||
|
noncebit = f"{nonce}:{ncvalue}:{cnonce}:auth:{HA2}"
|
||||||
|
respdig = KD(HA1, noncebit)
|
||||||
|
else:
|
||||||
|
# XXX handle auth-int.
|
||||||
|
return None
|
||||||
|
|
||||||
|
self._thread_local.last_nonce = nonce
|
||||||
|
|
||||||
|
# XXX should the partial digests be encoded too?
|
||||||
|
base = (
|
||||||
|
f'username="{self.username}", realm="{realm}", nonce="{nonce}", '
|
||||||
|
f'uri="{path}", response="{respdig}"'
|
||||||
|
)
|
||||||
|
if opaque:
|
||||||
|
base += f', opaque="{opaque}"'
|
||||||
|
if algorithm:
|
||||||
|
base += f', algorithm="{algorithm}"'
|
||||||
|
if entdig:
|
||||||
|
base += f', digest="{entdig}"'
|
||||||
|
if qop:
|
||||||
|
base += f', qop="auth", nc={ncvalue}, cnonce="{cnonce}"'
|
||||||
|
|
||||||
|
return f"Digest {base}"
|
||||||
|
|
||||||
|
def handle_redirect(self, r: Response, **kwargs: Any) -> None:
|
||||||
|
"""Reset num_401_calls counter on redirects."""
|
||||||
|
if r.is_redirect:
|
||||||
|
self._thread_local.num_401_calls = 1
|
||||||
|
|
||||||
|
def handle_401(self, r: Response, **kwargs: Any) -> Response:
|
||||||
|
"""
|
||||||
|
Takes the given response and tries digest-auth, if needed.
|
||||||
|
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
# If response is not 4xx, do not auth
|
||||||
|
# See https://github.com/psf/requests/issues/3772
|
||||||
|
if not 400 <= r.status_code < 500:
|
||||||
|
self._thread_local.num_401_calls = 1
|
||||||
|
return r
|
||||||
|
|
||||||
|
if self._thread_local.pos is not None:
|
||||||
|
# Rewind the file position indicator of the body to where
|
||||||
|
# it was to resend the request.
|
||||||
|
if (seek := getattr(r.request.body, "seek", None)) is not None:
|
||||||
|
seek(self._thread_local.pos)
|
||||||
|
s_auth = r.headers.get("www-authenticate", "")
|
||||||
|
|
||||||
|
if "digest" in s_auth.lower() and self._thread_local.num_401_calls < 2:
|
||||||
|
self._thread_local.num_401_calls += 1
|
||||||
|
pat = re.compile(r"digest ", flags=re.IGNORECASE)
|
||||||
|
self._thread_local.chal = parse_dict_header(pat.sub("", s_auth, count=1))
|
||||||
|
|
||||||
|
# Consume content and release the original connection
|
||||||
|
# to allow our new request to reuse the same one.
|
||||||
|
r.content
|
||||||
|
r.close()
|
||||||
|
prep = r.request.copy()
|
||||||
|
cookie_jar = cast("CookieJar", prep._cookies) # type: ignore[reportPrivateUsage]
|
||||||
|
extract_cookies_to_jar(cookie_jar, r.request, r.raw)
|
||||||
|
prep.prepare_cookies(cookie_jar)
|
||||||
|
|
||||||
|
_digest_auth = self.build_digest_header(
|
||||||
|
cast(str, prep.method), cast(str, prep.url)
|
||||||
|
)
|
||||||
|
if _digest_auth:
|
||||||
|
prep.headers["Authorization"] = _digest_auth
|
||||||
|
_r = r.connection.send(prep, **kwargs)
|
||||||
|
_r.history.append(r)
|
||||||
|
_r.request = prep
|
||||||
|
|
||||||
|
return _r
|
||||||
|
|
||||||
|
self._thread_local.num_401_calls = 1
|
||||||
|
return r
|
||||||
|
|
||||||
|
def __call__(self, r: PreparedRequest) -> PreparedRequest:
|
||||||
|
# Initialize per-thread state, if needed
|
||||||
|
self.init_per_thread_state()
|
||||||
|
# If we have a saved nonce, skip the 401
|
||||||
|
if self._thread_local.last_nonce:
|
||||||
|
_digest_auth = self.build_digest_header(
|
||||||
|
cast(str, r.method), cast(str, r.url)
|
||||||
|
)
|
||||||
|
if _digest_auth:
|
||||||
|
r.headers["Authorization"] = _digest_auth
|
||||||
|
if (tell := getattr(r.body, "tell", None)) is not None:
|
||||||
|
self._thread_local.pos = tell()
|
||||||
|
else:
|
||||||
|
# In the case of HTTPDigestAuth being reused and the body of
|
||||||
|
# the previous request was a file-like object, pos has the
|
||||||
|
# file position of the previous body. Ensure it's set to
|
||||||
|
# None.
|
||||||
|
self._thread_local.pos = None
|
||||||
|
r.register_hook("response", self.handle_401)
|
||||||
|
r.register_hook("response", self.handle_redirect)
|
||||||
|
self._thread_local.num_401_calls = 1
|
||||||
|
|
||||||
|
return r
|
||||||
|
|
||||||
|
def __eq__(self, other: object) -> bool:
|
||||||
|
return all(
|
||||||
|
[
|
||||||
|
self.username == getattr(other, "username", None),
|
||||||
|
self.password == getattr(other, "password", None),
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
def __ne__(self, other: Any) -> bool:
|
||||||
|
return not self == other
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
#!/usr/bin/env python
|
||||||
|
|
||||||
|
"""
|
||||||
|
requests.certs
|
||||||
|
~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This module returns the preferred default CA certificate bundle. There is
|
||||||
|
only one — the one from the certifi package.
|
||||||
|
|
||||||
|
If you are packaging Requests, e.g., for a Linux distribution or a managed
|
||||||
|
environment, you can change the definition of where() to return a separately
|
||||||
|
packaged CA bundle.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from certifi import where
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
print(where())
|
||||||
@@ -0,0 +1,113 @@
|
|||||||
|
"""
|
||||||
|
requests.compat
|
||||||
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This module previously handled import compatibility issues
|
||||||
|
between Python 2 and Python 3. It remains for backwards
|
||||||
|
compatibility until the next major version.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# pyright: reportUnusedImport=false
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import importlib
|
||||||
|
import sys
|
||||||
|
from types import ModuleType
|
||||||
|
|
||||||
|
# -------
|
||||||
|
# urllib3
|
||||||
|
# -------
|
||||||
|
from urllib3 import (
|
||||||
|
__version__ as urllib3_version, # type: ignore[reportPrivateImportUsage]
|
||||||
|
)
|
||||||
|
|
||||||
|
# Detect which major version of urllib3 is being used.
|
||||||
|
try:
|
||||||
|
is_urllib3_1 = int(urllib3_version.split(".")[0]) == 1
|
||||||
|
except (TypeError, AttributeError):
|
||||||
|
# If we can't discern a version, prefer old functionality.
|
||||||
|
is_urllib3_1 = True
|
||||||
|
|
||||||
|
# -------------------
|
||||||
|
# Character Detection
|
||||||
|
# -------------------
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_char_detection() -> ModuleType | None:
|
||||||
|
"""Find supported character detection libraries."""
|
||||||
|
chardet = None
|
||||||
|
for lib in ("chardet", "charset_normalizer"):
|
||||||
|
if chardet is None:
|
||||||
|
try:
|
||||||
|
chardet = importlib.import_module(lib)
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
return chardet
|
||||||
|
|
||||||
|
|
||||||
|
chardet = _resolve_char_detection()
|
||||||
|
|
||||||
|
# -------
|
||||||
|
# Pythons
|
||||||
|
# -------
|
||||||
|
|
||||||
|
# Syntax sugar.
|
||||||
|
_ver = sys.version_info
|
||||||
|
|
||||||
|
#: Python 2.x?
|
||||||
|
is_py2 = _ver[0] == 2
|
||||||
|
|
||||||
|
#: Python 3.x?
|
||||||
|
is_py3 = _ver[0] == 3
|
||||||
|
|
||||||
|
# json/simplejson module import resolution
|
||||||
|
has_simplejson = False
|
||||||
|
try:
|
||||||
|
import simplejson as json # type: ignore[import-not-found]
|
||||||
|
|
||||||
|
has_simplejson = True
|
||||||
|
except ImportError:
|
||||||
|
import json
|
||||||
|
|
||||||
|
if has_simplejson:
|
||||||
|
from simplejson import JSONDecodeError # type: ignore[import-not-found]
|
||||||
|
else:
|
||||||
|
from json import JSONDecodeError
|
||||||
|
|
||||||
|
# Keep OrderedDict for backwards compatibility.
|
||||||
|
from collections import OrderedDict
|
||||||
|
from collections.abc import Callable, Mapping, MutableMapping
|
||||||
|
from http import cookiejar as cookielib
|
||||||
|
from http.cookies import Morsel
|
||||||
|
from io import StringIO
|
||||||
|
|
||||||
|
# --------------
|
||||||
|
# Legacy Imports
|
||||||
|
# --------------
|
||||||
|
from urllib.parse import (
|
||||||
|
quote,
|
||||||
|
quote_plus,
|
||||||
|
unquote,
|
||||||
|
unquote_plus,
|
||||||
|
urldefrag,
|
||||||
|
urlencode,
|
||||||
|
urljoin,
|
||||||
|
urlparse,
|
||||||
|
urlsplit,
|
||||||
|
urlunparse,
|
||||||
|
)
|
||||||
|
from urllib.request import (
|
||||||
|
getproxies,
|
||||||
|
getproxies_environment,
|
||||||
|
parse_http_list,
|
||||||
|
proxy_bypass,
|
||||||
|
proxy_bypass_environment, # type: ignore[attr-defined] # https://github.com/python/cpython/issues/145331
|
||||||
|
)
|
||||||
|
|
||||||
|
builtin_str = str
|
||||||
|
str = str
|
||||||
|
bytes = bytes
|
||||||
|
basestring = (str, bytes)
|
||||||
|
numeric_types = (int, float)
|
||||||
|
integer_types = (int,)
|
||||||
@@ -0,0 +1,625 @@
|
|||||||
|
"""
|
||||||
|
requests.cookies
|
||||||
|
~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Compatibility code to be able to use `http.cookiejar.CookieJar` with requests.
|
||||||
|
|
||||||
|
requests.utils imports from here, so be careful with imports.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import calendar
|
||||||
|
import copy
|
||||||
|
import time
|
||||||
|
from collections.abc import Iterator, MutableMapping
|
||||||
|
from http.cookiejar import Cookie, CookieJar, CookiePolicy
|
||||||
|
from typing import TYPE_CHECKING, Any, TypeVar, overload
|
||||||
|
|
||||||
|
from ._internal_utils import to_native_string
|
||||||
|
from ._types import is_prepared as _is_prepared
|
||||||
|
from .compat import Morsel, cookielib, urlparse, urlunparse
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from _typeshed import SupportsKeysAndGetItem
|
||||||
|
|
||||||
|
from .models import PreparedRequest
|
||||||
|
|
||||||
|
import threading
|
||||||
|
|
||||||
|
|
||||||
|
class MockRequest:
|
||||||
|
"""Wraps a `requests.PreparedRequest` to mimic a `urllib2.Request`.
|
||||||
|
|
||||||
|
The code in `http.cookiejar.CookieJar` expects this interface in order to correctly
|
||||||
|
manage cookie policies, i.e., determine whether a cookie can be set, given the
|
||||||
|
domains of the request and the cookie.
|
||||||
|
|
||||||
|
The original request object is read-only. The client is responsible for collecting
|
||||||
|
the new headers via `get_new_headers()` and interpreting them appropriately. You
|
||||||
|
probably want `get_cookie_header`, defined below.
|
||||||
|
"""
|
||||||
|
|
||||||
|
type: str
|
||||||
|
|
||||||
|
def __init__(self, request: PreparedRequest) -> None:
|
||||||
|
assert _is_prepared(request)
|
||||||
|
self._r = request
|
||||||
|
self._new_headers: dict[str, str] = {}
|
||||||
|
self.type = urlparse(self._r.url).scheme
|
||||||
|
|
||||||
|
def get_type(self) -> str:
|
||||||
|
return self.type
|
||||||
|
|
||||||
|
def get_host(self) -> str:
|
||||||
|
return urlparse(self._r.url).netloc
|
||||||
|
|
||||||
|
def get_origin_req_host(self) -> str:
|
||||||
|
return self.get_host()
|
||||||
|
|
||||||
|
def get_full_url(self) -> str:
|
||||||
|
# Only return the response's URL if the user hadn't set the Host
|
||||||
|
# header
|
||||||
|
if not self._r.headers.get("Host"):
|
||||||
|
return self._r.url
|
||||||
|
# If they did set it, retrieve it and reconstruct the expected domain
|
||||||
|
host = to_native_string(self._r.headers["Host"], encoding="utf-8")
|
||||||
|
parsed = urlparse(self._r.url)
|
||||||
|
# Reconstruct the URL as we expect it
|
||||||
|
return urlunparse(
|
||||||
|
[
|
||||||
|
parsed.scheme,
|
||||||
|
host,
|
||||||
|
parsed.path,
|
||||||
|
parsed.params,
|
||||||
|
parsed.query,
|
||||||
|
parsed.fragment,
|
||||||
|
]
|
||||||
|
)
|
||||||
|
|
||||||
|
def is_unverifiable(self) -> bool:
|
||||||
|
return True
|
||||||
|
|
||||||
|
def has_header(self, name: str) -> bool:
|
||||||
|
return name in self._r.headers or name in self._new_headers
|
||||||
|
|
||||||
|
def get_header(self, name: str, default: str | None = None) -> str | None:
|
||||||
|
return self._r.headers.get(name, self._new_headers.get(name, default)) # type: ignore[return-value]
|
||||||
|
|
||||||
|
def add_header(self, key: str, val: str) -> None:
|
||||||
|
"""cookiejar has no legitimate use for this method; add it back if you find one."""
|
||||||
|
raise NotImplementedError(
|
||||||
|
"Cookie headers should be added with add_unredirected_header()"
|
||||||
|
)
|
||||||
|
|
||||||
|
def add_unredirected_header(self, name: str, value: str) -> None:
|
||||||
|
self._new_headers[name] = value
|
||||||
|
|
||||||
|
def get_new_headers(self) -> dict[str, str]:
|
||||||
|
return self._new_headers
|
||||||
|
|
||||||
|
@property
|
||||||
|
def unverifiable(self) -> bool:
|
||||||
|
return self.is_unverifiable()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def origin_req_host(self) -> str:
|
||||||
|
return self.get_origin_req_host()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def host(self) -> str:
|
||||||
|
return self.get_host()
|
||||||
|
|
||||||
|
|
||||||
|
class MockResponse:
|
||||||
|
"""Wraps a `httplib.HTTPMessage` to mimic a `urllib.addinfourl`.
|
||||||
|
|
||||||
|
...what? Basically, expose the parsed HTTP headers from the server response
|
||||||
|
the way `http.cookiejar` expects to see them.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, headers: Any) -> None:
|
||||||
|
"""Make a MockResponse for `cookiejar` to read.
|
||||||
|
|
||||||
|
:param headers: a httplib.HTTPMessage or analogous carrying the headers
|
||||||
|
"""
|
||||||
|
self._headers = headers
|
||||||
|
|
||||||
|
def info(self) -> Any:
|
||||||
|
return self._headers
|
||||||
|
|
||||||
|
def getheaders(self, name: str) -> Any:
|
||||||
|
self._headers.getheaders(name)
|
||||||
|
|
||||||
|
|
||||||
|
def extract_cookies_to_jar(
|
||||||
|
jar: CookieJar, request: PreparedRequest, response: Any
|
||||||
|
) -> None:
|
||||||
|
"""Extract the cookies from the response into a CookieJar.
|
||||||
|
|
||||||
|
:param jar: http.cookiejar.CookieJar (not necessarily a RequestsCookieJar)
|
||||||
|
:param request: our own requests.Request object
|
||||||
|
:param response: urllib3.HTTPResponse object
|
||||||
|
"""
|
||||||
|
if not (hasattr(response, "_original_response") and response._original_response):
|
||||||
|
return
|
||||||
|
# the _original_response field is the wrapped httplib.HTTPResponse object,
|
||||||
|
req = MockRequest(request)
|
||||||
|
# pull out the HTTPMessage with the headers and put it in the mock:
|
||||||
|
res = MockResponse(response._original_response.msg)
|
||||||
|
jar.extract_cookies(res, req) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
|
||||||
|
def get_cookie_header(jar: CookieJar, request: PreparedRequest) -> str | None:
|
||||||
|
"""
|
||||||
|
Produce an appropriate Cookie header string to be sent with `request`, or None.
|
||||||
|
|
||||||
|
:rtype: str
|
||||||
|
"""
|
||||||
|
r = MockRequest(request)
|
||||||
|
jar.add_cookie_header(r) # type: ignore[arg-type]
|
||||||
|
return r.get_new_headers().get("Cookie")
|
||||||
|
|
||||||
|
|
||||||
|
def remove_cookie_by_name(
|
||||||
|
cookiejar: CookieJar, name: str, domain: str | None = None, path: str | None = None
|
||||||
|
) -> None:
|
||||||
|
"""Unsets a cookie by name, by default over all domains and paths.
|
||||||
|
|
||||||
|
Wraps CookieJar.clear(), is O(n).
|
||||||
|
"""
|
||||||
|
clearables: list[tuple[str, str, str]] = []
|
||||||
|
for cookie in cookiejar:
|
||||||
|
if cookie.name != name:
|
||||||
|
continue
|
||||||
|
if domain is not None and domain != cookie.domain:
|
||||||
|
continue
|
||||||
|
if path is not None and path != cookie.path:
|
||||||
|
continue
|
||||||
|
clearables.append((cookie.domain, cookie.path, cookie.name))
|
||||||
|
|
||||||
|
for domain, path, name in clearables:
|
||||||
|
cookiejar.clear(domain, path, name)
|
||||||
|
|
||||||
|
|
||||||
|
class CookieConflictError(RuntimeError):
|
||||||
|
"""There are two cookies that meet the criteria specified in the cookie jar.
|
||||||
|
Use .get and .set and include domain and path args in order to be more specific.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
class RequestsCookieJar(CookieJar, MutableMapping[str, str | None]): # type: ignore[misc]
|
||||||
|
"""Compatibility class; is a http.cookiejar.CookieJar, but exposes a dict
|
||||||
|
interface.
|
||||||
|
|
||||||
|
This is the CookieJar we create by default for requests and sessions that
|
||||||
|
don't specify one, since some clients may expect response.cookies and
|
||||||
|
session.cookies to support dict operations.
|
||||||
|
|
||||||
|
Requests does not use the dict interface internally; it's just for
|
||||||
|
compatibility with external client code. All requests code should work
|
||||||
|
out of the box with externally provided instances of ``CookieJar``, e.g.
|
||||||
|
``LWPCookieJar`` and ``FileCookieJar``.
|
||||||
|
|
||||||
|
Unlike a regular CookieJar, this class is pickleable.
|
||||||
|
|
||||||
|
.. warning:: dictionary operations that are normally O(1) may be O(n).
|
||||||
|
"""
|
||||||
|
|
||||||
|
_policy: CookiePolicy
|
||||||
|
|
||||||
|
def get( # type: ignore[override]
|
||||||
|
self,
|
||||||
|
name: str,
|
||||||
|
default: str | None = None,
|
||||||
|
domain: str | None = None,
|
||||||
|
path: str | None = None,
|
||||||
|
) -> str | None:
|
||||||
|
"""Dict-like get() that also supports optional domain and path args in
|
||||||
|
order to resolve naming collisions from using one cookie jar over
|
||||||
|
multiple domains.
|
||||||
|
|
||||||
|
.. warning:: operation is O(n), not O(1).
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
return self._find_no_duplicates(name, domain, path)
|
||||||
|
except KeyError:
|
||||||
|
return default
|
||||||
|
|
||||||
|
def set(
|
||||||
|
self, name: str, value: str | Morsel[dict[str, str]] | None, **kwargs: Any
|
||||||
|
) -> Cookie | None:
|
||||||
|
"""Dict-like set() that also supports optional domain and path args in
|
||||||
|
order to resolve naming collisions from using one cookie jar over
|
||||||
|
multiple domains.
|
||||||
|
"""
|
||||||
|
# support client code that unsets cookies by assignment of a None value:
|
||||||
|
if value is None:
|
||||||
|
remove_cookie_by_name(
|
||||||
|
self, name, domain=kwargs.get("domain"), path=kwargs.get("path")
|
||||||
|
)
|
||||||
|
return
|
||||||
|
|
||||||
|
if isinstance(value, Morsel):
|
||||||
|
c = morsel_to_cookie(value)
|
||||||
|
else:
|
||||||
|
c = create_cookie(name, value, **kwargs)
|
||||||
|
self.set_cookie(c)
|
||||||
|
return c
|
||||||
|
|
||||||
|
def iterkeys(self) -> Iterator[str]:
|
||||||
|
"""Dict-like iterkeys() that returns an iterator of names of cookies
|
||||||
|
from the jar.
|
||||||
|
|
||||||
|
.. seealso:: itervalues() and iteritems().
|
||||||
|
"""
|
||||||
|
for cookie in iter(self):
|
||||||
|
yield cookie.name
|
||||||
|
|
||||||
|
def keys(self) -> list[str]: # type: ignore[override]
|
||||||
|
"""Dict-like keys() that returns a list of names of cookies from the
|
||||||
|
jar.
|
||||||
|
|
||||||
|
.. seealso:: values() and items().
|
||||||
|
"""
|
||||||
|
return list(self.iterkeys())
|
||||||
|
|
||||||
|
def itervalues(self) -> Iterator[str | None]:
|
||||||
|
"""Dict-like itervalues() that returns an iterator of values of cookies
|
||||||
|
from the jar.
|
||||||
|
|
||||||
|
.. seealso:: iterkeys() and iteritems().
|
||||||
|
"""
|
||||||
|
for cookie in iter(self):
|
||||||
|
yield cookie.value
|
||||||
|
|
||||||
|
def values(self) -> list[str | None]: # type: ignore[override]
|
||||||
|
"""Dict-like values() that returns a list of values of cookies from the
|
||||||
|
jar.
|
||||||
|
|
||||||
|
.. seealso:: keys() and items().
|
||||||
|
"""
|
||||||
|
return list(self.itervalues())
|
||||||
|
|
||||||
|
def iteritems(self) -> Iterator[tuple[str, str | None]]:
|
||||||
|
"""Dict-like iteritems() that returns an iterator of name-value tuples
|
||||||
|
from the jar.
|
||||||
|
|
||||||
|
.. seealso:: iterkeys() and itervalues().
|
||||||
|
"""
|
||||||
|
for cookie in iter(self):
|
||||||
|
yield cookie.name, cookie.value
|
||||||
|
|
||||||
|
def items(self) -> list[tuple[str, str | None]]: # type: ignore[override]
|
||||||
|
"""Dict-like items() that returns a list of name-value tuples from the
|
||||||
|
jar. Allows client-code to call ``dict(RequestsCookieJar)`` and get a
|
||||||
|
vanilla python dict of key value pairs.
|
||||||
|
|
||||||
|
.. seealso:: keys() and values().
|
||||||
|
"""
|
||||||
|
return list(self.iteritems())
|
||||||
|
|
||||||
|
def list_domains(self) -> list[str]:
|
||||||
|
"""Utility method to list all the domains in the jar."""
|
||||||
|
domains: list[str] = []
|
||||||
|
for cookie in iter(self):
|
||||||
|
if cookie.domain not in domains:
|
||||||
|
domains.append(cookie.domain)
|
||||||
|
return domains
|
||||||
|
|
||||||
|
def list_paths(self) -> list[str]:
|
||||||
|
"""Utility method to list all the paths in the jar."""
|
||||||
|
paths: list[str] = []
|
||||||
|
for cookie in iter(self):
|
||||||
|
if cookie.path not in paths:
|
||||||
|
paths.append(cookie.path)
|
||||||
|
return paths
|
||||||
|
|
||||||
|
def multiple_domains(self) -> bool:
|
||||||
|
"""Returns True if there are multiple domains in the jar.
|
||||||
|
Returns False otherwise.
|
||||||
|
|
||||||
|
:rtype: bool
|
||||||
|
"""
|
||||||
|
domains: list[str] = []
|
||||||
|
for cookie in iter(self):
|
||||||
|
if cookie.domain is not None and cookie.domain in domains: # type: ignore[reportUnnecessaryComparison] # defensive check
|
||||||
|
return True
|
||||||
|
domains.append(cookie.domain)
|
||||||
|
return False # there is only one domain in jar
|
||||||
|
|
||||||
|
def get_dict(
|
||||||
|
self, domain: str | None = None, path: str | None = None
|
||||||
|
) -> dict[str, str | None]:
|
||||||
|
"""Takes as an argument an optional domain and path and returns a plain
|
||||||
|
old Python dict of name-value pairs of cookies that meet the
|
||||||
|
requirements.
|
||||||
|
|
||||||
|
:rtype: dict
|
||||||
|
"""
|
||||||
|
dictionary: dict[str, str | None] = {}
|
||||||
|
for cookie in iter(self):
|
||||||
|
if (domain is None or cookie.domain == domain) and (
|
||||||
|
path is None or cookie.path == path
|
||||||
|
):
|
||||||
|
dictionary[cookie.name] = cookie.value
|
||||||
|
return dictionary
|
||||||
|
|
||||||
|
def __iter__(self) -> Iterator[Cookie]: # type: ignore[override]
|
||||||
|
"""RequestCookieJar's __iter__ comes from CookieJar not MutableMapping."""
|
||||||
|
return super().__iter__()
|
||||||
|
|
||||||
|
def __contains__(self, name: object) -> bool:
|
||||||
|
try:
|
||||||
|
return super().__contains__(name)
|
||||||
|
except CookieConflictError:
|
||||||
|
return True
|
||||||
|
|
||||||
|
def __getitem__(self, name: str) -> str | None:
|
||||||
|
"""Dict-like __getitem__() for compatibility with client code. Throws
|
||||||
|
exception if there are more than one cookie with name. In that case,
|
||||||
|
use the more explicit get() method instead.
|
||||||
|
|
||||||
|
.. warning:: operation is O(n), not O(1).
|
||||||
|
"""
|
||||||
|
return self._find_no_duplicates(name)
|
||||||
|
|
||||||
|
def __setitem__(
|
||||||
|
self, name: str, value: str | Morsel[dict[str, str]] | None
|
||||||
|
) -> None:
|
||||||
|
"""Dict-like __setitem__ for compatibility with client code. Throws
|
||||||
|
exception if there is already a cookie of that name in the jar. In that
|
||||||
|
case, use the more explicit set() method instead.
|
||||||
|
"""
|
||||||
|
self.set(name, value)
|
||||||
|
|
||||||
|
def __delitem__(self, name: str) -> None:
|
||||||
|
"""Deletes a cookie given a name. Wraps ``http.cookiejar.CookieJar``'s
|
||||||
|
``remove_cookie_by_name()``.
|
||||||
|
"""
|
||||||
|
remove_cookie_by_name(self, name)
|
||||||
|
|
||||||
|
def set_cookie(self, cookie: Cookie, *args: Any, **kwargs: Any) -> None:
|
||||||
|
if (
|
||||||
|
(value := cookie.value) is not None
|
||||||
|
and value.startswith('"')
|
||||||
|
and value.endswith('"')
|
||||||
|
):
|
||||||
|
cookie.value = value.replace('\\"', "")
|
||||||
|
return super().set_cookie(cookie, *args, **kwargs)
|
||||||
|
|
||||||
|
def update( # type: ignore[override]
|
||||||
|
self, other: CookieJar | SupportsKeysAndGetItem[str, str]
|
||||||
|
) -> None:
|
||||||
|
"""Updates this jar with cookies from another CookieJar or dict-like"""
|
||||||
|
if isinstance(other, cookielib.CookieJar):
|
||||||
|
for cookie in other:
|
||||||
|
self.set_cookie(copy.copy(cookie))
|
||||||
|
else:
|
||||||
|
super().update(other)
|
||||||
|
|
||||||
|
def _find(
|
||||||
|
self, name: str, domain: str | None = None, path: str | None = None
|
||||||
|
) -> str | None:
|
||||||
|
"""Requests uses this method internally to get cookie values.
|
||||||
|
|
||||||
|
If there are conflicting cookies, _find arbitrarily chooses one.
|
||||||
|
See _find_no_duplicates if you want an exception thrown if there are
|
||||||
|
conflicting cookies.
|
||||||
|
|
||||||
|
:param name: a string containing name of cookie
|
||||||
|
:param domain: (optional) string containing domain of cookie
|
||||||
|
:param path: (optional) string containing path of cookie
|
||||||
|
:return: cookie.value
|
||||||
|
"""
|
||||||
|
for cookie in iter(self):
|
||||||
|
if cookie.name == name:
|
||||||
|
if domain is None or cookie.domain == domain:
|
||||||
|
if path is None or cookie.path == path:
|
||||||
|
return cookie.value
|
||||||
|
|
||||||
|
raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
|
||||||
|
|
||||||
|
def _find_no_duplicates(
|
||||||
|
self, name: str, domain: str | None = None, path: str | None = None
|
||||||
|
) -> str:
|
||||||
|
"""Both ``__get_item__`` and ``get`` call this function: it's never
|
||||||
|
used elsewhere in Requests.
|
||||||
|
|
||||||
|
:param name: a string containing name of cookie
|
||||||
|
:param domain: (optional) string containing domain of cookie
|
||||||
|
:param path: (optional) string containing path of cookie
|
||||||
|
:raises KeyError: if cookie is not found
|
||||||
|
:raises CookieConflictError: if there are multiple cookies
|
||||||
|
that match name and optionally domain and path
|
||||||
|
:return: cookie.value
|
||||||
|
"""
|
||||||
|
toReturn = None
|
||||||
|
for cookie in iter(self):
|
||||||
|
if cookie.name == name:
|
||||||
|
if domain is None or cookie.domain == domain:
|
||||||
|
if path is None or cookie.path == path:
|
||||||
|
if toReturn is not None:
|
||||||
|
# if there are multiple cookies that meet passed in criteria
|
||||||
|
raise CookieConflictError(
|
||||||
|
f"There are multiple cookies with name, {name!r}"
|
||||||
|
)
|
||||||
|
# we will eventually return this as long as no cookie conflict
|
||||||
|
toReturn = cookie.value
|
||||||
|
|
||||||
|
if toReturn is not None:
|
||||||
|
return toReturn
|
||||||
|
raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
|
||||||
|
|
||||||
|
def __getstate__(self) -> dict[str, Any]:
|
||||||
|
"""Unlike a normal CookieJar, this class is pickleable."""
|
||||||
|
state = self.__dict__.copy()
|
||||||
|
# remove the unpickleable RLock object
|
||||||
|
state.pop("_cookies_lock")
|
||||||
|
return state
|
||||||
|
|
||||||
|
def __setstate__(self, state: dict[str, Any]) -> None:
|
||||||
|
"""Unlike a normal CookieJar, this class is pickleable."""
|
||||||
|
self.__dict__.update(state)
|
||||||
|
if "_cookies_lock" not in self.__dict__:
|
||||||
|
self._cookies_lock = threading.RLock()
|
||||||
|
|
||||||
|
def copy(self) -> RequestsCookieJar:
|
||||||
|
"""Return a copy of this RequestsCookieJar."""
|
||||||
|
new_cj = RequestsCookieJar()
|
||||||
|
new_cj.set_policy(self.get_policy())
|
||||||
|
new_cj.update(self)
|
||||||
|
return new_cj
|
||||||
|
|
||||||
|
def get_policy(self) -> CookiePolicy:
|
||||||
|
"""Return the CookiePolicy instance used."""
|
||||||
|
return self._policy
|
||||||
|
|
||||||
|
|
||||||
|
def _copy_cookie_jar(jar: CookieJar | None) -> CookieJar | None: # type: ignore[reportUnusedFunction] # cross-module usage in models.py
|
||||||
|
if jar is None:
|
||||||
|
return None
|
||||||
|
|
||||||
|
if copy_method := getattr(jar, "copy", None):
|
||||||
|
# We're dealing with an instance of RequestsCookieJar
|
||||||
|
return copy_method()
|
||||||
|
# We're dealing with a generic CookieJar instance
|
||||||
|
new_jar = copy.copy(jar)
|
||||||
|
new_jar.clear()
|
||||||
|
for cookie in jar:
|
||||||
|
new_jar.set_cookie(copy.copy(cookie))
|
||||||
|
return new_jar
|
||||||
|
|
||||||
|
|
||||||
|
def create_cookie(name: str, value: str, **kwargs: Any) -> Cookie:
|
||||||
|
"""Make a cookie from underspecified parameters.
|
||||||
|
|
||||||
|
By default, the pair of `name` and `value` will be set for the domain ''
|
||||||
|
and sent on every request (this is sometimes called a "supercookie").
|
||||||
|
"""
|
||||||
|
result: dict[str, Any] = {
|
||||||
|
"version": 0,
|
||||||
|
"name": name,
|
||||||
|
"value": value,
|
||||||
|
"port": None,
|
||||||
|
"domain": "",
|
||||||
|
"path": "/",
|
||||||
|
"secure": False,
|
||||||
|
"expires": None,
|
||||||
|
"discard": True,
|
||||||
|
"comment": None,
|
||||||
|
"comment_url": None,
|
||||||
|
"rest": {"HttpOnly": None},
|
||||||
|
"rfc2109": False,
|
||||||
|
}
|
||||||
|
|
||||||
|
badargs = set(kwargs) - set(result)
|
||||||
|
if badargs:
|
||||||
|
raise TypeError(
|
||||||
|
f"create_cookie() got unexpected keyword arguments: {list(badargs)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
result.update(kwargs)
|
||||||
|
result["port_specified"] = bool(result["port"])
|
||||||
|
result["domain_specified"] = bool(result["domain"])
|
||||||
|
result["domain_initial_dot"] = result["domain"].startswith(".")
|
||||||
|
result["path_specified"] = bool(result["path"])
|
||||||
|
|
||||||
|
return cookielib.Cookie(**result)
|
||||||
|
|
||||||
|
|
||||||
|
def morsel_to_cookie(morsel: Morsel[Any]) -> Cookie:
|
||||||
|
"""Convert a Morsel object into a Cookie containing the one k/v pair."""
|
||||||
|
|
||||||
|
expires: int | None = None
|
||||||
|
if morsel["max-age"]:
|
||||||
|
try:
|
||||||
|
expires = int(time.time() + int(morsel["max-age"]))
|
||||||
|
except ValueError:
|
||||||
|
raise TypeError(f"max-age: {morsel['max-age']} must be integer")
|
||||||
|
elif morsel["expires"]:
|
||||||
|
time_template = "%a, %d-%b-%Y %H:%M:%S GMT"
|
||||||
|
expires = calendar.timegm(time.strptime(morsel["expires"], time_template))
|
||||||
|
return create_cookie(
|
||||||
|
comment=morsel["comment"],
|
||||||
|
comment_url=bool(morsel["comment"]),
|
||||||
|
discard=False,
|
||||||
|
domain=morsel["domain"],
|
||||||
|
expires=expires,
|
||||||
|
name=morsel.key,
|
||||||
|
path=morsel["path"],
|
||||||
|
port=None,
|
||||||
|
rest={"HttpOnly": morsel["httponly"]},
|
||||||
|
rfc2109=False,
|
||||||
|
secure=bool(morsel["secure"]),
|
||||||
|
value=morsel.value,
|
||||||
|
version=morsel["version"] or 0,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
_CookieJarT = TypeVar("_CookieJarT", bound=CookieJar)
|
||||||
|
|
||||||
|
|
||||||
|
@overload
|
||||||
|
def cookiejar_from_dict(
|
||||||
|
cookie_dict: dict[str, str] | None,
|
||||||
|
cookiejar: None = None,
|
||||||
|
overwrite: bool = True,
|
||||||
|
) -> RequestsCookieJar: ...
|
||||||
|
|
||||||
|
|
||||||
|
@overload
|
||||||
|
def cookiejar_from_dict(
|
||||||
|
cookie_dict: dict[str, str] | None,
|
||||||
|
cookiejar: _CookieJarT,
|
||||||
|
overwrite: bool = True,
|
||||||
|
) -> _CookieJarT: ...
|
||||||
|
|
||||||
|
|
||||||
|
def cookiejar_from_dict(
|
||||||
|
cookie_dict: dict[str, str] | None,
|
||||||
|
cookiejar: CookieJar | None = None,
|
||||||
|
overwrite: bool = True,
|
||||||
|
) -> CookieJar:
|
||||||
|
"""Returns a CookieJar from a key/value dictionary.
|
||||||
|
|
||||||
|
:param cookie_dict: Dict of key/values to insert into CookieJar.
|
||||||
|
:param cookiejar: (optional) A cookiejar to add the cookies to.
|
||||||
|
:param overwrite: (optional) If False, will not replace cookies
|
||||||
|
already in the jar with new ones.
|
||||||
|
:rtype: CookieJar
|
||||||
|
"""
|
||||||
|
if cookiejar is None:
|
||||||
|
cookiejar = RequestsCookieJar()
|
||||||
|
|
||||||
|
if cookie_dict is not None:
|
||||||
|
names_from_jar = [cookie.name for cookie in cookiejar]
|
||||||
|
for name in cookie_dict:
|
||||||
|
if overwrite or (name not in names_from_jar):
|
||||||
|
cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))
|
||||||
|
|
||||||
|
return cookiejar
|
||||||
|
|
||||||
|
|
||||||
|
def merge_cookies(
|
||||||
|
cookiejar: CookieJar, cookies: dict[str, str] | CookieJar | None
|
||||||
|
) -> CookieJar:
|
||||||
|
"""Add cookies to cookiejar and returns a merged CookieJar.
|
||||||
|
|
||||||
|
:param cookiejar: CookieJar object to add the cookies to.
|
||||||
|
:param cookies: Dictionary or CookieJar object to be added.
|
||||||
|
:rtype: CookieJar
|
||||||
|
"""
|
||||||
|
if not isinstance(cookiejar, cookielib.CookieJar): # type: ignore[reportUnnecessaryIsInstance] # runtime guard
|
||||||
|
raise ValueError("You can only merge into CookieJar")
|
||||||
|
|
||||||
|
if isinstance(cookies, dict):
|
||||||
|
cookiejar = cookiejar_from_dict(cookies, cookiejar=cookiejar, overwrite=False)
|
||||||
|
elif isinstance(cookies, cookielib.CookieJar):
|
||||||
|
if update_method := getattr(cookiejar, "update", None):
|
||||||
|
update_method(cookies)
|
||||||
|
else:
|
||||||
|
for cookie_in_jar in cookies:
|
||||||
|
cookiejar.set_cookie(cookie_in_jar)
|
||||||
|
|
||||||
|
return cookiejar
|
||||||
+162
@@ -0,0 +1,162 @@
|
|||||||
|
"""
|
||||||
|
requests.exceptions
|
||||||
|
~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This module contains the set of Requests' exceptions.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
|
from urllib3.exceptions import HTTPError as BaseHTTPError
|
||||||
|
|
||||||
|
from .compat import JSONDecodeError as CompatJSONDecodeError
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from .models import PreparedRequest, Request, Response
|
||||||
|
|
||||||
|
|
||||||
|
class RequestException(IOError):
|
||||||
|
"""There was an ambiguous exception that occurred while handling your
|
||||||
|
request.
|
||||||
|
"""
|
||||||
|
|
||||||
|
response: Response | None
|
||||||
|
request: Request | PreparedRequest | None
|
||||||
|
|
||||||
|
def __init__(self, *args: Any, **kwargs: Any) -> None:
|
||||||
|
"""Initialize RequestException with `request` and `response` objects."""
|
||||||
|
response: Response | None = kwargs.pop("response", None)
|
||||||
|
self.response = response
|
||||||
|
self.request = kwargs.pop("request", None)
|
||||||
|
if response is not None and not self.request and hasattr(response, "request"):
|
||||||
|
self.request = response.request
|
||||||
|
super().__init__(*args, **kwargs)
|
||||||
|
|
||||||
|
|
||||||
|
class InvalidJSONError(RequestException):
|
||||||
|
"""A JSON error occurred."""
|
||||||
|
|
||||||
|
|
||||||
|
class JSONDecodeError(InvalidJSONError, CompatJSONDecodeError):
|
||||||
|
"""Couldn't decode the text into json"""
|
||||||
|
|
||||||
|
def __init__(self, *args: Any, **kwargs: Any) -> None:
|
||||||
|
"""
|
||||||
|
Construct the JSONDecodeError instance first with all
|
||||||
|
args. Then use it's args to construct the IOError so that
|
||||||
|
the json specific args aren't used as IOError specific args
|
||||||
|
and the error message from JSONDecodeError is preserved.
|
||||||
|
"""
|
||||||
|
CompatJSONDecodeError.__init__(self, *args)
|
||||||
|
InvalidJSONError.__init__(self, *self.args, **kwargs)
|
||||||
|
|
||||||
|
def __reduce__(self) -> tuple[Any, ...] | str:
|
||||||
|
"""
|
||||||
|
The __reduce__ method called when pickling the object must
|
||||||
|
be the one from the JSONDecodeError (be it json/simplejson)
|
||||||
|
as it expects all the arguments for instantiation, not just
|
||||||
|
one like the IOError, and the MRO would by default call the
|
||||||
|
__reduce__ method from the IOError due to the inheritance order.
|
||||||
|
"""
|
||||||
|
return CompatJSONDecodeError.__reduce__(self)
|
||||||
|
|
||||||
|
|
||||||
|
class HTTPError(RequestException):
|
||||||
|
"""An HTTP error occurred."""
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectionError(RequestException):
|
||||||
|
"""A Connection error occurred."""
|
||||||
|
|
||||||
|
|
||||||
|
class ProxyError(ConnectionError):
|
||||||
|
"""A proxy error occurred."""
|
||||||
|
|
||||||
|
|
||||||
|
class SSLError(ConnectionError):
|
||||||
|
"""An SSL error occurred."""
|
||||||
|
|
||||||
|
|
||||||
|
class Timeout(RequestException):
|
||||||
|
"""The request timed out.
|
||||||
|
|
||||||
|
Catching this error will catch both
|
||||||
|
:exc:`~requests.exceptions.ConnectTimeout` and
|
||||||
|
:exc:`~requests.exceptions.ReadTimeout` errors.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
class ConnectTimeout(ConnectionError, Timeout):
|
||||||
|
"""The request timed out while trying to connect to the remote server.
|
||||||
|
|
||||||
|
Requests that produced this error are safe to retry.
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
class ReadTimeout(Timeout):
|
||||||
|
"""The server did not send any data in the allotted amount of time."""
|
||||||
|
|
||||||
|
|
||||||
|
class URLRequired(RequestException):
|
||||||
|
"""A valid URL is required to make a request."""
|
||||||
|
|
||||||
|
|
||||||
|
class TooManyRedirects(RequestException):
|
||||||
|
"""Too many redirects."""
|
||||||
|
|
||||||
|
|
||||||
|
class MissingSchema(RequestException, ValueError):
|
||||||
|
"""The URL scheme (e.g. http or https) is missing."""
|
||||||
|
|
||||||
|
|
||||||
|
class InvalidSchema(RequestException, ValueError):
|
||||||
|
"""The URL scheme provided is either invalid or unsupported."""
|
||||||
|
|
||||||
|
|
||||||
|
class InvalidURL(RequestException, ValueError):
|
||||||
|
"""The URL provided was somehow invalid."""
|
||||||
|
|
||||||
|
|
||||||
|
class InvalidHeader(RequestException, ValueError):
|
||||||
|
"""The header value provided was somehow invalid."""
|
||||||
|
|
||||||
|
|
||||||
|
class InvalidProxyURL(InvalidURL):
|
||||||
|
"""The proxy URL provided is invalid."""
|
||||||
|
|
||||||
|
|
||||||
|
class ChunkedEncodingError(RequestException):
|
||||||
|
"""The server declared chunked encoding but sent an invalid chunk."""
|
||||||
|
|
||||||
|
|
||||||
|
class ContentDecodingError(RequestException, BaseHTTPError):
|
||||||
|
"""Failed to decode response content."""
|
||||||
|
|
||||||
|
|
||||||
|
class StreamConsumedError(RequestException, TypeError):
|
||||||
|
"""The content for this response was already consumed."""
|
||||||
|
|
||||||
|
|
||||||
|
class RetryError(RequestException):
|
||||||
|
"""Custom retries logic failed"""
|
||||||
|
|
||||||
|
|
||||||
|
class UnrewindableBodyError(RequestException):
|
||||||
|
"""Requests encountered an error when trying to rewind a body."""
|
||||||
|
|
||||||
|
|
||||||
|
# Warnings
|
||||||
|
|
||||||
|
|
||||||
|
class RequestsWarning(Warning):
|
||||||
|
"""Base warning for Requests."""
|
||||||
|
|
||||||
|
|
||||||
|
class FileModeWarning(RequestsWarning, DeprecationWarning):
|
||||||
|
"""A file was opened in text mode, but Requests determined its binary length."""
|
||||||
|
|
||||||
|
|
||||||
|
class RequestsDependencyWarning(RequestsWarning):
|
||||||
|
"""An imported dependency doesn't match the expected version range."""
|
||||||
@@ -0,0 +1,134 @@
|
|||||||
|
"""Module containing bug report helper(s)."""
|
||||||
|
|
||||||
|
# pyright: reportUnknownMemberType=false
|
||||||
|
|
||||||
|
import json
|
||||||
|
import platform
|
||||||
|
import ssl
|
||||||
|
import sys
|
||||||
|
from typing import Any
|
||||||
|
|
||||||
|
import idna
|
||||||
|
import urllib3
|
||||||
|
|
||||||
|
from . import __version__ as requests_version
|
||||||
|
|
||||||
|
try:
|
||||||
|
import charset_normalizer
|
||||||
|
except ImportError:
|
||||||
|
charset_normalizer = None
|
||||||
|
|
||||||
|
try:
|
||||||
|
import chardet # type: ignore[import-not-found]
|
||||||
|
except ImportError:
|
||||||
|
chardet = None
|
||||||
|
|
||||||
|
try:
|
||||||
|
from urllib3.contrib import pyopenssl
|
||||||
|
except ImportError:
|
||||||
|
pyopenssl = None
|
||||||
|
OpenSSL = None
|
||||||
|
cryptography = None
|
||||||
|
else:
|
||||||
|
import cryptography # type: ignore[import-not-found]
|
||||||
|
import OpenSSL # type: ignore[import-not-found]
|
||||||
|
|
||||||
|
|
||||||
|
def _implementation():
|
||||||
|
"""Return a dict with the Python implementation and version.
|
||||||
|
|
||||||
|
Provide both the name and the version of the Python implementation
|
||||||
|
currently running. For example, on CPython 3.10.3 it will return
|
||||||
|
{'name': 'CPython', 'version': '3.10.3'}.
|
||||||
|
|
||||||
|
This function works best on CPython and PyPy: in particular, it probably
|
||||||
|
doesn't work for Jython or IronPython. Future investigation should be done
|
||||||
|
to work out the correct shape of the code for those platforms.
|
||||||
|
"""
|
||||||
|
implementation = platform.python_implementation()
|
||||||
|
|
||||||
|
if implementation == "CPython":
|
||||||
|
implementation_version = platform.python_version()
|
||||||
|
elif implementation == "PyPy":
|
||||||
|
pypy = sys.pypy_version_info # type: ignore[attr-defined]
|
||||||
|
implementation_version = f"{pypy.major}.{pypy.minor}.{pypy.micro}"
|
||||||
|
if sys.pypy_version_info.releaselevel != "final": # type: ignore[attr-defined]
|
||||||
|
implementation_version = "".join(
|
||||||
|
[implementation_version, sys.pypy_version_info.releaselevel] # type: ignore[attr-defined]
|
||||||
|
)
|
||||||
|
elif implementation == "Jython":
|
||||||
|
implementation_version = platform.python_version() # Complete Guess
|
||||||
|
elif implementation == "IronPython":
|
||||||
|
implementation_version = platform.python_version() # Complete Guess
|
||||||
|
else:
|
||||||
|
implementation_version = "Unknown"
|
||||||
|
|
||||||
|
return {"name": implementation, "version": implementation_version}
|
||||||
|
|
||||||
|
|
||||||
|
def info() -> dict[str, Any]:
|
||||||
|
"""Generate information for a bug report."""
|
||||||
|
try:
|
||||||
|
platform_info = {
|
||||||
|
"system": platform.system(),
|
||||||
|
"release": platform.release(),
|
||||||
|
}
|
||||||
|
except OSError:
|
||||||
|
platform_info = {
|
||||||
|
"system": "Unknown",
|
||||||
|
"release": "Unknown",
|
||||||
|
}
|
||||||
|
|
||||||
|
implementation_info = _implementation()
|
||||||
|
urllib3_info = {"version": urllib3.__version__} # type: ignore[reportPrivateImportUsage]
|
||||||
|
charset_normalizer_info = {"version": None}
|
||||||
|
chardet_info: dict[str, str | None] = {"version": None}
|
||||||
|
if charset_normalizer:
|
||||||
|
charset_normalizer_info = {"version": charset_normalizer.__version__}
|
||||||
|
if chardet:
|
||||||
|
chardet_info = {"version": chardet.__version__}
|
||||||
|
|
||||||
|
pyopenssl_info: dict[str, str | None] = {
|
||||||
|
"version": None,
|
||||||
|
"openssl_version": "",
|
||||||
|
}
|
||||||
|
if OpenSSL:
|
||||||
|
pyopenssl_info = {
|
||||||
|
"version": OpenSSL.__version__,
|
||||||
|
"openssl_version": f"{OpenSSL.SSL.OPENSSL_VERSION_NUMBER:x}",
|
||||||
|
}
|
||||||
|
cryptography_info = {
|
||||||
|
"version": getattr(cryptography, "__version__", ""),
|
||||||
|
}
|
||||||
|
idna_info = {
|
||||||
|
"version": getattr(idna, "__version__", ""),
|
||||||
|
}
|
||||||
|
|
||||||
|
system_ssl = ssl.OPENSSL_VERSION_NUMBER
|
||||||
|
system_ssl_info = {"version": f"{system_ssl:x}" if system_ssl is not None else ""} # type: ignore[reportUnnecessaryComparison]
|
||||||
|
|
||||||
|
return {
|
||||||
|
"platform": platform_info,
|
||||||
|
"implementation": implementation_info,
|
||||||
|
"system_ssl": system_ssl_info,
|
||||||
|
"using_pyopenssl": pyopenssl is not None,
|
||||||
|
"using_charset_normalizer": chardet is None,
|
||||||
|
"pyOpenSSL": pyopenssl_info,
|
||||||
|
"urllib3": urllib3_info,
|
||||||
|
"chardet": chardet_info,
|
||||||
|
"charset_normalizer": charset_normalizer_info,
|
||||||
|
"cryptography": cryptography_info,
|
||||||
|
"idna": idna_info,
|
||||||
|
"requests": {
|
||||||
|
"version": requests_version,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
"""Pretty-print the bug information as JSON."""
|
||||||
|
print(json.dumps(info(), sort_keys=True, indent=2))
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
"""
|
||||||
|
requests.hooks
|
||||||
|
~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This module provides the capabilities for the Requests hooks system.
|
||||||
|
|
||||||
|
Available hooks:
|
||||||
|
|
||||||
|
``response``:
|
||||||
|
The response generated from a Request.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections.abc import Callable, Iterable
|
||||||
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from . import _types as _t
|
||||||
|
from .models import Response
|
||||||
|
|
||||||
|
HOOKS: list[str] = ["response"]
|
||||||
|
|
||||||
|
|
||||||
|
def default_hooks() -> dict[str, list[_t.HookType]]:
|
||||||
|
return {event: [] for event in HOOKS}
|
||||||
|
|
||||||
|
|
||||||
|
# TODO: response is the only one
|
||||||
|
|
||||||
|
|
||||||
|
def dispatch_hook(
|
||||||
|
key: str,
|
||||||
|
hooks: _t.HooksInputType | None,
|
||||||
|
hook_data: Response,
|
||||||
|
**kwargs: Any,
|
||||||
|
) -> Response:
|
||||||
|
"""Dispatches a hook dictionary on a given piece of data."""
|
||||||
|
hooks_dict = hooks or {}
|
||||||
|
hook_list: Iterable[_t.HookType] | _t.HookType | None = hooks_dict.get(key)
|
||||||
|
if hook_list:
|
||||||
|
if isinstance(hook_list, Callable):
|
||||||
|
hook_list = [hook_list]
|
||||||
|
for hook in hook_list:
|
||||||
|
_hook_data = hook(hook_data, **kwargs)
|
||||||
|
if _hook_data is not None:
|
||||||
|
hook_data = _hook_data
|
||||||
|
return hook_data
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
|||||||
|
import sys
|
||||||
|
|
||||||
|
from .compat import chardet
|
||||||
|
|
||||||
|
# This code exists for backwards compatibility reasons.
|
||||||
|
# I don't like it either. Just look the other way. :)
|
||||||
|
|
||||||
|
for package in ("urllib3", "idna"):
|
||||||
|
locals()[package] = __import__(package)
|
||||||
|
# This traversal is apparently necessary such that the identities are
|
||||||
|
# preserved (requests.packages.urllib3.* is urllib3.*)
|
||||||
|
for mod in list(sys.modules):
|
||||||
|
if mod == package or mod.startswith(f"{package}."):
|
||||||
|
sys.modules[f"requests.packages.{mod}"] = sys.modules[mod]
|
||||||
|
|
||||||
|
if chardet is not None:
|
||||||
|
target = chardet.__name__
|
||||||
|
for mod in list(sys.modules):
|
||||||
|
if mod == target or mod.startswith(f"{target}."):
|
||||||
|
imported_mod = sys.modules[mod]
|
||||||
|
sys.modules[f"requests.packages.{mod}"] = imported_mod
|
||||||
|
mod = mod.replace(target, "chardet")
|
||||||
|
sys.modules[f"requests.packages.{mod}"] = imported_mod
|
||||||
@@ -0,0 +1,920 @@
|
|||||||
|
"""
|
||||||
|
requests.sessions
|
||||||
|
~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This module provides a Session object to manage and persist settings across
|
||||||
|
requests (cookies, auth, proxies).
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from collections import OrderedDict
|
||||||
|
from collections.abc import Generator, Mapping, MutableMapping
|
||||||
|
from datetime import timedelta
|
||||||
|
from typing import TYPE_CHECKING, Any, cast
|
||||||
|
|
||||||
|
from ._internal_utils import to_native_string
|
||||||
|
from ._types import is_prepared as _is_prepared
|
||||||
|
from .adapters import HTTPAdapter
|
||||||
|
from .auth import _basic_auth_str # type: ignore[reportPrivateUsage]
|
||||||
|
from .compat import cookielib, urljoin, urlparse
|
||||||
|
from .cookies import (
|
||||||
|
RequestsCookieJar,
|
||||||
|
cookiejar_from_dict,
|
||||||
|
extract_cookies_to_jar,
|
||||||
|
merge_cookies,
|
||||||
|
)
|
||||||
|
from .exceptions import (
|
||||||
|
ChunkedEncodingError,
|
||||||
|
ContentDecodingError,
|
||||||
|
InvalidSchema,
|
||||||
|
TooManyRedirects,
|
||||||
|
)
|
||||||
|
from .hooks import default_hooks, dispatch_hook
|
||||||
|
|
||||||
|
# formerly defined here, reexposed here for backward compatibility
|
||||||
|
from .models import ( # noqa: F401
|
||||||
|
DEFAULT_REDIRECT_LIMIT,
|
||||||
|
REDIRECT_STATI, # type: ignore[reportUnusedImport]
|
||||||
|
PreparedRequest,
|
||||||
|
Request,
|
||||||
|
Response,
|
||||||
|
)
|
||||||
|
from .status_codes import codes
|
||||||
|
from .structures import CaseInsensitiveDict
|
||||||
|
from .utils import ( # noqa: F401
|
||||||
|
DEFAULT_PORTS,
|
||||||
|
default_headers,
|
||||||
|
get_auth_from_url,
|
||||||
|
get_environ_proxies,
|
||||||
|
get_netrc_auth,
|
||||||
|
requote_uri,
|
||||||
|
resolve_proxies,
|
||||||
|
rewind_body,
|
||||||
|
should_bypass_proxies, # type: ignore[reportUnusedImport] # re-export for external consumers
|
||||||
|
to_key_val_list,
|
||||||
|
)
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from http.cookiejar import CookieJar
|
||||||
|
|
||||||
|
from typing_extensions import Self, Unpack
|
||||||
|
|
||||||
|
from . import _types as _t
|
||||||
|
from .adapters import BaseAdapter
|
||||||
|
|
||||||
|
# Preferred clock, based on which one is more accurate on a given system.
|
||||||
|
if sys.platform == "win32":
|
||||||
|
preferred_clock = time.perf_counter
|
||||||
|
else:
|
||||||
|
preferred_clock = time.time
|
||||||
|
|
||||||
|
|
||||||
|
def merge_setting(
|
||||||
|
request_setting: Any, session_setting: Any, dict_class: type = OrderedDict
|
||||||
|
) -> Any:
|
||||||
|
"""Determines appropriate setting for a given request, taking into account
|
||||||
|
the explicit setting on that request, and the setting in the session. If a
|
||||||
|
setting is a dictionary, they will be merged together using `dict_class`
|
||||||
|
"""
|
||||||
|
|
||||||
|
if session_setting is None:
|
||||||
|
return request_setting
|
||||||
|
|
||||||
|
if request_setting is None:
|
||||||
|
return session_setting
|
||||||
|
|
||||||
|
# Bypass if not a dictionary (e.g. verify)
|
||||||
|
if not (
|
||||||
|
isinstance(session_setting, Mapping) and isinstance(request_setting, Mapping)
|
||||||
|
):
|
||||||
|
return request_setting
|
||||||
|
|
||||||
|
merged_setting = dict_class(to_key_val_list(session_setting)) # type: ignore[arg-type] # isinstance narrows Any to Mapping[Unknown]
|
||||||
|
merged_setting.update(to_key_val_list(request_setting)) # type: ignore[arg-type]
|
||||||
|
|
||||||
|
# Remove keys that are set to None. Extract keys first to avoid altering
|
||||||
|
# the dictionary during iteration.
|
||||||
|
none_keys = [k for (k, v) in merged_setting.items() if v is None]
|
||||||
|
for key in none_keys:
|
||||||
|
del merged_setting[key]
|
||||||
|
|
||||||
|
return merged_setting
|
||||||
|
|
||||||
|
|
||||||
|
def merge_hooks(
|
||||||
|
request_hooks: _t.HooksType,
|
||||||
|
session_hooks: _t.HooksType,
|
||||||
|
dict_class: type = OrderedDict,
|
||||||
|
) -> _t.HooksType:
|
||||||
|
"""Properly merges both requests and session hooks.
|
||||||
|
|
||||||
|
This is necessary because when request_hooks == {'response': []}, the
|
||||||
|
merge breaks Session hooks entirely.
|
||||||
|
"""
|
||||||
|
if session_hooks is None or session_hooks.get("response") == []:
|
||||||
|
return request_hooks
|
||||||
|
|
||||||
|
if request_hooks is None or request_hooks.get("response") == []:
|
||||||
|
return session_hooks
|
||||||
|
|
||||||
|
return merge_setting(request_hooks, session_hooks, dict_class)
|
||||||
|
|
||||||
|
|
||||||
|
class SessionRedirectMixin:
|
||||||
|
max_redirects: int
|
||||||
|
trust_env: bool
|
||||||
|
cookies: RequestsCookieJar
|
||||||
|
|
||||||
|
def send(self, request: PreparedRequest, **kwargs: Any) -> Response: ...
|
||||||
|
|
||||||
|
def get_redirect_target(self, resp: Response) -> str | None:
|
||||||
|
"""Receives a Response. Returns a redirect URI or ``None``"""
|
||||||
|
# Due to the nature of how requests processes redirects this method will
|
||||||
|
# be called at least once upon the original response and at least twice
|
||||||
|
# on each subsequent redirect response (if any).
|
||||||
|
# If a custom mixin is used to handle this logic, it may be advantageous
|
||||||
|
# to cache the redirect location onto the response object as a private
|
||||||
|
# attribute.
|
||||||
|
if resp.is_redirect:
|
||||||
|
location = resp.headers["location"]
|
||||||
|
# Currently the underlying http module on py3 decode headers
|
||||||
|
# in latin1, but empirical evidence suggests that latin1 is very
|
||||||
|
# rarely used with non-ASCII characters in HTTP headers.
|
||||||
|
# It is more likely to get UTF8 header rather than latin1.
|
||||||
|
# This causes incorrect handling of UTF8 encoded location headers.
|
||||||
|
# To solve this, we re-encode the location in latin1.
|
||||||
|
location = location.encode("latin1")
|
||||||
|
return to_native_string(location, "utf8")
|
||||||
|
return None
|
||||||
|
|
||||||
|
def should_strip_auth(self, old_url: str, new_url: str) -> bool:
|
||||||
|
"""Decide whether Authorization header should be removed when redirecting"""
|
||||||
|
old_parsed = urlparse(old_url)
|
||||||
|
new_parsed = urlparse(new_url)
|
||||||
|
if old_parsed.hostname != new_parsed.hostname:
|
||||||
|
return True
|
||||||
|
# Special case: allow http -> https redirect when using the standard
|
||||||
|
# ports. This isn't specified by RFC 7235, but is kept to avoid
|
||||||
|
# breaking backwards compatibility with older versions of requests
|
||||||
|
# that allowed any redirects on the same host.
|
||||||
|
if (
|
||||||
|
old_parsed.scheme == "http"
|
||||||
|
and old_parsed.port in (80, None)
|
||||||
|
and new_parsed.scheme == "https"
|
||||||
|
and new_parsed.port in (443, None)
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Handle default port usage corresponding to scheme.
|
||||||
|
changed_port = old_parsed.port != new_parsed.port
|
||||||
|
changed_scheme = old_parsed.scheme != new_parsed.scheme
|
||||||
|
default_port = (DEFAULT_PORTS.get(old_parsed.scheme, None), None)
|
||||||
|
if (
|
||||||
|
not changed_scheme
|
||||||
|
and old_parsed.port in default_port
|
||||||
|
and new_parsed.port in default_port
|
||||||
|
):
|
||||||
|
return False
|
||||||
|
|
||||||
|
# Standard case: root URI must match
|
||||||
|
return changed_port or changed_scheme
|
||||||
|
|
||||||
|
def resolve_redirects(
|
||||||
|
self,
|
||||||
|
resp: Response,
|
||||||
|
req: PreparedRequest,
|
||||||
|
stream: bool = False,
|
||||||
|
timeout: _t.TimeoutType = None,
|
||||||
|
verify: _t.VerifyType = True,
|
||||||
|
cert: _t.CertType = None,
|
||||||
|
proxies: dict[str, str] | None = None,
|
||||||
|
yield_requests: bool = False,
|
||||||
|
**adapter_kwargs: Any,
|
||||||
|
) -> Generator[Response, None, None]:
|
||||||
|
"""Receives a Response. Returns a generator of Responses or Requests."""
|
||||||
|
|
||||||
|
hist: list[Response] = [] # keep track of history
|
||||||
|
|
||||||
|
url = self.get_redirect_target(resp)
|
||||||
|
previous_fragment = urlparse(req.url).fragment
|
||||||
|
while url:
|
||||||
|
prepared_request = req.copy()
|
||||||
|
|
||||||
|
# Update history and keep track of redirects.
|
||||||
|
resp.history = hist[:]
|
||||||
|
hist.append(resp)
|
||||||
|
|
||||||
|
try:
|
||||||
|
resp.content # Consume socket so it can be released
|
||||||
|
except (ChunkedEncodingError, ContentDecodingError, RuntimeError):
|
||||||
|
resp.raw.read(decode_content=False)
|
||||||
|
|
||||||
|
if len(resp.history) >= self.max_redirects:
|
||||||
|
raise TooManyRedirects(
|
||||||
|
f"Exceeded {self.max_redirects} redirects.", response=resp
|
||||||
|
)
|
||||||
|
|
||||||
|
# Release the connection back into the pool.
|
||||||
|
resp.close()
|
||||||
|
|
||||||
|
# Handle redirection without scheme (see: RFC 1808 Section 4)
|
||||||
|
if url.startswith("//"):
|
||||||
|
parsed_rurl = urlparse(resp.url)
|
||||||
|
url = ":".join([to_native_string(parsed_rurl.scheme), url])
|
||||||
|
|
||||||
|
# Normalize url case and attach previous fragment if needed (RFC 7231 7.1.2)
|
||||||
|
parsed = urlparse(url)
|
||||||
|
if parsed.fragment == "" and previous_fragment:
|
||||||
|
parsed = parsed._replace(fragment=previous_fragment)
|
||||||
|
elif parsed.fragment:
|
||||||
|
previous_fragment = parsed.fragment
|
||||||
|
url = parsed.geturl()
|
||||||
|
|
||||||
|
# Facilitate relative 'location' headers, as allowed by RFC 7231.
|
||||||
|
# (e.g. '/path/to/resource' instead of 'http://domain.tld/path/to/resource')
|
||||||
|
# Compliant with RFC3986, we percent encode the url.
|
||||||
|
if not parsed.netloc:
|
||||||
|
url = urljoin(resp.url, requote_uri(url))
|
||||||
|
else:
|
||||||
|
url = requote_uri(url)
|
||||||
|
|
||||||
|
prepared_request.url = to_native_string(url)
|
||||||
|
|
||||||
|
self.rebuild_method(prepared_request, resp)
|
||||||
|
|
||||||
|
# https://github.com/psf/requests/issues/1084
|
||||||
|
if resp.status_code not in (
|
||||||
|
codes.temporary_redirect,
|
||||||
|
codes.permanent_redirect,
|
||||||
|
):
|
||||||
|
# https://github.com/psf/requests/issues/3490
|
||||||
|
purged_headers = ("Content-Length", "Content-Type", "Transfer-Encoding")
|
||||||
|
for header in purged_headers:
|
||||||
|
prepared_request.headers.pop(header, None)
|
||||||
|
prepared_request.body = None
|
||||||
|
|
||||||
|
headers = prepared_request.headers
|
||||||
|
headers.pop("Cookie", None)
|
||||||
|
|
||||||
|
# Extract any cookies sent on the response to the cookiejar
|
||||||
|
# in the new request. Because we've mutated our copied prepared
|
||||||
|
# request, use the old one that we haven't yet touched.
|
||||||
|
cookie_jar = cast("CookieJar", prepared_request._cookies) # type: ignore[reportPrivateUsage]
|
||||||
|
extract_cookies_to_jar(cookie_jar, req, resp.raw)
|
||||||
|
merge_cookies(cookie_jar, self.cookies)
|
||||||
|
prepared_request.prepare_cookies(cookie_jar)
|
||||||
|
|
||||||
|
# Rebuild auth and proxy information.
|
||||||
|
proxies = self.rebuild_proxies(prepared_request, proxies)
|
||||||
|
self.rebuild_auth(prepared_request, resp)
|
||||||
|
|
||||||
|
# A failed tell() sets `_body_position` to `object()`. This non-None
|
||||||
|
# value ensures `rewindable` will be True, allowing us to raise an
|
||||||
|
# UnrewindableBodyError, instead of hanging the connection.
|
||||||
|
rewindable = prepared_request._body_position is not None and ( # type: ignore[reportPrivateUsage]
|
||||||
|
"Content-Length" in headers or "Transfer-Encoding" in headers
|
||||||
|
)
|
||||||
|
|
||||||
|
# Attempt to rewind consumed file-like object.
|
||||||
|
if rewindable:
|
||||||
|
rewind_body(prepared_request)
|
||||||
|
|
||||||
|
# Override the original request.
|
||||||
|
req = prepared_request
|
||||||
|
|
||||||
|
if yield_requests:
|
||||||
|
yield req # type: ignore[misc] # Internal use only, returns PreparedRequest
|
||||||
|
else:
|
||||||
|
resp = self.send(
|
||||||
|
req,
|
||||||
|
stream=stream,
|
||||||
|
timeout=timeout,
|
||||||
|
verify=verify,
|
||||||
|
cert=cert,
|
||||||
|
proxies=proxies,
|
||||||
|
allow_redirects=False,
|
||||||
|
**adapter_kwargs,
|
||||||
|
)
|
||||||
|
|
||||||
|
extract_cookies_to_jar(self.cookies, prepared_request, resp.raw)
|
||||||
|
|
||||||
|
# extract redirect url, if any, for the next loop
|
||||||
|
url = self.get_redirect_target(resp)
|
||||||
|
yield resp
|
||||||
|
|
||||||
|
def rebuild_auth(
|
||||||
|
self, prepared_request: PreparedRequest, response: Response
|
||||||
|
) -> None:
|
||||||
|
"""When being redirected we may want to strip authentication from the
|
||||||
|
request to avoid leaking credentials. This method intelligently removes
|
||||||
|
and reapplies authentication where possible to avoid credential loss.
|
||||||
|
"""
|
||||||
|
original_request = response.request
|
||||||
|
assert _is_prepared(original_request)
|
||||||
|
assert _is_prepared(prepared_request)
|
||||||
|
|
||||||
|
headers = prepared_request.headers
|
||||||
|
original_url = original_request.url
|
||||||
|
url = prepared_request.url
|
||||||
|
|
||||||
|
if "Authorization" in headers and self.should_strip_auth(original_url, url):
|
||||||
|
# If we get redirected to a new host, we should strip out any
|
||||||
|
# authentication headers.
|
||||||
|
del headers["Authorization"]
|
||||||
|
|
||||||
|
# .netrc might have more auth for us on our new host.
|
||||||
|
new_auth = get_netrc_auth(url) if self.trust_env else None
|
||||||
|
if new_auth is not None:
|
||||||
|
prepared_request.prepare_auth(new_auth)
|
||||||
|
|
||||||
|
def rebuild_proxies(
|
||||||
|
self,
|
||||||
|
prepared_request: PreparedRequest,
|
||||||
|
proxies: dict[str, str] | None,
|
||||||
|
) -> dict[str, str]:
|
||||||
|
"""This method re-evaluates the proxy configuration by considering the
|
||||||
|
environment variables. If we are redirected to a URL covered by
|
||||||
|
NO_PROXY, we strip the proxy configuration. Otherwise, we set missing
|
||||||
|
proxy keys for this URL (in case they were stripped by a previous
|
||||||
|
redirect).
|
||||||
|
|
||||||
|
This method also replaces the Proxy-Authorization header where
|
||||||
|
necessary.
|
||||||
|
|
||||||
|
:rtype: dict
|
||||||
|
"""
|
||||||
|
assert _is_prepared(prepared_request)
|
||||||
|
headers = prepared_request.headers
|
||||||
|
scheme = urlparse(prepared_request.url).scheme
|
||||||
|
new_proxies = resolve_proxies(prepared_request, proxies, self.trust_env)
|
||||||
|
|
||||||
|
if "Proxy-Authorization" in headers:
|
||||||
|
del headers["Proxy-Authorization"]
|
||||||
|
|
||||||
|
try:
|
||||||
|
username, password = get_auth_from_url(new_proxies[scheme])
|
||||||
|
except KeyError:
|
||||||
|
username, password = None, None
|
||||||
|
|
||||||
|
# urllib3 handles proxy authorization for us in the standard adapter.
|
||||||
|
# Avoid appending this to TLS tunneled requests where it may be leaked.
|
||||||
|
if not scheme.startswith("https") and username and password:
|
||||||
|
headers["Proxy-Authorization"] = _basic_auth_str(username, password)
|
||||||
|
|
||||||
|
return new_proxies
|
||||||
|
|
||||||
|
def rebuild_method(
|
||||||
|
self, prepared_request: PreparedRequest, response: Response
|
||||||
|
) -> None:
|
||||||
|
"""When being redirected we may want to change the method of the request
|
||||||
|
based on certain specs or browser behavior.
|
||||||
|
"""
|
||||||
|
method = prepared_request.method
|
||||||
|
|
||||||
|
# https://tools.ietf.org/html/rfc7231#section-6.4.4
|
||||||
|
if response.status_code == codes.see_other and method != "HEAD":
|
||||||
|
method = "GET"
|
||||||
|
|
||||||
|
# Do what the browsers do, despite standards...
|
||||||
|
# First, turn 302s into GETs.
|
||||||
|
if response.status_code == codes.found and method != "HEAD":
|
||||||
|
method = "GET"
|
||||||
|
|
||||||
|
# Second, if a POST is responded to with a 301, turn it into a GET.
|
||||||
|
# This bizarre behaviour is explained in Issue 1704.
|
||||||
|
if response.status_code == codes.moved and method == "POST":
|
||||||
|
method = "GET"
|
||||||
|
|
||||||
|
prepared_request.method = method
|
||||||
|
|
||||||
|
|
||||||
|
class Session(SessionRedirectMixin):
|
||||||
|
"""A Requests session.
|
||||||
|
|
||||||
|
Provides cookie persistence, connection-pooling, and configuration.
|
||||||
|
|
||||||
|
Basic Usage::
|
||||||
|
|
||||||
|
>>> import requests
|
||||||
|
>>> s = requests.Session()
|
||||||
|
>>> s.get('https://httpbin.org/get')
|
||||||
|
<Response [200]>
|
||||||
|
|
||||||
|
Or as a context manager::
|
||||||
|
|
||||||
|
>>> with requests.Session() as s:
|
||||||
|
... s.get('https://httpbin.org/get')
|
||||||
|
<Response [200]>
|
||||||
|
"""
|
||||||
|
|
||||||
|
headers: CaseInsensitiveDict[str]
|
||||||
|
auth: _t.AuthType
|
||||||
|
proxies: dict[str, str]
|
||||||
|
hooks: dict[str, list[_t.HookType]]
|
||||||
|
params: MutableMapping[str, Any]
|
||||||
|
stream: bool
|
||||||
|
verify: _t.VerifyType
|
||||||
|
cert: _t.CertType
|
||||||
|
max_redirects: int
|
||||||
|
trust_env: bool
|
||||||
|
cookies: RequestsCookieJar
|
||||||
|
adapters: MutableMapping[str, BaseAdapter]
|
||||||
|
|
||||||
|
__attrs__: list[str] = [
|
||||||
|
"headers",
|
||||||
|
"cookies",
|
||||||
|
"auth",
|
||||||
|
"proxies",
|
||||||
|
"hooks",
|
||||||
|
"params",
|
||||||
|
"verify",
|
||||||
|
"cert",
|
||||||
|
"adapters",
|
||||||
|
"stream",
|
||||||
|
"trust_env",
|
||||||
|
"max_redirects",
|
||||||
|
]
|
||||||
|
|
||||||
|
def __init__(self) -> None:
|
||||||
|
#: A case-insensitive dictionary of headers to be sent on each
|
||||||
|
#: :class:`Request <Request>` sent from this
|
||||||
|
#: :class:`Session <Session>`.
|
||||||
|
self.headers = default_headers()
|
||||||
|
|
||||||
|
#: Default Authentication tuple or object to attach to
|
||||||
|
#: :class:`Request <Request>`.
|
||||||
|
self.auth = None
|
||||||
|
|
||||||
|
#: Dictionary mapping protocol or protocol and host to the URL of the proxy
|
||||||
|
#: (e.g. {'http': 'foo.bar:3128', 'http://host.name': 'foo.bar:4012'}) to
|
||||||
|
#: be used on each :class:`Request <Request>`.
|
||||||
|
self.proxies = {}
|
||||||
|
|
||||||
|
#: Event-handling hooks.
|
||||||
|
self.hooks = default_hooks()
|
||||||
|
|
||||||
|
#: Dictionary of querystring data to attach to each
|
||||||
|
#: :class:`Request <Request>`. The dictionary values may be lists for
|
||||||
|
#: representing multivalued query parameters.
|
||||||
|
self.params = {}
|
||||||
|
|
||||||
|
#: Stream response content default.
|
||||||
|
self.stream = False
|
||||||
|
|
||||||
|
#: SSL Verification default.
|
||||||
|
#: Defaults to `True`, requiring requests to verify the TLS certificate at the
|
||||||
|
#: remote end.
|
||||||
|
#: If verify is set to `False`, requests will accept any TLS certificate
|
||||||
|
#: presented by the server, and will ignore hostname mismatches and/or
|
||||||
|
#: expired certificates, which will make your application vulnerable to
|
||||||
|
#: man-in-the-middle (MitM) attacks.
|
||||||
|
#: Only set this to `False` for testing.
|
||||||
|
#: If verify is set to a string, it must be the path to a CA bundle file
|
||||||
|
#: that will be used to verify the TLS certificate.
|
||||||
|
self.verify = True
|
||||||
|
|
||||||
|
#: SSL client certificate default, if String, path to ssl client
|
||||||
|
#: cert file (.pem). If Tuple, ('cert', 'key') pair.
|
||||||
|
self.cert = None
|
||||||
|
|
||||||
|
#: Maximum number of redirects allowed. If the request exceeds this
|
||||||
|
#: limit, a :class:`TooManyRedirects` exception is raised.
|
||||||
|
#: This defaults to requests.models.DEFAULT_REDIRECT_LIMIT, which is
|
||||||
|
#: 30.
|
||||||
|
self.max_redirects = DEFAULT_REDIRECT_LIMIT
|
||||||
|
|
||||||
|
#: Trust environment settings for proxy configuration, default
|
||||||
|
#: authentication and similar.
|
||||||
|
self.trust_env = True
|
||||||
|
|
||||||
|
#: A CookieJar containing all currently outstanding cookies set on this
|
||||||
|
#: session. By default it is a
|
||||||
|
#: :class:`RequestsCookieJar <requests.cookies.RequestsCookieJar>`, but
|
||||||
|
#: may be any other ``cookielib.CookieJar`` compatible object.
|
||||||
|
self.cookies = cookiejar_from_dict({})
|
||||||
|
|
||||||
|
# Default connection adapters.
|
||||||
|
self.adapters = OrderedDict()
|
||||||
|
self.mount("https://", HTTPAdapter())
|
||||||
|
self.mount("http://", HTTPAdapter())
|
||||||
|
|
||||||
|
def __enter__(self) -> Self:
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __exit__(self, *args: Any) -> None:
|
||||||
|
self.close()
|
||||||
|
|
||||||
|
def prepare_request(self, request: Request) -> PreparedRequest:
|
||||||
|
"""Constructs a :class:`PreparedRequest <PreparedRequest>` for
|
||||||
|
transmission and returns it. The :class:`PreparedRequest` has settings
|
||||||
|
merged from the :class:`Request <Request>` instance and those of the
|
||||||
|
:class:`Session`.
|
||||||
|
|
||||||
|
:param request: :class:`Request` instance to prepare with this
|
||||||
|
session's settings.
|
||||||
|
:rtype: requests.PreparedRequest
|
||||||
|
"""
|
||||||
|
url = cast("_t.UriType", request.url)
|
||||||
|
method = cast(str, request.method)
|
||||||
|
|
||||||
|
cookies = request.cookies or {}
|
||||||
|
|
||||||
|
# Bootstrap CookieJar.
|
||||||
|
if not isinstance(cookies, cookielib.CookieJar):
|
||||||
|
cookies = cookiejar_from_dict(cookies)
|
||||||
|
|
||||||
|
# Merge with session cookies
|
||||||
|
merged_cookies = merge_cookies(
|
||||||
|
merge_cookies(RequestsCookieJar(), self.cookies), cookies
|
||||||
|
)
|
||||||
|
|
||||||
|
# Set environment's basic authentication if not explicitly set.
|
||||||
|
auth = request.auth
|
||||||
|
if self.trust_env and not auth and not self.auth:
|
||||||
|
auth = get_netrc_auth(url)
|
||||||
|
|
||||||
|
p = PreparedRequest()
|
||||||
|
p.prepare(
|
||||||
|
method=method.upper(),
|
||||||
|
url=url,
|
||||||
|
files=request.files,
|
||||||
|
data=request.data,
|
||||||
|
json=request.json,
|
||||||
|
headers=merge_setting(
|
||||||
|
request.headers, self.headers, dict_class=CaseInsensitiveDict
|
||||||
|
),
|
||||||
|
params=merge_setting(request.params, self.params),
|
||||||
|
auth=merge_setting(auth, self.auth),
|
||||||
|
cookies=merged_cookies,
|
||||||
|
hooks=merge_hooks(request.hooks, self.hooks),
|
||||||
|
)
|
||||||
|
return p
|
||||||
|
|
||||||
|
def request(
|
||||||
|
self,
|
||||||
|
method: str,
|
||||||
|
url: _t.UriType,
|
||||||
|
params: _t.ParamsType = None,
|
||||||
|
data: _t.DataType = None,
|
||||||
|
headers: _t.HeadersType = None,
|
||||||
|
cookies: RequestsCookieJar | CookieJar | dict[str, str] | None = None,
|
||||||
|
files: _t.FilesType = None,
|
||||||
|
auth: _t.AuthType = None,
|
||||||
|
timeout: _t.TimeoutType = None,
|
||||||
|
allow_redirects: bool = True,
|
||||||
|
proxies: dict[str, str] | None = None,
|
||||||
|
hooks: _t.HooksInputType | None = None,
|
||||||
|
stream: bool | None = None,
|
||||||
|
verify: _t.VerifyType | None = None,
|
||||||
|
cert: _t.CertType = None,
|
||||||
|
json: _t.JsonType = None,
|
||||||
|
) -> Response:
|
||||||
|
"""Constructs a :class:`Request <Request>`, prepares it and sends it.
|
||||||
|
Returns :class:`Response <Response>` object.
|
||||||
|
|
||||||
|
:param method: method for the new :class:`Request` object.
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param params: (optional) Dictionary or bytes to be sent in the query
|
||||||
|
string for the :class:`Request`.
|
||||||
|
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
|
||||||
|
object to send in the body of the :class:`Request`.
|
||||||
|
:param json: (optional) json to send in the body of the
|
||||||
|
:class:`Request`.
|
||||||
|
:param headers: (optional) Dictionary of HTTP Headers to send with the
|
||||||
|
:class:`Request`.
|
||||||
|
:param cookies: (optional) Dict or CookieJar object to send with the
|
||||||
|
:class:`Request`.
|
||||||
|
:param files: (optional) Dictionary of ``'filename': file-like-objects``
|
||||||
|
for multipart encoding upload.
|
||||||
|
:param auth: (optional) Auth tuple or callable to enable
|
||||||
|
Basic/Digest/Custom HTTP Auth.
|
||||||
|
:param timeout: (optional) How many seconds to wait for the server to send
|
||||||
|
data before giving up, as a float, or a :ref:`(connect timeout,
|
||||||
|
read timeout) <timeouts>` tuple.
|
||||||
|
:type timeout: float or tuple
|
||||||
|
:param allow_redirects: (optional) Set to True by default.
|
||||||
|
:type allow_redirects: bool
|
||||||
|
:param proxies: (optional) Dictionary mapping protocol or protocol and
|
||||||
|
hostname to the URL of the proxy.
|
||||||
|
:param hooks: (optional) Dictionary mapping hook name to one event or
|
||||||
|
list of events, event must be callable.
|
||||||
|
:param stream: (optional) whether to immediately download the response
|
||||||
|
content. Defaults to ``False``.
|
||||||
|
:param verify: (optional) Either a boolean, in which case it controls whether we verify
|
||||||
|
the server's TLS certificate, or a string, in which case it must be a path
|
||||||
|
to a CA bundle to use. Defaults to ``True``. When set to
|
||||||
|
``False``, requests will accept any TLS certificate presented by
|
||||||
|
the server, and will ignore hostname mismatches and/or expired
|
||||||
|
certificates, which will make your application vulnerable to
|
||||||
|
man-in-the-middle (MitM) attacks. Setting verify to ``False``
|
||||||
|
may be useful during local development or testing.
|
||||||
|
:param cert: (optional) if String, path to ssl client cert file (.pem).
|
||||||
|
If Tuple, ('cert', 'key') pair.
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
if isinstance(url, bytes):
|
||||||
|
url = url.decode("utf-8")
|
||||||
|
|
||||||
|
# Create the Request.
|
||||||
|
req = Request(
|
||||||
|
method=method.upper(),
|
||||||
|
url=url,
|
||||||
|
headers=headers,
|
||||||
|
files=files,
|
||||||
|
data=data or {},
|
||||||
|
json=json,
|
||||||
|
params=params or {},
|
||||||
|
auth=auth,
|
||||||
|
cookies=cookies,
|
||||||
|
hooks=hooks,
|
||||||
|
)
|
||||||
|
prep = self.prepare_request(req)
|
||||||
|
|
||||||
|
assert _is_prepared(prep)
|
||||||
|
|
||||||
|
proxies = proxies or {}
|
||||||
|
|
||||||
|
settings = self.merge_environment_settings(
|
||||||
|
prep.url, proxies, stream, verify, cert
|
||||||
|
)
|
||||||
|
|
||||||
|
# Send the request.
|
||||||
|
send_kwargs = {
|
||||||
|
"timeout": timeout,
|
||||||
|
"allow_redirects": allow_redirects,
|
||||||
|
}
|
||||||
|
send_kwargs.update(settings)
|
||||||
|
resp = self.send(prep, **send_kwargs)
|
||||||
|
|
||||||
|
return resp
|
||||||
|
|
||||||
|
def get(
|
||||||
|
self,
|
||||||
|
url: _t.UriType,
|
||||||
|
params: _t.ParamsType = None,
|
||||||
|
**kwargs: Unpack[_t.GetKwargs],
|
||||||
|
) -> Response:
|
||||||
|
r"""Sends a GET request. Returns :class:`Response` object.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param params: (optional) Dictionary, list of tuples or bytes to send
|
||||||
|
in the query string for the :class:`Request`.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes.
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
kwargs.setdefault("allow_redirects", True)
|
||||||
|
return self.request("GET", url, params=params, **kwargs)
|
||||||
|
|
||||||
|
def options(self, url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]) -> Response:
|
||||||
|
r"""Sends a OPTIONS request. Returns :class:`Response` object.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes.
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
kwargs.setdefault("allow_redirects", True)
|
||||||
|
return self.request("OPTIONS", url, **kwargs)
|
||||||
|
|
||||||
|
def head(self, url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]) -> Response:
|
||||||
|
r"""Sends a HEAD request. Returns :class:`Response` object.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes.
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
kwargs.setdefault("allow_redirects", False)
|
||||||
|
return self.request("HEAD", url, **kwargs)
|
||||||
|
|
||||||
|
def post(
|
||||||
|
self,
|
||||||
|
url: _t.UriType,
|
||||||
|
data: _t.DataType = None,
|
||||||
|
json: _t.JsonType = None,
|
||||||
|
**kwargs: Unpack[_t.PostKwargs],
|
||||||
|
) -> Response:
|
||||||
|
r"""Sends a POST request. Returns :class:`Response` object.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
|
||||||
|
object to send in the body of the :class:`Request`.
|
||||||
|
:param json: (optional) json to send in the body of the :class:`Request`.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes.
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
return self.request("POST", url, data=data, json=json, **kwargs)
|
||||||
|
|
||||||
|
def put(
|
||||||
|
self, url: _t.UriType, data: _t.DataType = None, **kwargs: Unpack[_t.DataKwargs]
|
||||||
|
) -> Response:
|
||||||
|
r"""Sends a PUT request. Returns :class:`Response` object.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
|
||||||
|
object to send in the body of the :class:`Request`.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes.
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
return self.request("PUT", url, data=data, **kwargs)
|
||||||
|
|
||||||
|
def patch(
|
||||||
|
self, url: _t.UriType, data: _t.DataType = None, **kwargs: Unpack[_t.DataKwargs]
|
||||||
|
) -> Response:
|
||||||
|
r"""Sends a PATCH request. Returns :class:`Response` object.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
|
||||||
|
object to send in the body of the :class:`Request`.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes.
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
return self.request("PATCH", url, data=data, **kwargs)
|
||||||
|
|
||||||
|
def delete(self, url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]) -> Response:
|
||||||
|
r"""Sends a DELETE request. Returns :class:`Response` object.
|
||||||
|
|
||||||
|
:param url: URL for the new :class:`Request` object.
|
||||||
|
:param \*\*kwargs: Optional arguments that ``request`` takes.
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
|
||||||
|
return self.request("DELETE", url, **kwargs)
|
||||||
|
|
||||||
|
def send(self, request: PreparedRequest, **kwargs: Any) -> Response:
|
||||||
|
"""Send a given PreparedRequest.
|
||||||
|
|
||||||
|
:rtype: requests.Response
|
||||||
|
"""
|
||||||
|
# Set defaults that the hooks can utilize to ensure they always have
|
||||||
|
# the correct parameters to reproduce the previous request.
|
||||||
|
kwargs.setdefault("stream", self.stream)
|
||||||
|
kwargs.setdefault("verify", self.verify)
|
||||||
|
kwargs.setdefault("cert", self.cert)
|
||||||
|
if "proxies" not in kwargs:
|
||||||
|
kwargs["proxies"] = resolve_proxies(request, self.proxies, self.trust_env)
|
||||||
|
|
||||||
|
# It's possible that users might accidentally send a Request object.
|
||||||
|
# Guard against that specific failure case.
|
||||||
|
if isinstance(request, Request):
|
||||||
|
raise ValueError("You can only send PreparedRequests.")
|
||||||
|
|
||||||
|
assert _is_prepared(request)
|
||||||
|
|
||||||
|
# Set up variables needed for resolve_redirects and dispatching of hooks
|
||||||
|
allow_redirects = kwargs.pop("allow_redirects", True)
|
||||||
|
stream = kwargs.get("stream")
|
||||||
|
hooks = request.hooks
|
||||||
|
|
||||||
|
# Get the appropriate adapter to use
|
||||||
|
adapter = self.get_adapter(url=request.url)
|
||||||
|
|
||||||
|
# Start time (approximately) of the request
|
||||||
|
start = preferred_clock()
|
||||||
|
|
||||||
|
# Send the request
|
||||||
|
r = adapter.send(request, **kwargs)
|
||||||
|
|
||||||
|
# Total elapsed time of the request (approximately)
|
||||||
|
elapsed = preferred_clock() - start
|
||||||
|
r.elapsed = timedelta(seconds=elapsed)
|
||||||
|
|
||||||
|
# Response manipulation hooks
|
||||||
|
r = dispatch_hook("response", hooks, r, **kwargs)
|
||||||
|
|
||||||
|
# Persist cookies
|
||||||
|
if r.history:
|
||||||
|
# If the hooks create history then we want those cookies too
|
||||||
|
for resp in r.history:
|
||||||
|
extract_cookies_to_jar(self.cookies, resp.request, resp.raw)
|
||||||
|
|
||||||
|
extract_cookies_to_jar(self.cookies, request, r.raw)
|
||||||
|
|
||||||
|
# Resolve redirects if allowed.
|
||||||
|
if allow_redirects:
|
||||||
|
# Redirect resolving generator.
|
||||||
|
gen = self.resolve_redirects(r, request, **kwargs)
|
||||||
|
history = [resp for resp in gen]
|
||||||
|
else:
|
||||||
|
history = []
|
||||||
|
|
||||||
|
# Shuffle things around if there's history.
|
||||||
|
if history:
|
||||||
|
# Insert the first (original) request at the start
|
||||||
|
history.insert(0, r)
|
||||||
|
# Get the last request made
|
||||||
|
r = history.pop()
|
||||||
|
r.history = history
|
||||||
|
|
||||||
|
# If redirects aren't being followed, store the response on the Request for Response.next().
|
||||||
|
if not allow_redirects:
|
||||||
|
try:
|
||||||
|
r._next = next( # type: ignore[assignment] # yield_requests=True returns PreparedRequest
|
||||||
|
self.resolve_redirects(r, request, yield_requests=True, **kwargs)
|
||||||
|
)
|
||||||
|
except StopIteration:
|
||||||
|
pass
|
||||||
|
|
||||||
|
if not stream:
|
||||||
|
r.content
|
||||||
|
|
||||||
|
return r
|
||||||
|
|
||||||
|
def merge_environment_settings(
|
||||||
|
self,
|
||||||
|
url: str,
|
||||||
|
proxies: dict[str, str] | None,
|
||||||
|
stream: bool | None,
|
||||||
|
verify: _t.VerifyType | None,
|
||||||
|
cert: _t.CertType,
|
||||||
|
) -> dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Check the environment and merge it with some settings.
|
||||||
|
|
||||||
|
:rtype: dict
|
||||||
|
"""
|
||||||
|
# Gather clues from the surrounding environment.
|
||||||
|
if self.trust_env:
|
||||||
|
# Set environment's proxies.
|
||||||
|
no_proxy = proxies.get("no_proxy") if proxies is not None else None
|
||||||
|
env_proxies = get_environ_proxies(url, no_proxy=no_proxy)
|
||||||
|
if proxies is not None:
|
||||||
|
for k, v in env_proxies.items():
|
||||||
|
proxies.setdefault(k, v)
|
||||||
|
|
||||||
|
# Look for requests environment configuration
|
||||||
|
# and be compatible with cURL.
|
||||||
|
if verify is True or verify is None:
|
||||||
|
verify = (
|
||||||
|
os.environ.get("REQUESTS_CA_BUNDLE")
|
||||||
|
or os.environ.get("CURL_CA_BUNDLE")
|
||||||
|
or verify
|
||||||
|
)
|
||||||
|
|
||||||
|
# Merge all the kwargs.
|
||||||
|
proxies = merge_setting(proxies, self.proxies)
|
||||||
|
stream = merge_setting(stream, self.stream)
|
||||||
|
verify = merge_setting(verify, self.verify)
|
||||||
|
cert = merge_setting(cert, self.cert)
|
||||||
|
|
||||||
|
return {"proxies": proxies, "stream": stream, "verify": verify, "cert": cert}
|
||||||
|
|
||||||
|
def get_adapter(self, url: str) -> BaseAdapter:
|
||||||
|
"""
|
||||||
|
Returns the appropriate connection adapter for the given URL.
|
||||||
|
|
||||||
|
:rtype: requests.adapters.BaseAdapter
|
||||||
|
"""
|
||||||
|
for prefix, adapter in self.adapters.items():
|
||||||
|
if url.lower().startswith(prefix.lower()):
|
||||||
|
return adapter
|
||||||
|
|
||||||
|
# Nothing matches :-/
|
||||||
|
raise InvalidSchema(f"No connection adapters were found for {url!r}")
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
"""Closes all adapters and as such the session"""
|
||||||
|
for v in self.adapters.values():
|
||||||
|
v.close()
|
||||||
|
|
||||||
|
def mount(self, prefix: str, adapter: BaseAdapter) -> None:
|
||||||
|
"""Registers a connection adapter to a prefix.
|
||||||
|
|
||||||
|
Adapters are sorted in descending order by prefix length.
|
||||||
|
"""
|
||||||
|
self.adapters[prefix] = adapter
|
||||||
|
keys_to_move = [k for k in self.adapters if len(k) < len(prefix)]
|
||||||
|
|
||||||
|
for key in keys_to_move:
|
||||||
|
self.adapters[key] = self.adapters.pop(key)
|
||||||
|
|
||||||
|
def __getstate__(self) -> dict[str, Any]:
|
||||||
|
state = {attr: getattr(self, attr, None) for attr in self.__attrs__}
|
||||||
|
return state
|
||||||
|
|
||||||
|
def __setstate__(self, state: dict[str, Any]) -> None:
|
||||||
|
for attr, value in state.items():
|
||||||
|
setattr(self, attr, value)
|
||||||
|
|
||||||
|
|
||||||
|
def session() -> Session:
|
||||||
|
"""
|
||||||
|
Returns a :class:`Session` for context-management.
|
||||||
|
|
||||||
|
.. deprecated:: 1.0.0
|
||||||
|
|
||||||
|
This method has been deprecated since version 1.0.0 and is only kept for
|
||||||
|
backwards compatibility. New code should use :class:`~requests.sessions.Session`
|
||||||
|
to create a session. This may be removed at a future date.
|
||||||
|
|
||||||
|
:rtype: Session
|
||||||
|
"""
|
||||||
|
return Session()
|
||||||
+128
@@ -0,0 +1,128 @@
|
|||||||
|
r"""
|
||||||
|
The ``codes`` object defines a mapping from common names for HTTP statuses
|
||||||
|
to their numerical codes, accessible either as attributes or as dictionary
|
||||||
|
items.
|
||||||
|
|
||||||
|
Example::
|
||||||
|
|
||||||
|
>>> import requests
|
||||||
|
>>> requests.codes['temporary_redirect']
|
||||||
|
307
|
||||||
|
>>> requests.codes.teapot
|
||||||
|
418
|
||||||
|
>>> requests.codes['\o/']
|
||||||
|
200
|
||||||
|
|
||||||
|
Some codes have multiple names, and both upper- and lower-case versions of
|
||||||
|
the names are allowed. For example, ``codes.ok``, ``codes.OK``, and
|
||||||
|
``codes.okay`` all correspond to the HTTP status code 200.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from .structures import LookupDict
|
||||||
|
|
||||||
|
_codes = {
|
||||||
|
# Informational.
|
||||||
|
100: ("continue",),
|
||||||
|
101: ("switching_protocols",),
|
||||||
|
102: ("processing", "early-hints"),
|
||||||
|
103: ("checkpoint",),
|
||||||
|
122: ("uri_too_long", "request_uri_too_long"),
|
||||||
|
200: ("ok", "okay", "all_ok", "all_okay", "all_good", "\\o/", "✓"),
|
||||||
|
201: ("created",),
|
||||||
|
202: ("accepted",),
|
||||||
|
203: ("non_authoritative_info", "non_authoritative_information"),
|
||||||
|
204: ("no_content",),
|
||||||
|
205: ("reset_content", "reset"),
|
||||||
|
206: ("partial_content", "partial"),
|
||||||
|
207: ("multi_status", "multiple_status", "multi_stati", "multiple_stati"),
|
||||||
|
208: ("already_reported",),
|
||||||
|
226: ("im_used",),
|
||||||
|
# Redirection.
|
||||||
|
300: ("multiple_choices",),
|
||||||
|
301: ("moved_permanently", "moved", "\\o-"),
|
||||||
|
302: ("found",),
|
||||||
|
303: ("see_other", "other"),
|
||||||
|
304: ("not_modified",),
|
||||||
|
305: ("use_proxy",),
|
||||||
|
306: ("switch_proxy",),
|
||||||
|
307: ("temporary_redirect", "temporary_moved", "temporary"),
|
||||||
|
308: (
|
||||||
|
"permanent_redirect",
|
||||||
|
"resume_incomplete",
|
||||||
|
"resume",
|
||||||
|
), # "resume" and "resume_incomplete" to be removed in 3.0
|
||||||
|
# Client Error.
|
||||||
|
400: ("bad_request", "bad"),
|
||||||
|
401: ("unauthorized",),
|
||||||
|
402: ("payment_required", "payment"),
|
||||||
|
403: ("forbidden",),
|
||||||
|
404: ("not_found", "-o-"),
|
||||||
|
405: ("method_not_allowed", "not_allowed"),
|
||||||
|
406: ("not_acceptable",),
|
||||||
|
407: ("proxy_authentication_required", "proxy_auth", "proxy_authentication"),
|
||||||
|
408: ("request_timeout", "timeout"),
|
||||||
|
409: ("conflict",),
|
||||||
|
410: ("gone",),
|
||||||
|
411: ("length_required",),
|
||||||
|
412: ("precondition_failed", "precondition"),
|
||||||
|
413: ("request_entity_too_large", "content_too_large"),
|
||||||
|
414: ("request_uri_too_large", "uri_too_long"),
|
||||||
|
415: ("unsupported_media_type", "unsupported_media", "media_type"),
|
||||||
|
416: (
|
||||||
|
"requested_range_not_satisfiable",
|
||||||
|
"requested_range",
|
||||||
|
"range_not_satisfiable",
|
||||||
|
),
|
||||||
|
417: ("expectation_failed",),
|
||||||
|
418: ("im_a_teapot", "teapot", "i_am_a_teapot"),
|
||||||
|
421: ("misdirected_request",),
|
||||||
|
422: ("unprocessable_entity", "unprocessable", "unprocessable_content"),
|
||||||
|
423: ("locked",),
|
||||||
|
424: ("failed_dependency", "dependency"),
|
||||||
|
425: ("unordered_collection", "unordered", "too_early"),
|
||||||
|
426: ("upgrade_required", "upgrade"),
|
||||||
|
428: ("precondition_required", "precondition"),
|
||||||
|
429: ("too_many_requests", "too_many"),
|
||||||
|
431: ("header_fields_too_large", "fields_too_large"),
|
||||||
|
444: ("no_response", "none"),
|
||||||
|
449: ("retry_with", "retry"),
|
||||||
|
450: ("blocked_by_windows_parental_controls", "parental_controls"),
|
||||||
|
451: ("unavailable_for_legal_reasons", "legal_reasons"),
|
||||||
|
499: ("client_closed_request",),
|
||||||
|
# Server Error.
|
||||||
|
500: ("internal_server_error", "server_error", "/o\\", "✗"),
|
||||||
|
501: ("not_implemented",),
|
||||||
|
502: ("bad_gateway",),
|
||||||
|
503: ("service_unavailable", "unavailable"),
|
||||||
|
504: ("gateway_timeout",),
|
||||||
|
505: ("http_version_not_supported", "http_version"),
|
||||||
|
506: ("variant_also_negotiates",),
|
||||||
|
507: ("insufficient_storage",),
|
||||||
|
509: ("bandwidth_limit_exceeded", "bandwidth"),
|
||||||
|
510: ("not_extended",),
|
||||||
|
511: ("network_authentication_required", "network_auth", "network_authentication"),
|
||||||
|
}
|
||||||
|
|
||||||
|
codes: LookupDict[int] = LookupDict(name="status_codes")
|
||||||
|
|
||||||
|
|
||||||
|
def _init():
|
||||||
|
for code, titles in _codes.items():
|
||||||
|
for title in titles:
|
||||||
|
setattr(codes, title, code)
|
||||||
|
if not title.startswith(("\\", "/")):
|
||||||
|
setattr(codes, title.upper(), code)
|
||||||
|
|
||||||
|
def doc(code: int) -> str:
|
||||||
|
names = ", ".join(f"``{n}``" for n in _codes[code])
|
||||||
|
return "* %d: %s" % (code, names)
|
||||||
|
|
||||||
|
global __doc__
|
||||||
|
__doc__ = (
|
||||||
|
__doc__ + "\n" + "\n".join(doc(code) for code in sorted(_codes))
|
||||||
|
if __doc__ is not None
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
_init()
|
||||||
+130
@@ -0,0 +1,130 @@
|
|||||||
|
"""
|
||||||
|
requests.structures
|
||||||
|
~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
Data structures that power Requests.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from collections import OrderedDict
|
||||||
|
from collections.abc import Iterable, Iterator, Mapping
|
||||||
|
from typing import Any, Generic, TypeVar, overload
|
||||||
|
|
||||||
|
from .compat import MutableMapping
|
||||||
|
|
||||||
|
_VT = TypeVar("_VT")
|
||||||
|
_D = TypeVar("_D")
|
||||||
|
|
||||||
|
|
||||||
|
class CaseInsensitiveDict(MutableMapping[str, _VT], Generic[_VT]):
|
||||||
|
"""A case-insensitive ``dict``-like object.
|
||||||
|
|
||||||
|
Implements all methods and operations of
|
||||||
|
``MutableMapping`` as well as dict's ``copy``. Also
|
||||||
|
provides ``lower_items``.
|
||||||
|
|
||||||
|
All keys are expected to be strings. The structure remembers the
|
||||||
|
case of the last key to be set, and ``iter(instance)``,
|
||||||
|
``keys()``, ``items()``, ``iterkeys()``, and ``iteritems()``
|
||||||
|
will contain case-sensitive keys. However, querying and contains
|
||||||
|
testing is case insensitive::
|
||||||
|
|
||||||
|
cid = CaseInsensitiveDict()
|
||||||
|
cid['Accept'] = 'application/json'
|
||||||
|
cid['aCCEPT'] == 'application/json' # True
|
||||||
|
list(cid) == ['Accept'] # True
|
||||||
|
|
||||||
|
For example, ``headers['content-encoding']`` will return the
|
||||||
|
value of a ``'Content-Encoding'`` response header, regardless
|
||||||
|
of how the header name was originally stored.
|
||||||
|
|
||||||
|
If the constructor, ``.update``, or equality comparison
|
||||||
|
operations are given keys that have equal ``.lower()``s, the
|
||||||
|
behavior is undefined.
|
||||||
|
"""
|
||||||
|
|
||||||
|
_store: OrderedDict[str, tuple[str, _VT]]
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
data: Mapping[str, _VT] | Iterable[tuple[str, _VT]] | None = None,
|
||||||
|
**kwargs: _VT,
|
||||||
|
) -> None:
|
||||||
|
self._store = OrderedDict()
|
||||||
|
if data is None:
|
||||||
|
data = {}
|
||||||
|
self.update(data, **kwargs)
|
||||||
|
|
||||||
|
def __setitem__(self, key: str, value: _VT) -> None:
|
||||||
|
# Use the lowercased key for lookups, but store the actual
|
||||||
|
# key alongside the value.
|
||||||
|
self._store[key.lower()] = (key, value)
|
||||||
|
|
||||||
|
def __getitem__(self, key: str) -> _VT:
|
||||||
|
return self._store[key.lower()][1]
|
||||||
|
|
||||||
|
def __delitem__(self, key: str) -> None:
|
||||||
|
del self._store[key.lower()]
|
||||||
|
|
||||||
|
def __iter__(self) -> Iterator[str]:
|
||||||
|
return (casedkey for casedkey, _ in self._store.values())
|
||||||
|
|
||||||
|
def __len__(self) -> int:
|
||||||
|
return len(self._store)
|
||||||
|
|
||||||
|
def lower_items(self) -> Iterator[tuple[str, _VT]]:
|
||||||
|
"""Like iteritems(), but with all lowercase keys."""
|
||||||
|
return ((lowerkey, keyval[1]) for (lowerkey, keyval) in self._store.items())
|
||||||
|
|
||||||
|
def __eq__(self, other: object) -> bool:
|
||||||
|
if isinstance(other, Mapping):
|
||||||
|
other_dict: CaseInsensitiveDict[Any] = CaseInsensitiveDict(other) # type: ignore[reportUnknownArgumentType]
|
||||||
|
else:
|
||||||
|
return NotImplemented
|
||||||
|
# Compare insensitively
|
||||||
|
return dict(self.lower_items()) == dict(other_dict.lower_items())
|
||||||
|
|
||||||
|
# Copy is required
|
||||||
|
def copy(self) -> CaseInsensitiveDict[_VT]:
|
||||||
|
return CaseInsensitiveDict(self._store.values())
|
||||||
|
|
||||||
|
def __repr__(self) -> str:
|
||||||
|
return str(dict(self.items()))
|
||||||
|
|
||||||
|
|
||||||
|
class LookupDict(dict[str, _VT]):
|
||||||
|
"""Dictionary lookup object."""
|
||||||
|
|
||||||
|
name: Any
|
||||||
|
|
||||||
|
def __init__(self, name: Any = None) -> None:
|
||||||
|
self.name = name
|
||||||
|
super().__init__()
|
||||||
|
|
||||||
|
def __repr__(self) -> str:
|
||||||
|
return f"<lookup '{self.name}'>"
|
||||||
|
|
||||||
|
def __getattr__(self, key: str) -> _VT | None:
|
||||||
|
# We need this for type checkers to infer typing
|
||||||
|
# on attribute access with status_codes.py
|
||||||
|
if key in self.__dict__:
|
||||||
|
return self.__dict__[key]
|
||||||
|
else:
|
||||||
|
raise AttributeError(
|
||||||
|
f"'{type(self).__name__}' object has no attribute '{key}'"
|
||||||
|
)
|
||||||
|
|
||||||
|
def __getitem__(self, key: str) -> _VT | None: # type: ignore[override]
|
||||||
|
# We allow fall-through here, so values default to None
|
||||||
|
|
||||||
|
return self.__dict__.get(key, None)
|
||||||
|
|
||||||
|
@overload
|
||||||
|
def get(self, key: str, default: None = None) -> _VT | None: ...
|
||||||
|
|
||||||
|
@overload
|
||||||
|
def get(self, key: str, default: _D | _VT) -> _D | _VT: ...
|
||||||
|
|
||||||
|
def get(self, key: str, default: _D | None = None) -> _VT | _D | None:
|
||||||
|
return self.__dict__.get(key, default)
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,211 @@
|
|||||||
|
"""
|
||||||
|
Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
# Set default logging handler to avoid "No handler found" warnings.
|
||||||
|
import logging
|
||||||
|
import sys
|
||||||
|
import typing
|
||||||
|
import warnings
|
||||||
|
from logging import NullHandler
|
||||||
|
|
||||||
|
from . import exceptions
|
||||||
|
from ._base_connection import _TYPE_BODY
|
||||||
|
from ._collections import HTTPHeaderDict
|
||||||
|
from ._version import __version__
|
||||||
|
from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
|
||||||
|
from .filepost import _TYPE_FIELDS, encode_multipart_formdata
|
||||||
|
from .poolmanager import PoolManager, ProxyManager, proxy_from_url
|
||||||
|
from .response import BaseHTTPResponse, HTTPResponse
|
||||||
|
from .util.request import make_headers
|
||||||
|
from .util.retry import Retry
|
||||||
|
from .util.timeout import Timeout
|
||||||
|
|
||||||
|
# Ensure that Python is compiled with OpenSSL 1.1.1+
|
||||||
|
# If the 'ssl' module isn't available at all that's
|
||||||
|
# fine, we only care if the module is available.
|
||||||
|
try:
|
||||||
|
import ssl
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
else:
|
||||||
|
if not ssl.OPENSSL_VERSION.startswith("OpenSSL "): # Defensive:
|
||||||
|
warnings.warn(
|
||||||
|
"urllib3 v2 only supports OpenSSL 1.1.1+, currently "
|
||||||
|
f"the 'ssl' module is compiled with {ssl.OPENSSL_VERSION!r}. "
|
||||||
|
"See: https://github.com/urllib3/urllib3/issues/3020",
|
||||||
|
exceptions.NotOpenSSLWarning,
|
||||||
|
)
|
||||||
|
elif ssl.OPENSSL_VERSION_INFO < (1, 1, 1): # Defensive:
|
||||||
|
raise ImportError(
|
||||||
|
"urllib3 v2 only supports OpenSSL 1.1.1+, currently "
|
||||||
|
f"the 'ssl' module is compiled with {ssl.OPENSSL_VERSION!r}. "
|
||||||
|
"See: https://github.com/urllib3/urllib3/issues/2168"
|
||||||
|
)
|
||||||
|
|
||||||
|
__author__ = "Andrey Petrov ([email protected])"
|
||||||
|
__license__ = "MIT"
|
||||||
|
__version__ = __version__
|
||||||
|
|
||||||
|
__all__ = (
|
||||||
|
"HTTPConnectionPool",
|
||||||
|
"HTTPHeaderDict",
|
||||||
|
"HTTPSConnectionPool",
|
||||||
|
"PoolManager",
|
||||||
|
"ProxyManager",
|
||||||
|
"HTTPResponse",
|
||||||
|
"Retry",
|
||||||
|
"Timeout",
|
||||||
|
"add_stderr_logger",
|
||||||
|
"connection_from_url",
|
||||||
|
"disable_warnings",
|
||||||
|
"encode_multipart_formdata",
|
||||||
|
"make_headers",
|
||||||
|
"proxy_from_url",
|
||||||
|
"request",
|
||||||
|
"BaseHTTPResponse",
|
||||||
|
)
|
||||||
|
|
||||||
|
logging.getLogger(__name__).addHandler(NullHandler())
|
||||||
|
|
||||||
|
|
||||||
|
def add_stderr_logger(
|
||||||
|
level: int = logging.DEBUG,
|
||||||
|
) -> logging.StreamHandler[typing.TextIO]:
|
||||||
|
"""
|
||||||
|
Helper for quickly adding a StreamHandler to the logger. Useful for
|
||||||
|
debugging.
|
||||||
|
|
||||||
|
Returns the handler after adding it.
|
||||||
|
"""
|
||||||
|
# This method needs to be in this __init__.py to get the __name__ correct
|
||||||
|
# even if urllib3 is vendored within another package.
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
handler = logging.StreamHandler()
|
||||||
|
handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(message)s"))
|
||||||
|
logger.addHandler(handler)
|
||||||
|
logger.setLevel(level)
|
||||||
|
logger.debug("Added a stderr logging handler to logger: %s", __name__)
|
||||||
|
return handler
|
||||||
|
|
||||||
|
|
||||||
|
# ... Clean up.
|
||||||
|
del NullHandler
|
||||||
|
|
||||||
|
|
||||||
|
# All warning filters *must* be appended unless you're really certain that they
|
||||||
|
# shouldn't be: otherwise, it's very hard for users to use most Python
|
||||||
|
# mechanisms to silence them.
|
||||||
|
# SecurityWarning's always go off by default.
|
||||||
|
warnings.simplefilter("always", exceptions.SecurityWarning, append=True)
|
||||||
|
# InsecurePlatformWarning's don't vary between requests, so we keep it default.
|
||||||
|
warnings.simplefilter("default", exceptions.InsecurePlatformWarning, append=True)
|
||||||
|
|
||||||
|
|
||||||
|
def disable_warnings(category: type[Warning] = exceptions.HTTPWarning) -> None:
|
||||||
|
"""
|
||||||
|
Helper for quickly disabling all urllib3 warnings.
|
||||||
|
"""
|
||||||
|
warnings.simplefilter("ignore", category)
|
||||||
|
|
||||||
|
|
||||||
|
_DEFAULT_POOL = PoolManager()
|
||||||
|
|
||||||
|
|
||||||
|
def request(
|
||||||
|
method: str,
|
||||||
|
url: str,
|
||||||
|
*,
|
||||||
|
body: _TYPE_BODY | None = None,
|
||||||
|
fields: _TYPE_FIELDS | None = None,
|
||||||
|
headers: typing.Mapping[str, str] | None = None,
|
||||||
|
preload_content: bool | None = True,
|
||||||
|
decode_content: bool | None = True,
|
||||||
|
redirect: bool | None = True,
|
||||||
|
retries: Retry | bool | int | None = None,
|
||||||
|
timeout: Timeout | float | int | None = 3,
|
||||||
|
json: typing.Any | None = None,
|
||||||
|
) -> BaseHTTPResponse:
|
||||||
|
"""
|
||||||
|
A convenience, top-level request method. It uses a module-global ``PoolManager`` instance.
|
||||||
|
Therefore, its side effects could be shared across dependencies relying on it.
|
||||||
|
To avoid side effects create a new ``PoolManager`` instance and use it instead.
|
||||||
|
The method does not accept low-level ``**urlopen_kw`` keyword arguments.
|
||||||
|
|
||||||
|
:param method:
|
||||||
|
HTTP request method (such as GET, POST, PUT, etc.)
|
||||||
|
|
||||||
|
:param url:
|
||||||
|
The URL to perform the request on.
|
||||||
|
|
||||||
|
:param body:
|
||||||
|
Data to send in the request body, either :class:`str`, :class:`bytes`,
|
||||||
|
an iterable of :class:`str`/:class:`bytes`, or a file-like object.
|
||||||
|
|
||||||
|
:param fields:
|
||||||
|
Data to encode and send in the request body.
|
||||||
|
|
||||||
|
:param headers:
|
||||||
|
Dictionary of custom headers to send, such as User-Agent,
|
||||||
|
If-None-Match, etc.
|
||||||
|
|
||||||
|
:param bool preload_content:
|
||||||
|
If True, the response's body will be preloaded into memory.
|
||||||
|
|
||||||
|
:param bool decode_content:
|
||||||
|
If True, will attempt to decode the body based on the
|
||||||
|
'content-encoding' header.
|
||||||
|
|
||||||
|
:param redirect:
|
||||||
|
If True, automatically handle redirects (status codes 301, 302,
|
||||||
|
303, 307, 308). Each redirect counts as a retry. Disabling retries
|
||||||
|
will disable redirect, too.
|
||||||
|
|
||||||
|
:param retries:
|
||||||
|
Configure the number of retries to allow before raising a
|
||||||
|
:class:`~urllib3.exceptions.MaxRetryError` exception.
|
||||||
|
|
||||||
|
If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a
|
||||||
|
:class:`~urllib3.util.retry.Retry` object for fine-grained control
|
||||||
|
over different types of retries.
|
||||||
|
Pass an integer number to retry connection errors that many times,
|
||||||
|
but no other types of errors. Pass zero to never retry.
|
||||||
|
|
||||||
|
If ``False``, then retries are disabled and any exception is raised
|
||||||
|
immediately. Also, instead of raising a MaxRetryError on redirects,
|
||||||
|
the redirect response will be returned.
|
||||||
|
|
||||||
|
:type retries: :class:`~urllib3.util.retry.Retry`, False, or an int.
|
||||||
|
|
||||||
|
:param timeout:
|
||||||
|
If specified, overrides the default timeout for this one
|
||||||
|
request. It may be a float (in seconds) or an instance of
|
||||||
|
:class:`urllib3.util.Timeout`.
|
||||||
|
|
||||||
|
:param json:
|
||||||
|
Data to encode and send as JSON with UTF-encoded in the request body.
|
||||||
|
The ``"Content-Type"`` header will be set to ``"application/json"``
|
||||||
|
unless specified otherwise.
|
||||||
|
"""
|
||||||
|
|
||||||
|
return _DEFAULT_POOL.request(
|
||||||
|
method,
|
||||||
|
url,
|
||||||
|
body=body,
|
||||||
|
fields=fields,
|
||||||
|
headers=headers,
|
||||||
|
preload_content=preload_content,
|
||||||
|
decode_content=decode_content,
|
||||||
|
redirect=redirect,
|
||||||
|
retries=retries,
|
||||||
|
timeout=timeout,
|
||||||
|
json=json,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
if sys.platform == "emscripten":
|
||||||
|
from .contrib.emscripten import inject_into_urllib3 # noqa: 401
|
||||||
|
|
||||||
|
inject_into_urllib3()
|
||||||
+167
@@ -0,0 +1,167 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import typing
|
||||||
|
|
||||||
|
from .util.connection import _TYPE_SOCKET_OPTIONS
|
||||||
|
from .util.timeout import _DEFAULT_TIMEOUT, _TYPE_TIMEOUT
|
||||||
|
from .util.url import Url
|
||||||
|
|
||||||
|
_TYPE_BODY = typing.Union[
|
||||||
|
bytes, typing.IO[typing.Any], typing.Iterable[bytes | str], str
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class ProxyConfig(typing.NamedTuple):
|
||||||
|
ssl_context: ssl.SSLContext | None
|
||||||
|
use_forwarding_for_https: bool
|
||||||
|
assert_hostname: None | str | typing.Literal[False]
|
||||||
|
assert_fingerprint: str | None
|
||||||
|
|
||||||
|
|
||||||
|
class _ResponseOptions(typing.NamedTuple):
|
||||||
|
# TODO: Remove this in favor of a better
|
||||||
|
# HTTP request/response lifecycle tracking.
|
||||||
|
request_method: str
|
||||||
|
request_url: str
|
||||||
|
preload_content: bool
|
||||||
|
decode_content: bool
|
||||||
|
enforce_content_length: bool
|
||||||
|
|
||||||
|
|
||||||
|
if typing.TYPE_CHECKING:
|
||||||
|
import ssl
|
||||||
|
from typing import Protocol
|
||||||
|
|
||||||
|
from .response import BaseHTTPResponse
|
||||||
|
|
||||||
|
class BaseHTTPConnection(Protocol):
|
||||||
|
default_port: typing.ClassVar[int]
|
||||||
|
default_socket_options: typing.ClassVar[_TYPE_SOCKET_OPTIONS]
|
||||||
|
|
||||||
|
host: str
|
||||||
|
port: int
|
||||||
|
timeout: None | (
|
||||||
|
float
|
||||||
|
) # Instance doesn't store _DEFAULT_TIMEOUT, must be resolved.
|
||||||
|
blocksize: int
|
||||||
|
source_address: tuple[str, int] | None
|
||||||
|
socket_options: _TYPE_SOCKET_OPTIONS | None
|
||||||
|
|
||||||
|
proxy: Url | None
|
||||||
|
proxy_config: ProxyConfig | None
|
||||||
|
|
||||||
|
is_verified: bool
|
||||||
|
proxy_is_verified: bool | None
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
host: str,
|
||||||
|
port: int | None = None,
|
||||||
|
*,
|
||||||
|
timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
|
||||||
|
source_address: tuple[str, int] | None = None,
|
||||||
|
blocksize: int = 8192,
|
||||||
|
socket_options: _TYPE_SOCKET_OPTIONS | None = ...,
|
||||||
|
proxy: Url | None = None,
|
||||||
|
proxy_config: ProxyConfig | None = None,
|
||||||
|
) -> None: ...
|
||||||
|
|
||||||
|
def set_tunnel(
|
||||||
|
self,
|
||||||
|
host: str,
|
||||||
|
port: int | None = None,
|
||||||
|
headers: typing.Mapping[str, str] | None = None,
|
||||||
|
scheme: str = "http",
|
||||||
|
) -> None: ...
|
||||||
|
|
||||||
|
def connect(self) -> None: ...
|
||||||
|
|
||||||
|
def request(
|
||||||
|
self,
|
||||||
|
method: str,
|
||||||
|
url: str,
|
||||||
|
body: _TYPE_BODY | None = None,
|
||||||
|
headers: typing.Mapping[str, str] | None = None,
|
||||||
|
# We know *at least* botocore is depending on the order of the
|
||||||
|
# first 3 parameters so to be safe we only mark the later ones
|
||||||
|
# as keyword-only to ensure we have space to extend.
|
||||||
|
*,
|
||||||
|
chunked: bool = False,
|
||||||
|
preload_content: bool = True,
|
||||||
|
decode_content: bool = True,
|
||||||
|
enforce_content_length: bool = True,
|
||||||
|
) -> None: ...
|
||||||
|
|
||||||
|
def getresponse(self) -> BaseHTTPResponse: ...
|
||||||
|
|
||||||
|
def close(self) -> None: ...
|
||||||
|
|
||||||
|
@property
|
||||||
|
def is_closed(self) -> bool:
|
||||||
|
"""Whether the connection either is brand new or has been previously closed.
|
||||||
|
If this property is True then both ``is_connected`` and ``has_connected_to_proxy``
|
||||||
|
properties must be False.
|
||||||
|
"""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def is_connected(self) -> bool:
|
||||||
|
"""Whether the connection is actively connected to any origin (proxy or target)"""
|
||||||
|
|
||||||
|
@property
|
||||||
|
def has_connected_to_proxy(self) -> bool:
|
||||||
|
"""Whether the connection has successfully connected to its proxy.
|
||||||
|
This returns False if no proxy is in use. Used to determine whether
|
||||||
|
errors are coming from the proxy layer or from tunnelling to the target origin.
|
||||||
|
"""
|
||||||
|
|
||||||
|
class BaseHTTPSConnection(BaseHTTPConnection, Protocol):
|
||||||
|
default_port: typing.ClassVar[int]
|
||||||
|
default_socket_options: typing.ClassVar[_TYPE_SOCKET_OPTIONS]
|
||||||
|
|
||||||
|
# Certificate verification methods
|
||||||
|
cert_reqs: int | str | None
|
||||||
|
assert_hostname: None | str | typing.Literal[False]
|
||||||
|
assert_fingerprint: str | None
|
||||||
|
ssl_context: ssl.SSLContext | None
|
||||||
|
|
||||||
|
# Trusted CAs
|
||||||
|
ca_certs: str | None
|
||||||
|
ca_cert_dir: str | None
|
||||||
|
ca_cert_data: None | str | bytes
|
||||||
|
|
||||||
|
# TLS version
|
||||||
|
ssl_minimum_version: int | None
|
||||||
|
ssl_maximum_version: int | None
|
||||||
|
ssl_version: int | str | None # Deprecated
|
||||||
|
|
||||||
|
# Client certificates
|
||||||
|
cert_file: str | None
|
||||||
|
key_file: str | None
|
||||||
|
key_password: str | None
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
host: str,
|
||||||
|
port: int | None = None,
|
||||||
|
*,
|
||||||
|
timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
|
||||||
|
source_address: tuple[str, int] | None = None,
|
||||||
|
blocksize: int = 16384,
|
||||||
|
socket_options: _TYPE_SOCKET_OPTIONS | None = ...,
|
||||||
|
proxy: Url | None = None,
|
||||||
|
proxy_config: ProxyConfig | None = None,
|
||||||
|
cert_reqs: int | str | None = None,
|
||||||
|
assert_hostname: None | str | typing.Literal[False] = None,
|
||||||
|
assert_fingerprint: str | None = None,
|
||||||
|
server_hostname: str | None = None,
|
||||||
|
ssl_context: ssl.SSLContext | None = None,
|
||||||
|
ca_certs: str | None = None,
|
||||||
|
ca_cert_dir: str | None = None,
|
||||||
|
ca_cert_data: None | str | bytes = None,
|
||||||
|
ssl_minimum_version: int | None = None,
|
||||||
|
ssl_maximum_version: int | None = None,
|
||||||
|
ssl_version: int | str | None = None, # Deprecated
|
||||||
|
cert_file: str | None = None,
|
||||||
|
key_file: str | None = None,
|
||||||
|
key_password: str | None = None,
|
||||||
|
) -> None: ...
|
||||||
+486
@@ -0,0 +1,486 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import typing
|
||||||
|
from collections import OrderedDict
|
||||||
|
from enum import Enum, auto
|
||||||
|
from threading import RLock
|
||||||
|
|
||||||
|
if typing.TYPE_CHECKING:
|
||||||
|
# We can only import Protocol if TYPE_CHECKING because it's a development
|
||||||
|
# dependency, and is not available at runtime.
|
||||||
|
from typing import Protocol
|
||||||
|
|
||||||
|
from typing_extensions import Self
|
||||||
|
|
||||||
|
class HasGettableStringKeys(Protocol):
|
||||||
|
def keys(self) -> typing.Iterator[str]: ...
|
||||||
|
|
||||||
|
def __getitem__(self, key: str) -> str: ...
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["RecentlyUsedContainer", "HTTPHeaderDict"]
|
||||||
|
|
||||||
|
|
||||||
|
# Key type
|
||||||
|
_KT = typing.TypeVar("_KT")
|
||||||
|
# Value type
|
||||||
|
_VT = typing.TypeVar("_VT")
|
||||||
|
# Default type
|
||||||
|
_DT = typing.TypeVar("_DT")
|
||||||
|
|
||||||
|
ValidHTTPHeaderSource = typing.Union[
|
||||||
|
"HTTPHeaderDict",
|
||||||
|
typing.Mapping[str, str],
|
||||||
|
typing.Iterable[tuple[str, str]],
|
||||||
|
"HasGettableStringKeys",
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class _Sentinel(Enum):
|
||||||
|
not_passed = auto()
|
||||||
|
|
||||||
|
|
||||||
|
def ensure_can_construct_http_header_dict(
|
||||||
|
potential: object,
|
||||||
|
) -> ValidHTTPHeaderSource | None:
|
||||||
|
if isinstance(potential, HTTPHeaderDict):
|
||||||
|
return potential
|
||||||
|
elif isinstance(potential, typing.Mapping):
|
||||||
|
# Full runtime checking of the contents of a Mapping is expensive, so for the
|
||||||
|
# purposes of typechecking, we assume that any Mapping is the right shape.
|
||||||
|
return typing.cast(typing.Mapping[str, str], potential)
|
||||||
|
elif isinstance(potential, typing.Iterable):
|
||||||
|
# Similarly to Mapping, full runtime checking of the contents of an Iterable is
|
||||||
|
# expensive, so for the purposes of typechecking, we assume that any Iterable
|
||||||
|
# is the right shape.
|
||||||
|
return typing.cast(typing.Iterable[tuple[str, str]], potential)
|
||||||
|
elif hasattr(potential, "keys") and hasattr(potential, "__getitem__"):
|
||||||
|
return typing.cast("HasGettableStringKeys", potential)
|
||||||
|
else:
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
class RecentlyUsedContainer(typing.Generic[_KT, _VT], typing.MutableMapping[_KT, _VT]):
|
||||||
|
"""
|
||||||
|
Provides a thread-safe dict-like container which maintains up to
|
||||||
|
``maxsize`` keys while throwing away the least-recently-used keys beyond
|
||||||
|
``maxsize``.
|
||||||
|
|
||||||
|
:param maxsize:
|
||||||
|
Maximum number of recent elements to retain.
|
||||||
|
|
||||||
|
:param dispose_func:
|
||||||
|
Every time an item is evicted from the container,
|
||||||
|
``dispose_func(value)`` is called. Callback which will get called
|
||||||
|
"""
|
||||||
|
|
||||||
|
_container: typing.OrderedDict[_KT, _VT]
|
||||||
|
_maxsize: int
|
||||||
|
dispose_func: typing.Callable[[_VT], None] | None
|
||||||
|
lock: RLock
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
maxsize: int = 10,
|
||||||
|
dispose_func: typing.Callable[[_VT], None] | None = None,
|
||||||
|
) -> None:
|
||||||
|
super().__init__()
|
||||||
|
self._maxsize = maxsize
|
||||||
|
self.dispose_func = dispose_func
|
||||||
|
self._container = OrderedDict()
|
||||||
|
self.lock = RLock()
|
||||||
|
|
||||||
|
def __getitem__(self, key: _KT) -> _VT:
|
||||||
|
# Re-insert the item, moving it to the end of the eviction line.
|
||||||
|
with self.lock:
|
||||||
|
item = self._container.pop(key)
|
||||||
|
self._container[key] = item
|
||||||
|
return item
|
||||||
|
|
||||||
|
def __setitem__(self, key: _KT, value: _VT) -> None:
|
||||||
|
evicted_item = None
|
||||||
|
with self.lock:
|
||||||
|
# Possibly evict the existing value of 'key'
|
||||||
|
try:
|
||||||
|
# If the key exists, we'll overwrite it, which won't change the
|
||||||
|
# size of the pool. Because accessing a key should move it to
|
||||||
|
# the end of the eviction line, we pop it out first.
|
||||||
|
evicted_item = key, self._container.pop(key)
|
||||||
|
self._container[key] = value
|
||||||
|
except KeyError:
|
||||||
|
# When the key does not exist, we insert the value first so that
|
||||||
|
# evicting works in all cases, including when self._maxsize is 0
|
||||||
|
self._container[key] = value
|
||||||
|
if len(self._container) > self._maxsize:
|
||||||
|
# If we didn't evict an existing value, and we've hit our maximum
|
||||||
|
# size, then we have to evict the least recently used item from
|
||||||
|
# the beginning of the container.
|
||||||
|
evicted_item = self._container.popitem(last=False)
|
||||||
|
|
||||||
|
# After releasing the lock on the pool, dispose of any evicted value.
|
||||||
|
if evicted_item is not None and self.dispose_func:
|
||||||
|
_, evicted_value = evicted_item
|
||||||
|
self.dispose_func(evicted_value)
|
||||||
|
|
||||||
|
def __delitem__(self, key: _KT) -> None:
|
||||||
|
with self.lock:
|
||||||
|
value = self._container.pop(key)
|
||||||
|
|
||||||
|
if self.dispose_func:
|
||||||
|
self.dispose_func(value)
|
||||||
|
|
||||||
|
def __len__(self) -> int:
|
||||||
|
with self.lock:
|
||||||
|
return len(self._container)
|
||||||
|
|
||||||
|
def __iter__(self) -> typing.NoReturn:
|
||||||
|
raise NotImplementedError(
|
||||||
|
"Iteration over this class is unlikely to be threadsafe."
|
||||||
|
)
|
||||||
|
|
||||||
|
def clear(self) -> None:
|
||||||
|
with self.lock:
|
||||||
|
# Copy pointers to all values, then wipe the mapping
|
||||||
|
values = list(self._container.values())
|
||||||
|
self._container.clear()
|
||||||
|
|
||||||
|
if self.dispose_func:
|
||||||
|
for value in values:
|
||||||
|
self.dispose_func(value)
|
||||||
|
|
||||||
|
def keys(self) -> set[_KT]: # type: ignore[override]
|
||||||
|
with self.lock:
|
||||||
|
return set(self._container.keys())
|
||||||
|
|
||||||
|
|
||||||
|
class HTTPHeaderDictItemView(set[tuple[str, str]]):
|
||||||
|
"""
|
||||||
|
HTTPHeaderDict is unusual for a Mapping[str, str] in that it has two modes of
|
||||||
|
address.
|
||||||
|
|
||||||
|
If we directly try to get an item with a particular name, we will get a string
|
||||||
|
back that is the concatenated version of all the values:
|
||||||
|
|
||||||
|
>>> d['X-Header-Name']
|
||||||
|
'Value1, Value2, Value3'
|
||||||
|
|
||||||
|
However, if we iterate over an HTTPHeaderDict's items, we will optionally combine
|
||||||
|
these values based on whether combine=True was called when building up the dictionary
|
||||||
|
|
||||||
|
>>> d = HTTPHeaderDict({"A": "1", "B": "foo"})
|
||||||
|
>>> d.add("A", "2", combine=True)
|
||||||
|
>>> d.add("B", "bar")
|
||||||
|
>>> list(d.items())
|
||||||
|
[
|
||||||
|
('A', '1, 2'),
|
||||||
|
('B', 'foo'),
|
||||||
|
('B', 'bar'),
|
||||||
|
]
|
||||||
|
|
||||||
|
This class conforms to the interface required by the MutableMapping ABC while
|
||||||
|
also giving us the nonstandard iteration behavior we want; items with duplicate
|
||||||
|
keys, ordered by time of first insertion.
|
||||||
|
"""
|
||||||
|
|
||||||
|
_headers: HTTPHeaderDict
|
||||||
|
|
||||||
|
def __init__(self, headers: HTTPHeaderDict) -> None:
|
||||||
|
self._headers = headers
|
||||||
|
|
||||||
|
def __len__(self) -> int:
|
||||||
|
return len(list(self._headers.iteritems()))
|
||||||
|
|
||||||
|
def __iter__(self) -> typing.Iterator[tuple[str, str]]:
|
||||||
|
return self._headers.iteritems()
|
||||||
|
|
||||||
|
def __contains__(self, item: object) -> bool:
|
||||||
|
if isinstance(item, tuple) and len(item) == 2:
|
||||||
|
passed_key, passed_val = item
|
||||||
|
if isinstance(passed_key, str) and isinstance(passed_val, str):
|
||||||
|
return self._headers._has_value_for_header(passed_key, passed_val)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
class HTTPHeaderDict(typing.MutableMapping[str, str]):
|
||||||
|
"""
|
||||||
|
:param headers:
|
||||||
|
An iterable of field-value pairs. Must not contain multiple field names
|
||||||
|
when compared case-insensitively.
|
||||||
|
|
||||||
|
:param kwargs:
|
||||||
|
Additional field-value pairs to pass in to ``dict.update``.
|
||||||
|
|
||||||
|
A ``dict`` like container for storing HTTP Headers.
|
||||||
|
|
||||||
|
Field names are stored and compared case-insensitively in compliance with
|
||||||
|
RFC 7230. Iteration provides the first case-sensitive key seen for each
|
||||||
|
case-insensitive pair.
|
||||||
|
|
||||||
|
Using ``__setitem__`` syntax overwrites fields that compare equal
|
||||||
|
case-insensitively in order to maintain ``dict``'s api. For fields that
|
||||||
|
compare equal, instead create a new ``HTTPHeaderDict`` and use ``.add``
|
||||||
|
in a loop.
|
||||||
|
|
||||||
|
If multiple fields that are equal case-insensitively are passed to the
|
||||||
|
constructor or ``.update``, the behavior is undefined and some will be
|
||||||
|
lost.
|
||||||
|
|
||||||
|
>>> headers = HTTPHeaderDict()
|
||||||
|
>>> headers.add('Set-Cookie', 'foo=bar')
|
||||||
|
>>> headers.add('set-cookie', 'baz=quxx')
|
||||||
|
>>> headers['content-length'] = '7'
|
||||||
|
>>> headers['SET-cookie']
|
||||||
|
'foo=bar, baz=quxx'
|
||||||
|
>>> headers['Content-Length']
|
||||||
|
'7'
|
||||||
|
"""
|
||||||
|
|
||||||
|
_container: typing.MutableMapping[str, list[str]]
|
||||||
|
|
||||||
|
def __init__(self, headers: ValidHTTPHeaderSource | None = None, **kwargs: str):
|
||||||
|
super().__init__()
|
||||||
|
self._container = {} # 'dict' is insert-ordered
|
||||||
|
if headers is not None:
|
||||||
|
if isinstance(headers, HTTPHeaderDict):
|
||||||
|
self._copy_from(headers)
|
||||||
|
else:
|
||||||
|
self.extend(headers)
|
||||||
|
if kwargs:
|
||||||
|
self.extend(kwargs)
|
||||||
|
|
||||||
|
def __setitem__(self, key: str, val: str) -> None:
|
||||||
|
# avoid a bytes/str comparison by decoding before httplib
|
||||||
|
if isinstance(key, bytes):
|
||||||
|
key = key.decode("latin-1")
|
||||||
|
self._container[key.lower()] = [key, val]
|
||||||
|
|
||||||
|
def __getitem__(self, key: str) -> str:
|
||||||
|
if isinstance(key, bytes):
|
||||||
|
key = key.decode("latin-1")
|
||||||
|
val = self._container[key.lower()]
|
||||||
|
return ", ".join(val[1:])
|
||||||
|
|
||||||
|
def __delitem__(self, key: str) -> None:
|
||||||
|
if isinstance(key, bytes):
|
||||||
|
key = key.decode("latin-1")
|
||||||
|
del self._container[key.lower()]
|
||||||
|
|
||||||
|
def __contains__(self, key: object) -> bool:
|
||||||
|
if isinstance(key, bytes):
|
||||||
|
key = key.decode("latin-1")
|
||||||
|
if isinstance(key, str):
|
||||||
|
return key.lower() in self._container
|
||||||
|
return False
|
||||||
|
|
||||||
|
def setdefault(self, key: str, default: str = "") -> str:
|
||||||
|
return super().setdefault(key, default)
|
||||||
|
|
||||||
|
def __eq__(self, other: object) -> bool:
|
||||||
|
maybe_constructable = ensure_can_construct_http_header_dict(other)
|
||||||
|
if maybe_constructable is None:
|
||||||
|
return False
|
||||||
|
else:
|
||||||
|
other_as_http_header_dict = type(self)(maybe_constructable)
|
||||||
|
|
||||||
|
return {k.lower(): v for k, v in self.itermerged()} == {
|
||||||
|
k.lower(): v for k, v in other_as_http_header_dict.itermerged()
|
||||||
|
}
|
||||||
|
|
||||||
|
def __ne__(self, other: object) -> bool:
|
||||||
|
return not self.__eq__(other)
|
||||||
|
|
||||||
|
def __len__(self) -> int:
|
||||||
|
return len(self._container)
|
||||||
|
|
||||||
|
def __iter__(self) -> typing.Iterator[str]:
|
||||||
|
# Only provide the originally cased names
|
||||||
|
for vals in self._container.values():
|
||||||
|
yield vals[0]
|
||||||
|
|
||||||
|
def discard(self, key: str) -> None:
|
||||||
|
try:
|
||||||
|
del self[key]
|
||||||
|
except KeyError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def add(self, key: str, val: str, *, combine: bool = False) -> None:
|
||||||
|
"""Adds a (name, value) pair, doesn't overwrite the value if it already
|
||||||
|
exists.
|
||||||
|
|
||||||
|
If this is called with combine=True, instead of adding a new header value
|
||||||
|
as a distinct item during iteration, this will instead append the value to
|
||||||
|
any existing header value with a comma. If no existing header value exists
|
||||||
|
for the key, then the value will simply be added, ignoring the combine parameter.
|
||||||
|
|
||||||
|
>>> headers = HTTPHeaderDict(foo='bar')
|
||||||
|
>>> headers.add('Foo', 'baz')
|
||||||
|
>>> headers['foo']
|
||||||
|
'bar, baz'
|
||||||
|
>>> list(headers.items())
|
||||||
|
[('foo', 'bar'), ('foo', 'baz')]
|
||||||
|
>>> headers.add('foo', 'quz', combine=True)
|
||||||
|
>>> list(headers.items())
|
||||||
|
[('foo', 'bar, baz, quz')]
|
||||||
|
"""
|
||||||
|
# avoid a bytes/str comparison by decoding before httplib
|
||||||
|
if isinstance(key, bytes):
|
||||||
|
key = key.decode("latin-1")
|
||||||
|
key_lower = key.lower()
|
||||||
|
new_vals = [key, val]
|
||||||
|
# Keep the common case aka no item present as fast as possible
|
||||||
|
vals = self._container.setdefault(key_lower, new_vals)
|
||||||
|
if new_vals is not vals:
|
||||||
|
# if there are values here, then there is at least the initial
|
||||||
|
# key/value pair
|
||||||
|
assert len(vals) >= 2
|
||||||
|
if combine:
|
||||||
|
vals[-1] = vals[-1] + ", " + val
|
||||||
|
else:
|
||||||
|
vals.append(val)
|
||||||
|
|
||||||
|
def extend(self, *args: ValidHTTPHeaderSource, **kwargs: str) -> None:
|
||||||
|
"""Generic import function for any type of header-like object.
|
||||||
|
Adapted version of MutableMapping.update in order to insert items
|
||||||
|
with self.add instead of self.__setitem__
|
||||||
|
"""
|
||||||
|
if len(args) > 1:
|
||||||
|
raise TypeError(
|
||||||
|
f"extend() takes at most 1 positional arguments ({len(args)} given)"
|
||||||
|
)
|
||||||
|
other = args[0] if len(args) >= 1 else ()
|
||||||
|
|
||||||
|
if isinstance(other, HTTPHeaderDict):
|
||||||
|
for key, val in other.iteritems():
|
||||||
|
self.add(key, val)
|
||||||
|
elif isinstance(other, typing.Mapping):
|
||||||
|
for key, val in other.items():
|
||||||
|
self.add(key, val)
|
||||||
|
elif isinstance(other, typing.Iterable):
|
||||||
|
for key, value in other:
|
||||||
|
self.add(key, value)
|
||||||
|
elif hasattr(other, "keys") and hasattr(other, "__getitem__"):
|
||||||
|
# THIS IS NOT A TYPESAFE BRANCH
|
||||||
|
# In this branch, the object has a `keys` attr but is not a Mapping or any of
|
||||||
|
# the other types indicated in the method signature. We do some stuff with
|
||||||
|
# it as though it partially implements the Mapping interface, but we're not
|
||||||
|
# doing that stuff safely AT ALL.
|
||||||
|
for key in other.keys():
|
||||||
|
self.add(key, other[key])
|
||||||
|
|
||||||
|
for key, value in kwargs.items():
|
||||||
|
self.add(key, value)
|
||||||
|
|
||||||
|
@typing.overload
|
||||||
|
def getlist(self, key: str) -> list[str]: ...
|
||||||
|
|
||||||
|
@typing.overload
|
||||||
|
def getlist(self, key: str, default: _DT) -> list[str] | _DT: ...
|
||||||
|
|
||||||
|
def getlist(
|
||||||
|
self, key: str, default: _Sentinel | _DT = _Sentinel.not_passed
|
||||||
|
) -> list[str] | _DT:
|
||||||
|
"""Returns a list of all the values for the named field. Returns an
|
||||||
|
empty list if the key doesn't exist."""
|
||||||
|
if isinstance(key, bytes):
|
||||||
|
key = key.decode("latin-1")
|
||||||
|
try:
|
||||||
|
vals = self._container[key.lower()]
|
||||||
|
except KeyError:
|
||||||
|
if default is _Sentinel.not_passed:
|
||||||
|
# _DT is unbound; empty list is instance of List[str]
|
||||||
|
return []
|
||||||
|
# _DT is bound; default is instance of _DT
|
||||||
|
return default
|
||||||
|
else:
|
||||||
|
# _DT may or may not be bound; vals[1:] is instance of List[str], which
|
||||||
|
# meets our external interface requirement of `Union[List[str], _DT]`.
|
||||||
|
return vals[1:]
|
||||||
|
|
||||||
|
def _prepare_for_method_change(self) -> Self:
|
||||||
|
"""
|
||||||
|
Remove content-specific header fields before changing the request
|
||||||
|
method to GET or HEAD according to RFC 9110, Section 15.4.
|
||||||
|
"""
|
||||||
|
content_specific_headers = [
|
||||||
|
"Content-Encoding",
|
||||||
|
"Content-Language",
|
||||||
|
"Content-Location",
|
||||||
|
"Content-Type",
|
||||||
|
"Content-Length",
|
||||||
|
"Digest",
|
||||||
|
"Last-Modified",
|
||||||
|
]
|
||||||
|
for header in content_specific_headers:
|
||||||
|
self.discard(header)
|
||||||
|
return self
|
||||||
|
|
||||||
|
# Backwards compatibility for httplib
|
||||||
|
getheaders = getlist
|
||||||
|
getallmatchingheaders = getlist
|
||||||
|
iget = getlist
|
||||||
|
|
||||||
|
# Backwards compatibility for http.cookiejar
|
||||||
|
get_all = getlist
|
||||||
|
|
||||||
|
def __repr__(self) -> str:
|
||||||
|
return f"{type(self).__name__}({dict(self.itermerged())})"
|
||||||
|
|
||||||
|
def _copy_from(self, other: HTTPHeaderDict) -> None:
|
||||||
|
for key in other:
|
||||||
|
val = other.getlist(key)
|
||||||
|
self._container[key.lower()] = [key, *val]
|
||||||
|
|
||||||
|
def copy(self) -> Self:
|
||||||
|
clone = type(self)()
|
||||||
|
clone._copy_from(self)
|
||||||
|
return clone
|
||||||
|
|
||||||
|
def iteritems(self) -> typing.Iterator[tuple[str, str]]:
|
||||||
|
"""Iterate over all header lines, including duplicate ones."""
|
||||||
|
for key in self:
|
||||||
|
vals = self._container[key.lower()]
|
||||||
|
for val in vals[1:]:
|
||||||
|
yield vals[0], val
|
||||||
|
|
||||||
|
def itermerged(self) -> typing.Iterator[tuple[str, str]]:
|
||||||
|
"""Iterate over all headers, merging duplicate ones together."""
|
||||||
|
for key in self:
|
||||||
|
val = self._container[key.lower()]
|
||||||
|
yield val[0], ", ".join(val[1:])
|
||||||
|
|
||||||
|
def items(self) -> HTTPHeaderDictItemView: # type: ignore[override]
|
||||||
|
return HTTPHeaderDictItemView(self)
|
||||||
|
|
||||||
|
def _has_value_for_header(self, header_name: str, potential_value: str) -> bool:
|
||||||
|
if header_name in self:
|
||||||
|
return potential_value in self._container[header_name.lower()][1:]
|
||||||
|
return False
|
||||||
|
|
||||||
|
def __ior__(self, other: object) -> HTTPHeaderDict:
|
||||||
|
# Supports extending a header dict in-place using operator |=
|
||||||
|
# combining items with add instead of __setitem__
|
||||||
|
maybe_constructable = ensure_can_construct_http_header_dict(other)
|
||||||
|
if maybe_constructable is None:
|
||||||
|
return NotImplemented
|
||||||
|
self.extend(maybe_constructable)
|
||||||
|
return self
|
||||||
|
|
||||||
|
def __or__(self, other: object) -> Self:
|
||||||
|
# Supports merging header dicts using operator |
|
||||||
|
# combining items with add instead of __setitem__
|
||||||
|
maybe_constructable = ensure_can_construct_http_header_dict(other)
|
||||||
|
if maybe_constructable is None:
|
||||||
|
return NotImplemented
|
||||||
|
result = self.copy()
|
||||||
|
result.extend(maybe_constructable)
|
||||||
|
return result
|
||||||
|
|
||||||
|
def __ror__(self, other: object) -> Self:
|
||||||
|
# Supports merging header dicts using operator | when other is on left side
|
||||||
|
# combining items with add instead of __setitem__
|
||||||
|
maybe_constructable = ensure_can_construct_http_header_dict(other)
|
||||||
|
if maybe_constructable is None:
|
||||||
|
return NotImplemented
|
||||||
|
result = type(self)(maybe_constructable)
|
||||||
|
result.extend(self)
|
||||||
|
return result
|
||||||
+278
@@ -0,0 +1,278 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json as _json
|
||||||
|
import typing
|
||||||
|
from urllib.parse import urlencode
|
||||||
|
|
||||||
|
from ._base_connection import _TYPE_BODY
|
||||||
|
from ._collections import HTTPHeaderDict
|
||||||
|
from .filepost import _TYPE_FIELDS, encode_multipart_formdata
|
||||||
|
from .response import BaseHTTPResponse
|
||||||
|
|
||||||
|
__all__ = ["RequestMethods"]
|
||||||
|
|
||||||
|
_TYPE_ENCODE_URL_FIELDS = typing.Union[
|
||||||
|
typing.Sequence[tuple[str, typing.Union[str, bytes]]],
|
||||||
|
typing.Mapping[str, typing.Union[str, bytes]],
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
class RequestMethods:
|
||||||
|
"""
|
||||||
|
Convenience mixin for classes who implement a :meth:`urlopen` method, such
|
||||||
|
as :class:`urllib3.HTTPConnectionPool` and
|
||||||
|
:class:`urllib3.PoolManager`.
|
||||||
|
|
||||||
|
Provides behavior for making common types of HTTP request methods and
|
||||||
|
decides which type of request field encoding to use.
|
||||||
|
|
||||||
|
Specifically,
|
||||||
|
|
||||||
|
:meth:`.request_encode_url` is for sending requests whose fields are
|
||||||
|
encoded in the URL (such as GET, HEAD, DELETE).
|
||||||
|
|
||||||
|
:meth:`.request_encode_body` is for sending requests whose fields are
|
||||||
|
encoded in the *body* of the request using multipart or www-form-urlencoded
|
||||||
|
(such as for POST, PUT, PATCH).
|
||||||
|
|
||||||
|
:meth:`.request` is for making any kind of request, it will look up the
|
||||||
|
appropriate encoding format and use one of the above two methods to make
|
||||||
|
the request.
|
||||||
|
|
||||||
|
Initializer parameters:
|
||||||
|
|
||||||
|
:param headers:
|
||||||
|
Headers to include with all requests, unless other headers are given
|
||||||
|
explicitly.
|
||||||
|
"""
|
||||||
|
|
||||||
|
_encode_url_methods = {"DELETE", "GET", "HEAD", "OPTIONS"}
|
||||||
|
|
||||||
|
def __init__(self, headers: typing.Mapping[str, str] | None = None) -> None:
|
||||||
|
self.headers = headers or {}
|
||||||
|
|
||||||
|
def urlopen(
|
||||||
|
self,
|
||||||
|
method: str,
|
||||||
|
url: str,
|
||||||
|
body: _TYPE_BODY | None = None,
|
||||||
|
headers: typing.Mapping[str, str] | None = None,
|
||||||
|
encode_multipart: bool = True,
|
||||||
|
multipart_boundary: str | None = None,
|
||||||
|
**kw: typing.Any,
|
||||||
|
) -> BaseHTTPResponse: # Abstract
|
||||||
|
raise NotImplementedError(
|
||||||
|
"Classes extending RequestMethods must implement "
|
||||||
|
"their own ``urlopen`` method."
|
||||||
|
)
|
||||||
|
|
||||||
|
def request(
|
||||||
|
self,
|
||||||
|
method: str,
|
||||||
|
url: str,
|
||||||
|
body: _TYPE_BODY | None = None,
|
||||||
|
fields: _TYPE_FIELDS | None = None,
|
||||||
|
headers: typing.Mapping[str, str] | None = None,
|
||||||
|
json: typing.Any | None = None,
|
||||||
|
**urlopen_kw: typing.Any,
|
||||||
|
) -> BaseHTTPResponse:
|
||||||
|
"""
|
||||||
|
Make a request using :meth:`urlopen` with the appropriate encoding of
|
||||||
|
``fields`` based on the ``method`` used.
|
||||||
|
|
||||||
|
This is a convenience method that requires the least amount of manual
|
||||||
|
effort. It can be used in most situations, while still having the
|
||||||
|
option to drop down to more specific methods when necessary, such as
|
||||||
|
:meth:`request_encode_url`, :meth:`request_encode_body`,
|
||||||
|
or even the lowest level :meth:`urlopen`.
|
||||||
|
|
||||||
|
:param method:
|
||||||
|
HTTP request method (such as GET, POST, PUT, etc.)
|
||||||
|
|
||||||
|
:param url:
|
||||||
|
The URL to perform the request on.
|
||||||
|
|
||||||
|
:param body:
|
||||||
|
Data to send in the request body, either :class:`str`, :class:`bytes`,
|
||||||
|
an iterable of :class:`str`/:class:`bytes`, or a file-like object.
|
||||||
|
|
||||||
|
:param fields:
|
||||||
|
Data to encode and send in the URL or request body, depending on ``method``.
|
||||||
|
|
||||||
|
:param headers:
|
||||||
|
Dictionary of custom headers to send, such as User-Agent,
|
||||||
|
If-None-Match, etc. If None, pool headers are used. If provided,
|
||||||
|
these headers completely replace any pool-specific headers.
|
||||||
|
|
||||||
|
:param json:
|
||||||
|
Data to encode and send as JSON with UTF-encoded in the request body.
|
||||||
|
The ``"Content-Type"`` header will be set to ``"application/json"``
|
||||||
|
unless specified otherwise.
|
||||||
|
"""
|
||||||
|
method = method.upper()
|
||||||
|
|
||||||
|
if json is not None and body is not None:
|
||||||
|
raise TypeError(
|
||||||
|
"request got values for both 'body' and 'json' parameters which are mutually exclusive"
|
||||||
|
)
|
||||||
|
|
||||||
|
if json is not None:
|
||||||
|
if headers is None:
|
||||||
|
headers = self.headers
|
||||||
|
|
||||||
|
if not ("content-type" in map(str.lower, headers.keys())):
|
||||||
|
headers = HTTPHeaderDict(headers)
|
||||||
|
headers["Content-Type"] = "application/json"
|
||||||
|
|
||||||
|
body = _json.dumps(json, separators=(",", ":"), ensure_ascii=False).encode(
|
||||||
|
"utf-8"
|
||||||
|
)
|
||||||
|
|
||||||
|
if body is not None:
|
||||||
|
urlopen_kw["body"] = body
|
||||||
|
|
||||||
|
if method in self._encode_url_methods:
|
||||||
|
return self.request_encode_url(
|
||||||
|
method,
|
||||||
|
url,
|
||||||
|
fields=fields, # type: ignore[arg-type]
|
||||||
|
headers=headers,
|
||||||
|
**urlopen_kw,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
return self.request_encode_body(
|
||||||
|
method, url, fields=fields, headers=headers, **urlopen_kw
|
||||||
|
)
|
||||||
|
|
||||||
|
def request_encode_url(
|
||||||
|
self,
|
||||||
|
method: str,
|
||||||
|
url: str,
|
||||||
|
fields: _TYPE_ENCODE_URL_FIELDS | None = None,
|
||||||
|
headers: typing.Mapping[str, str] | None = None,
|
||||||
|
**urlopen_kw: str,
|
||||||
|
) -> BaseHTTPResponse:
|
||||||
|
"""
|
||||||
|
Make a request using :meth:`urlopen` with the ``fields`` encoded in
|
||||||
|
the url. This is useful for request methods like GET, HEAD, DELETE, etc.
|
||||||
|
|
||||||
|
:param method:
|
||||||
|
HTTP request method (such as GET, POST, PUT, etc.)
|
||||||
|
|
||||||
|
:param url:
|
||||||
|
The URL to perform the request on.
|
||||||
|
|
||||||
|
:param fields:
|
||||||
|
Data to encode and send in the URL.
|
||||||
|
|
||||||
|
:param headers:
|
||||||
|
Dictionary of custom headers to send, such as User-Agent,
|
||||||
|
If-None-Match, etc. If None, pool headers are used. If provided,
|
||||||
|
these headers completely replace any pool-specific headers.
|
||||||
|
"""
|
||||||
|
if headers is None:
|
||||||
|
headers = self.headers
|
||||||
|
|
||||||
|
extra_kw: dict[str, typing.Any] = {"headers": headers}
|
||||||
|
extra_kw.update(urlopen_kw)
|
||||||
|
|
||||||
|
if fields:
|
||||||
|
url += "?" + urlencode(fields)
|
||||||
|
|
||||||
|
return self.urlopen(method, url, **extra_kw)
|
||||||
|
|
||||||
|
def request_encode_body(
|
||||||
|
self,
|
||||||
|
method: str,
|
||||||
|
url: str,
|
||||||
|
fields: _TYPE_FIELDS | None = None,
|
||||||
|
headers: typing.Mapping[str, str] | None = None,
|
||||||
|
encode_multipart: bool = True,
|
||||||
|
multipart_boundary: str | None = None,
|
||||||
|
**urlopen_kw: str,
|
||||||
|
) -> BaseHTTPResponse:
|
||||||
|
"""
|
||||||
|
Make a request using :meth:`urlopen` with the ``fields`` encoded in
|
||||||
|
the body. This is useful for request methods like POST, PUT, PATCH, etc.
|
||||||
|
|
||||||
|
When ``encode_multipart=True`` (default), then
|
||||||
|
:func:`urllib3.encode_multipart_formdata` is used to encode
|
||||||
|
the payload with the appropriate content type. Otherwise
|
||||||
|
:func:`urllib.parse.urlencode` is used with the
|
||||||
|
'application/x-www-form-urlencoded' content type.
|
||||||
|
|
||||||
|
Multipart encoding must be used when posting files, and it's reasonably
|
||||||
|
safe to use it in other times too. However, it may break request
|
||||||
|
signing, such as with OAuth.
|
||||||
|
|
||||||
|
Supports an optional ``fields`` parameter of key/value strings AND
|
||||||
|
key/filetuple. A filetuple is a (filename, data, MIME type) tuple where
|
||||||
|
the MIME type is optional. For example::
|
||||||
|
|
||||||
|
fields = {
|
||||||
|
'foo': 'bar',
|
||||||
|
'fakefile': ('foofile.txt', 'contents of foofile'),
|
||||||
|
'realfile': ('barfile.txt', open('realfile').read()),
|
||||||
|
'typedfile': ('bazfile.bin', open('bazfile').read(),
|
||||||
|
'image/jpeg'),
|
||||||
|
'nonamefile': 'contents of nonamefile field',
|
||||||
|
}
|
||||||
|
|
||||||
|
When uploading a file, providing a filename (the first parameter of the
|
||||||
|
tuple) is optional but recommended to best mimic behavior of browsers.
|
||||||
|
|
||||||
|
Note that if ``headers`` are supplied, the 'Content-Type' header will
|
||||||
|
be overwritten because it depends on the dynamic random boundary string
|
||||||
|
which is used to compose the body of the request. The random boundary
|
||||||
|
string can be explicitly set with the ``multipart_boundary`` parameter.
|
||||||
|
|
||||||
|
:param method:
|
||||||
|
HTTP request method (such as GET, POST, PUT, etc.)
|
||||||
|
|
||||||
|
:param url:
|
||||||
|
The URL to perform the request on.
|
||||||
|
|
||||||
|
:param fields:
|
||||||
|
Data to encode and send in the request body.
|
||||||
|
|
||||||
|
:param headers:
|
||||||
|
Dictionary of custom headers to send, such as User-Agent,
|
||||||
|
If-None-Match, etc. If None, pool headers are used. If provided,
|
||||||
|
these headers completely replace any pool-specific headers.
|
||||||
|
|
||||||
|
:param encode_multipart:
|
||||||
|
If True, encode the ``fields`` using the multipart/form-data MIME
|
||||||
|
format.
|
||||||
|
|
||||||
|
:param multipart_boundary:
|
||||||
|
If not specified, then a random boundary will be generated using
|
||||||
|
:func:`urllib3.filepost.choose_boundary`.
|
||||||
|
"""
|
||||||
|
if headers is None:
|
||||||
|
headers = self.headers
|
||||||
|
|
||||||
|
extra_kw: dict[str, typing.Any] = {"headers": HTTPHeaderDict(headers)}
|
||||||
|
body: bytes | str
|
||||||
|
|
||||||
|
if fields:
|
||||||
|
if "body" in urlopen_kw:
|
||||||
|
raise TypeError(
|
||||||
|
"request got values for both 'fields' and 'body', can only specify one."
|
||||||
|
)
|
||||||
|
|
||||||
|
if encode_multipart:
|
||||||
|
body, content_type = encode_multipart_formdata(
|
||||||
|
fields, boundary=multipart_boundary
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
body, content_type = (
|
||||||
|
urlencode(fields), # type: ignore[arg-type]
|
||||||
|
"application/x-www-form-urlencoded",
|
||||||
|
)
|
||||||
|
|
||||||
|
extra_kw["body"] = body
|
||||||
|
extra_kw["headers"].setdefault("Content-Type", content_type)
|
||||||
|
|
||||||
|
extra_kw.update(urlopen_kw)
|
||||||
|
|
||||||
|
return self.urlopen(method, url, **extra_kw)
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# file generated by vcs-versioning
|
||||||
|
# don't change, don't track in version control
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
__all__ = [
|
||||||
|
"__version__",
|
||||||
|
"__version_tuple__",
|
||||||
|
"version",
|
||||||
|
"version_tuple",
|
||||||
|
"__commit_id__",
|
||||||
|
"commit_id",
|
||||||
|
]
|
||||||
|
|
||||||
|
version: str
|
||||||
|
__version__: str
|
||||||
|
__version_tuple__: tuple[int | str, ...]
|
||||||
|
version_tuple: tuple[int | str, ...]
|
||||||
|
commit_id: str | None
|
||||||
|
__commit_id__: str | None
|
||||||
|
|
||||||
|
__version__ = version = '2.7.0'
|
||||||
|
__version_tuple__ = version_tuple = (2, 7, 0)
|
||||||
|
|
||||||
|
__commit_id__ = commit_id = None
|
||||||
+1099
File diff suppressed because it is too large
Load Diff
+1191
File diff suppressed because it is too large
Load Diff
+17
@@ -0,0 +1,17 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import urllib3.connection
|
||||||
|
|
||||||
|
from ...connectionpool import HTTPConnectionPool, HTTPSConnectionPool
|
||||||
|
from .connection import EmscriptenHTTPConnection, EmscriptenHTTPSConnection
|
||||||
|
|
||||||
|
|
||||||
|
def inject_into_urllib3() -> None:
|
||||||
|
# override connection classes to use emscripten specific classes
|
||||||
|
# n.b. mypy complains about the overriding of classes below
|
||||||
|
# if it isn't ignored
|
||||||
|
HTTPConnectionPool.ConnectionCls = EmscriptenHTTPConnection
|
||||||
|
HTTPSConnectionPool.ConnectionCls = EmscriptenHTTPSConnection
|
||||||
|
urllib3.connection.HTTPConnection = EmscriptenHTTPConnection # type: ignore[misc,assignment]
|
||||||
|
urllib3.connection.HTTPSConnection = EmscriptenHTTPSConnection # type: ignore[misc,assignment]
|
||||||
|
urllib3.connection.VerifiedHTTPSConnection = EmscriptenHTTPSConnection # type: ignore[assignment]
|
||||||
+260
@@ -0,0 +1,260 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import os
|
||||||
|
import typing
|
||||||
|
|
||||||
|
# use http.client.HTTPException for consistency with non-emscripten
|
||||||
|
from http.client import HTTPException as HTTPException # noqa: F401
|
||||||
|
from http.client import ResponseNotReady
|
||||||
|
|
||||||
|
from ..._base_connection import _TYPE_BODY
|
||||||
|
from ...connection import HTTPConnection, ProxyConfig, port_by_scheme
|
||||||
|
from ...exceptions import TimeoutError
|
||||||
|
from ...response import BaseHTTPResponse
|
||||||
|
from ...util.connection import _TYPE_SOCKET_OPTIONS
|
||||||
|
from ...util.timeout import _DEFAULT_TIMEOUT, _TYPE_TIMEOUT
|
||||||
|
from ...util.url import Url
|
||||||
|
from .fetch import _RequestError, _TimeoutError, send_request, send_streaming_request
|
||||||
|
from .request import EmscriptenRequest
|
||||||
|
from .response import EmscriptenHttpResponseWrapper, EmscriptenResponse
|
||||||
|
|
||||||
|
if typing.TYPE_CHECKING:
|
||||||
|
from ..._base_connection import BaseHTTPConnection, BaseHTTPSConnection
|
||||||
|
|
||||||
|
|
||||||
|
class EmscriptenHTTPConnection:
|
||||||
|
default_port: typing.ClassVar[int] = port_by_scheme["http"]
|
||||||
|
default_socket_options: typing.ClassVar[_TYPE_SOCKET_OPTIONS]
|
||||||
|
|
||||||
|
timeout: None | (float)
|
||||||
|
|
||||||
|
host: str
|
||||||
|
port: int
|
||||||
|
blocksize: int
|
||||||
|
source_address: tuple[str, int] | None
|
||||||
|
socket_options: _TYPE_SOCKET_OPTIONS | None
|
||||||
|
|
||||||
|
proxy: Url | None
|
||||||
|
proxy_config: ProxyConfig | None
|
||||||
|
|
||||||
|
is_verified: bool = False
|
||||||
|
proxy_is_verified: bool | None = None
|
||||||
|
|
||||||
|
response_class: type[BaseHTTPResponse] = EmscriptenHttpResponseWrapper
|
||||||
|
_response: EmscriptenResponse | None
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
host: str,
|
||||||
|
port: int = 0,
|
||||||
|
*,
|
||||||
|
timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
|
||||||
|
source_address: tuple[str, int] | None = None,
|
||||||
|
blocksize: int = 8192,
|
||||||
|
socket_options: _TYPE_SOCKET_OPTIONS | None = None,
|
||||||
|
proxy: Url | None = None,
|
||||||
|
proxy_config: ProxyConfig | None = None,
|
||||||
|
) -> None:
|
||||||
|
self.host = host
|
||||||
|
self.port = port
|
||||||
|
self.timeout = timeout if isinstance(timeout, float) else 0.0
|
||||||
|
self.scheme = "http"
|
||||||
|
self._closed = True
|
||||||
|
self._response = None
|
||||||
|
# ignore these things because we don't
|
||||||
|
# have control over that stuff
|
||||||
|
self.proxy = None
|
||||||
|
self.proxy_config = None
|
||||||
|
self.blocksize = blocksize
|
||||||
|
self.source_address = None
|
||||||
|
self.socket_options = None
|
||||||
|
self.is_verified = False
|
||||||
|
|
||||||
|
def set_tunnel(
|
||||||
|
self,
|
||||||
|
host: str,
|
||||||
|
port: int | None = 0,
|
||||||
|
headers: typing.Mapping[str, str] | None = None,
|
||||||
|
scheme: str = "http",
|
||||||
|
) -> None:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def connect(self) -> None:
|
||||||
|
pass
|
||||||
|
|
||||||
|
def request(
|
||||||
|
self,
|
||||||
|
method: str,
|
||||||
|
url: str,
|
||||||
|
body: _TYPE_BODY | None = None,
|
||||||
|
headers: typing.Mapping[str, str] | None = None,
|
||||||
|
# We know *at least* botocore is depending on the order of the
|
||||||
|
# first 3 parameters so to be safe we only mark the later ones
|
||||||
|
# as keyword-only to ensure we have space to extend.
|
||||||
|
*,
|
||||||
|
chunked: bool = False,
|
||||||
|
preload_content: bool = True,
|
||||||
|
decode_content: bool = True,
|
||||||
|
enforce_content_length: bool = True,
|
||||||
|
) -> None:
|
||||||
|
self._closed = False
|
||||||
|
if url.startswith("/"):
|
||||||
|
if self.port is not None:
|
||||||
|
port = f":{self.port}"
|
||||||
|
else:
|
||||||
|
port = ""
|
||||||
|
# no scheme / host / port included, make a full url
|
||||||
|
url = f"{self.scheme}://{self.host}{port}{url}"
|
||||||
|
request = EmscriptenRequest(
|
||||||
|
url=url,
|
||||||
|
method=method,
|
||||||
|
timeout=self.timeout if self.timeout else 0,
|
||||||
|
decode_content=decode_content,
|
||||||
|
)
|
||||||
|
request.set_body(body)
|
||||||
|
if headers:
|
||||||
|
for k, v in headers.items():
|
||||||
|
request.set_header(k, v)
|
||||||
|
self._response = None
|
||||||
|
try:
|
||||||
|
if not preload_content:
|
||||||
|
self._response = send_streaming_request(request)
|
||||||
|
if self._response is None:
|
||||||
|
self._response = send_request(request)
|
||||||
|
except _TimeoutError as e:
|
||||||
|
raise TimeoutError(e.message) from e
|
||||||
|
except _RequestError as e:
|
||||||
|
raise HTTPException(e.message) from e
|
||||||
|
|
||||||
|
def getresponse(self) -> BaseHTTPResponse:
|
||||||
|
if self._response is not None:
|
||||||
|
return EmscriptenHttpResponseWrapper(
|
||||||
|
internal_response=self._response,
|
||||||
|
url=self._response.request.url,
|
||||||
|
connection=self,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
raise ResponseNotReady()
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
self._closed = True
|
||||||
|
self._response = None
|
||||||
|
|
||||||
|
@property
|
||||||
|
def is_closed(self) -> bool:
|
||||||
|
"""Whether the connection either is brand new or has been previously closed.
|
||||||
|
If this property is True then both ``is_connected`` and ``has_connected_to_proxy``
|
||||||
|
properties must be False.
|
||||||
|
"""
|
||||||
|
return self._closed
|
||||||
|
|
||||||
|
@property
|
||||||
|
def is_connected(self) -> bool:
|
||||||
|
"""Whether the connection is actively connected to any origin (proxy or target)"""
|
||||||
|
return True
|
||||||
|
|
||||||
|
@property
|
||||||
|
def has_connected_to_proxy(self) -> bool:
|
||||||
|
"""Whether the connection has successfully connected to its proxy.
|
||||||
|
This returns False if no proxy is in use. Used to determine whether
|
||||||
|
errors are coming from the proxy layer or from tunnelling to the target origin.
|
||||||
|
"""
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
class EmscriptenHTTPSConnection(EmscriptenHTTPConnection):
|
||||||
|
default_port = port_by_scheme["https"]
|
||||||
|
# all this is basically ignored, as browser handles https
|
||||||
|
cert_reqs: int | str | None = None
|
||||||
|
ca_certs: str | None = None
|
||||||
|
ca_cert_dir: str | None = None
|
||||||
|
ca_cert_data: None | str | bytes = None
|
||||||
|
cert_file: str | None
|
||||||
|
key_file: str | None
|
||||||
|
key_password: str | None
|
||||||
|
ssl_context: typing.Any | None
|
||||||
|
ssl_version: int | str | None = None
|
||||||
|
ssl_minimum_version: int | None = None
|
||||||
|
ssl_maximum_version: int | None = None
|
||||||
|
assert_hostname: None | str | typing.Literal[False]
|
||||||
|
assert_fingerprint: str | None = None
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
host: str,
|
||||||
|
port: int = 0,
|
||||||
|
*,
|
||||||
|
timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
|
||||||
|
source_address: tuple[str, int] | None = None,
|
||||||
|
blocksize: int = 16384,
|
||||||
|
socket_options: (
|
||||||
|
None | _TYPE_SOCKET_OPTIONS
|
||||||
|
) = HTTPConnection.default_socket_options,
|
||||||
|
proxy: Url | None = None,
|
||||||
|
proxy_config: ProxyConfig | None = None,
|
||||||
|
cert_reqs: int | str | None = None,
|
||||||
|
assert_hostname: None | str | typing.Literal[False] = None,
|
||||||
|
assert_fingerprint: str | None = None,
|
||||||
|
server_hostname: str | None = None,
|
||||||
|
ssl_context: typing.Any | None = None,
|
||||||
|
ca_certs: str | None = None,
|
||||||
|
ca_cert_dir: str | None = None,
|
||||||
|
ca_cert_data: None | str | bytes = None,
|
||||||
|
ssl_minimum_version: int | None = None,
|
||||||
|
ssl_maximum_version: int | None = None,
|
||||||
|
ssl_version: int | str | None = None, # Deprecated
|
||||||
|
cert_file: str | None = None,
|
||||||
|
key_file: str | None = None,
|
||||||
|
key_password: str | None = None,
|
||||||
|
) -> None:
|
||||||
|
super().__init__(
|
||||||
|
host,
|
||||||
|
port=port,
|
||||||
|
timeout=timeout,
|
||||||
|
source_address=source_address,
|
||||||
|
blocksize=blocksize,
|
||||||
|
socket_options=socket_options,
|
||||||
|
proxy=proxy,
|
||||||
|
proxy_config=proxy_config,
|
||||||
|
)
|
||||||
|
self.scheme = "https"
|
||||||
|
|
||||||
|
self.key_file = key_file
|
||||||
|
self.cert_file = cert_file
|
||||||
|
self.key_password = key_password
|
||||||
|
self.ssl_context = ssl_context
|
||||||
|
self.server_hostname = server_hostname
|
||||||
|
self.assert_hostname = assert_hostname
|
||||||
|
self.assert_fingerprint = assert_fingerprint
|
||||||
|
self.ssl_version = ssl_version
|
||||||
|
self.ssl_minimum_version = ssl_minimum_version
|
||||||
|
self.ssl_maximum_version = ssl_maximum_version
|
||||||
|
self.ca_certs = ca_certs and os.path.expanduser(ca_certs)
|
||||||
|
self.ca_cert_dir = ca_cert_dir and os.path.expanduser(ca_cert_dir)
|
||||||
|
self.ca_cert_data = ca_cert_data
|
||||||
|
|
||||||
|
self.cert_reqs = None
|
||||||
|
|
||||||
|
# The browser will automatically verify all requests.
|
||||||
|
# We have no control over that setting.
|
||||||
|
self.is_verified = True
|
||||||
|
|
||||||
|
def set_cert(
|
||||||
|
self,
|
||||||
|
key_file: str | None = None,
|
||||||
|
cert_file: str | None = None,
|
||||||
|
cert_reqs: int | str | None = None,
|
||||||
|
key_password: str | None = None,
|
||||||
|
ca_certs: str | None = None,
|
||||||
|
assert_hostname: None | str | typing.Literal[False] = None,
|
||||||
|
assert_fingerprint: str | None = None,
|
||||||
|
ca_cert_dir: str | None = None,
|
||||||
|
ca_cert_data: None | str | bytes = None,
|
||||||
|
) -> None:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
# verify that this class implements BaseHTTP(s) connection correctly
|
||||||
|
if typing.TYPE_CHECKING:
|
||||||
|
_supports_http_protocol: BaseHTTPConnection = EmscriptenHTTPConnection("", 0)
|
||||||
|
_supports_https_protocol: BaseHTTPSConnection = EmscriptenHTTPSConnection("", 0)
|
||||||
+726
@@ -0,0 +1,726 @@
|
|||||||
|
"""
|
||||||
|
Support for streaming http requests in emscripten.
|
||||||
|
|
||||||
|
A few caveats -
|
||||||
|
|
||||||
|
If your browser (or Node.js) has WebAssembly JavaScript Promise Integration enabled
|
||||||
|
https://github.com/WebAssembly/js-promise-integration/blob/main/proposals/js-promise-integration/Overview.md
|
||||||
|
*and* you launch pyodide using `pyodide.runPythonAsync`, this will fetch data using the
|
||||||
|
JavaScript asynchronous fetch api (wrapped via `pyodide.ffi.call_sync`). In this case
|
||||||
|
timeouts and streaming should just work.
|
||||||
|
|
||||||
|
Otherwise, it uses a combination of XMLHttpRequest and a web-worker for streaming.
|
||||||
|
|
||||||
|
This approach has several caveats:
|
||||||
|
|
||||||
|
Firstly, you can't do streaming http in the main UI thread, because atomics.wait isn't allowed.
|
||||||
|
Streaming only works if you're running pyodide in a web worker.
|
||||||
|
|
||||||
|
Secondly, this uses an extra web worker and SharedArrayBuffer to do the asynchronous fetch
|
||||||
|
operation, so it requires that you have crossOriginIsolation enabled, by serving over https
|
||||||
|
(or from localhost) with the two headers below set:
|
||||||
|
|
||||||
|
Cross-Origin-Opener-Policy: same-origin
|
||||||
|
Cross-Origin-Embedder-Policy: require-corp
|
||||||
|
|
||||||
|
You can tell if cross origin isolation is successfully enabled by looking at the global crossOriginIsolated variable in
|
||||||
|
JavaScript console. If it isn't, streaming requests will fallback to XMLHttpRequest, i.e. getting the whole
|
||||||
|
request into a buffer and then returning it. it shows a warning in the JavaScript console in this case.
|
||||||
|
|
||||||
|
Finally, the webworker which does the streaming fetch is created on initial import, but will only be started once
|
||||||
|
control is returned to javascript. Call `await wait_for_streaming_ready()` to wait for streaming fetch.
|
||||||
|
|
||||||
|
NB: in this code, there are a lot of JavaScript objects. They are named js_*
|
||||||
|
to make it clear what type of object they are.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import io
|
||||||
|
import json
|
||||||
|
from email.parser import Parser
|
||||||
|
from importlib.resources import files
|
||||||
|
from typing import TYPE_CHECKING, Any
|
||||||
|
|
||||||
|
import js # type: ignore[import-not-found]
|
||||||
|
from pyodide.ffi import ( # type: ignore[import-not-found]
|
||||||
|
JsArray,
|
||||||
|
JsException,
|
||||||
|
JsProxy,
|
||||||
|
to_js,
|
||||||
|
)
|
||||||
|
|
||||||
|
if TYPE_CHECKING:
|
||||||
|
from typing_extensions import Buffer
|
||||||
|
|
||||||
|
from .request import EmscriptenRequest
|
||||||
|
from .response import EmscriptenResponse
|
||||||
|
|
||||||
|
"""
|
||||||
|
There are some headers that trigger unintended CORS preflight requests.
|
||||||
|
See also https://github.com/koenvo/pyodide-http/issues/22
|
||||||
|
"""
|
||||||
|
HEADERS_TO_IGNORE = ("user-agent",)
|
||||||
|
|
||||||
|
SUCCESS_HEADER = -1
|
||||||
|
SUCCESS_EOF = -2
|
||||||
|
ERROR_TIMEOUT = -3
|
||||||
|
ERROR_EXCEPTION = -4
|
||||||
|
|
||||||
|
|
||||||
|
class _RequestError(Exception):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
message: str | None = None,
|
||||||
|
*,
|
||||||
|
request: EmscriptenRequest | None = None,
|
||||||
|
response: EmscriptenResponse | None = None,
|
||||||
|
):
|
||||||
|
self.request = request
|
||||||
|
self.response = response
|
||||||
|
self.message = message
|
||||||
|
super().__init__(self.message)
|
||||||
|
|
||||||
|
|
||||||
|
class _StreamingError(_RequestError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class _TimeoutError(_RequestError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def _obj_from_dict(dict_val: dict[str, Any]) -> JsProxy:
|
||||||
|
return to_js(dict_val, dict_converter=js.Object.fromEntries)
|
||||||
|
|
||||||
|
|
||||||
|
class _ReadStream(io.RawIOBase):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
int_buffer: JsArray,
|
||||||
|
byte_buffer: JsArray,
|
||||||
|
timeout: float,
|
||||||
|
worker: JsProxy,
|
||||||
|
connection_id: int,
|
||||||
|
request: EmscriptenRequest,
|
||||||
|
):
|
||||||
|
self.int_buffer = int_buffer
|
||||||
|
self.byte_buffer = byte_buffer
|
||||||
|
self.read_pos = 0
|
||||||
|
self.read_len = 0
|
||||||
|
self.connection_id = connection_id
|
||||||
|
self.worker = worker
|
||||||
|
self.timeout = int(1000 * timeout) if timeout > 0 else None
|
||||||
|
self.is_live = True
|
||||||
|
self._is_closed = False
|
||||||
|
self.request: EmscriptenRequest | None = request
|
||||||
|
|
||||||
|
def __del__(self) -> None:
|
||||||
|
self.close()
|
||||||
|
|
||||||
|
# this is compatible with _base_connection
|
||||||
|
def is_closed(self) -> bool:
|
||||||
|
return self._is_closed
|
||||||
|
|
||||||
|
# for compatibility with RawIOBase
|
||||||
|
@property
|
||||||
|
def closed(self) -> bool:
|
||||||
|
return self.is_closed()
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
if self.is_closed():
|
||||||
|
return
|
||||||
|
self.read_len = 0
|
||||||
|
self.read_pos = 0
|
||||||
|
self.int_buffer = None
|
||||||
|
self.byte_buffer = None
|
||||||
|
self._is_closed = True
|
||||||
|
self.request = None
|
||||||
|
if self.is_live:
|
||||||
|
self.worker.postMessage(_obj_from_dict({"close": self.connection_id}))
|
||||||
|
self.is_live = False
|
||||||
|
super().close()
|
||||||
|
|
||||||
|
def readable(self) -> bool:
|
||||||
|
return True
|
||||||
|
|
||||||
|
def writable(self) -> bool:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def seekable(self) -> bool:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def readinto(self, byte_obj: Buffer) -> int:
|
||||||
|
if not self.int_buffer:
|
||||||
|
raise _StreamingError(
|
||||||
|
"No buffer for stream in _ReadStream.readinto",
|
||||||
|
request=self.request,
|
||||||
|
response=None,
|
||||||
|
)
|
||||||
|
if self.read_len == 0:
|
||||||
|
# wait for the worker to send something
|
||||||
|
js.Atomics.store(self.int_buffer, 0, ERROR_TIMEOUT)
|
||||||
|
self.worker.postMessage(_obj_from_dict({"getMore": self.connection_id}))
|
||||||
|
if (
|
||||||
|
js.Atomics.wait(self.int_buffer, 0, ERROR_TIMEOUT, self.timeout)
|
||||||
|
== "timed-out"
|
||||||
|
):
|
||||||
|
raise _TimeoutError
|
||||||
|
data_len = self.int_buffer[0]
|
||||||
|
if data_len > 0:
|
||||||
|
self.read_len = data_len
|
||||||
|
self.read_pos = 0
|
||||||
|
elif data_len == ERROR_EXCEPTION:
|
||||||
|
string_len = self.int_buffer[1]
|
||||||
|
# decode the error string
|
||||||
|
js_decoder = js.TextDecoder.new()
|
||||||
|
json_str = js_decoder.decode(self.byte_buffer.slice(0, string_len))
|
||||||
|
raise _StreamingError(
|
||||||
|
f"Exception thrown in fetch: {json_str}",
|
||||||
|
request=self.request,
|
||||||
|
response=None,
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# EOF, free the buffers and return zero
|
||||||
|
# and free the request
|
||||||
|
self.is_live = False
|
||||||
|
self.close()
|
||||||
|
return 0
|
||||||
|
# copy from int32array to python bytes
|
||||||
|
ret_length = min(self.read_len, len(memoryview(byte_obj)))
|
||||||
|
subarray = self.byte_buffer.subarray(
|
||||||
|
self.read_pos, self.read_pos + ret_length
|
||||||
|
).to_py()
|
||||||
|
memoryview(byte_obj)[0:ret_length] = subarray
|
||||||
|
self.read_len -= ret_length
|
||||||
|
self.read_pos += ret_length
|
||||||
|
return ret_length
|
||||||
|
|
||||||
|
|
||||||
|
class _StreamingFetcher:
|
||||||
|
def __init__(self) -> None:
|
||||||
|
# make web-worker and data buffer on startup
|
||||||
|
self.streaming_ready = False
|
||||||
|
streaming_worker_code = (
|
||||||
|
files(__package__)
|
||||||
|
.joinpath("emscripten_fetch_worker.js")
|
||||||
|
.read_text(encoding="utf-8")
|
||||||
|
)
|
||||||
|
js_data_blob = js.Blob.new(
|
||||||
|
to_js([streaming_worker_code], create_pyproxies=False),
|
||||||
|
_obj_from_dict({"type": "application/javascript"}),
|
||||||
|
)
|
||||||
|
|
||||||
|
def promise_resolver(js_resolve_fn: JsProxy, js_reject_fn: JsProxy) -> None:
|
||||||
|
def onMsg(e: JsProxy) -> None:
|
||||||
|
self.streaming_ready = True
|
||||||
|
js_resolve_fn(e)
|
||||||
|
|
||||||
|
def onErr(e: JsProxy) -> None:
|
||||||
|
js_reject_fn(e) # Defensive: never happens in ci
|
||||||
|
|
||||||
|
self.js_worker.onmessage = onMsg
|
||||||
|
self.js_worker.onerror = onErr
|
||||||
|
|
||||||
|
js_data_url = js.URL.createObjectURL(js_data_blob)
|
||||||
|
self.js_worker = js.globalThis.Worker.new(js_data_url)
|
||||||
|
self.js_worker_ready_promise = js.globalThis.Promise.new(promise_resolver)
|
||||||
|
|
||||||
|
def send(self, request: EmscriptenRequest) -> EmscriptenResponse:
|
||||||
|
headers = {
|
||||||
|
k: v for k, v in request.headers.items() if k not in HEADERS_TO_IGNORE
|
||||||
|
}
|
||||||
|
|
||||||
|
body = request.body
|
||||||
|
fetch_data = {"headers": headers, "body": to_js(body), "method": request.method}
|
||||||
|
# start the request off in the worker
|
||||||
|
timeout = int(1000 * request.timeout) if request.timeout > 0 else None
|
||||||
|
js_shared_buffer = js.SharedArrayBuffer.new(1048576)
|
||||||
|
js_int_buffer = js.Int32Array.new(js_shared_buffer)
|
||||||
|
js_byte_buffer = js.Uint8Array.new(js_shared_buffer, 8)
|
||||||
|
|
||||||
|
js.Atomics.store(js_int_buffer, 0, ERROR_TIMEOUT)
|
||||||
|
js.Atomics.notify(js_int_buffer, 0)
|
||||||
|
js_absolute_url = js.URL.new(request.url, js.location).href
|
||||||
|
self.js_worker.postMessage(
|
||||||
|
_obj_from_dict(
|
||||||
|
{
|
||||||
|
"buffer": js_shared_buffer,
|
||||||
|
"url": js_absolute_url,
|
||||||
|
"fetchParams": fetch_data,
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
# wait for the worker to send something
|
||||||
|
js.Atomics.wait(js_int_buffer, 0, ERROR_TIMEOUT, timeout)
|
||||||
|
if js_int_buffer[0] == ERROR_TIMEOUT:
|
||||||
|
raise _TimeoutError(
|
||||||
|
"Timeout connecting to streaming request",
|
||||||
|
request=request,
|
||||||
|
response=None,
|
||||||
|
)
|
||||||
|
elif js_int_buffer[0] == SUCCESS_HEADER:
|
||||||
|
# got response
|
||||||
|
# header length is in second int of intBuffer
|
||||||
|
string_len = js_int_buffer[1]
|
||||||
|
# decode the rest to a JSON string
|
||||||
|
js_decoder = js.TextDecoder.new()
|
||||||
|
# this does a copy (the slice) because decode can't work on shared array
|
||||||
|
# for some silly reason
|
||||||
|
json_str = js_decoder.decode(js_byte_buffer.slice(0, string_len))
|
||||||
|
# get it as an object
|
||||||
|
response_obj = json.loads(json_str)
|
||||||
|
return EmscriptenResponse(
|
||||||
|
request=request,
|
||||||
|
status_code=response_obj["status"],
|
||||||
|
headers=response_obj["headers"],
|
||||||
|
body=_ReadStream(
|
||||||
|
js_int_buffer,
|
||||||
|
js_byte_buffer,
|
||||||
|
request.timeout,
|
||||||
|
self.js_worker,
|
||||||
|
response_obj["connectionID"],
|
||||||
|
request,
|
||||||
|
),
|
||||||
|
)
|
||||||
|
elif js_int_buffer[0] == ERROR_EXCEPTION:
|
||||||
|
string_len = js_int_buffer[1]
|
||||||
|
# decode the error string
|
||||||
|
js_decoder = js.TextDecoder.new()
|
||||||
|
json_str = js_decoder.decode(js_byte_buffer.slice(0, string_len))
|
||||||
|
raise _StreamingError(
|
||||||
|
f"Exception thrown in fetch: {json_str}", request=request, response=None
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
raise _StreamingError(
|
||||||
|
f"Unknown status from worker in fetch: {js_int_buffer[0]}",
|
||||||
|
request=request,
|
||||||
|
response=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class _JSPIReadStream(io.RawIOBase):
|
||||||
|
"""
|
||||||
|
A read stream that uses pyodide.ffi.run_sync to read from a JavaScript fetch
|
||||||
|
response. This requires support for WebAssembly JavaScript Promise Integration
|
||||||
|
in the containing browser, and for pyodide to be launched via runPythonAsync.
|
||||||
|
|
||||||
|
:param js_read_stream:
|
||||||
|
The JavaScript stream reader
|
||||||
|
|
||||||
|
:param timeout:
|
||||||
|
Timeout in seconds
|
||||||
|
|
||||||
|
:param request:
|
||||||
|
The request we're handling
|
||||||
|
|
||||||
|
:param response:
|
||||||
|
The response this stream relates to
|
||||||
|
|
||||||
|
:param js_abort_controller:
|
||||||
|
A JavaScript AbortController object, used for timeouts
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
js_read_stream: Any,
|
||||||
|
timeout: float,
|
||||||
|
request: EmscriptenRequest,
|
||||||
|
response: EmscriptenResponse,
|
||||||
|
js_abort_controller: Any, # JavaScript AbortController for timeouts
|
||||||
|
):
|
||||||
|
self.js_read_stream = js_read_stream
|
||||||
|
self.timeout = timeout
|
||||||
|
self._is_closed = False
|
||||||
|
self._is_done = False
|
||||||
|
self.request: EmscriptenRequest | None = request
|
||||||
|
self.response: EmscriptenResponse | None = response
|
||||||
|
self.current_buffer = None
|
||||||
|
self.current_buffer_pos = 0
|
||||||
|
self.js_abort_controller = js_abort_controller
|
||||||
|
|
||||||
|
def __del__(self) -> None:
|
||||||
|
self.close()
|
||||||
|
|
||||||
|
# this is compatible with _base_connection
|
||||||
|
def is_closed(self) -> bool:
|
||||||
|
return self._is_closed
|
||||||
|
|
||||||
|
# for compatibility with RawIOBase
|
||||||
|
@property
|
||||||
|
def closed(self) -> bool:
|
||||||
|
return self.is_closed()
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
if self.is_closed():
|
||||||
|
return
|
||||||
|
self.read_len = 0
|
||||||
|
self.read_pos = 0
|
||||||
|
self.js_read_stream.cancel()
|
||||||
|
self.js_read_stream = None
|
||||||
|
self._is_closed = True
|
||||||
|
self._is_done = True
|
||||||
|
self.request = None
|
||||||
|
self.response = None
|
||||||
|
super().close()
|
||||||
|
|
||||||
|
def readable(self) -> bool:
|
||||||
|
return True
|
||||||
|
|
||||||
|
def writable(self) -> bool:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def seekable(self) -> bool:
|
||||||
|
return False
|
||||||
|
|
||||||
|
def _get_next_buffer(self) -> bool:
|
||||||
|
result_js = _run_sync_with_timeout(
|
||||||
|
self.js_read_stream.read(),
|
||||||
|
self.timeout,
|
||||||
|
self.js_abort_controller,
|
||||||
|
request=self.request,
|
||||||
|
response=self.response,
|
||||||
|
)
|
||||||
|
if result_js.done:
|
||||||
|
self._is_done = True
|
||||||
|
return False
|
||||||
|
else:
|
||||||
|
self.current_buffer = result_js.value.to_py()
|
||||||
|
self.current_buffer_pos = 0
|
||||||
|
return True
|
||||||
|
|
||||||
|
def readinto(self, byte_obj: Buffer) -> int:
|
||||||
|
if self.current_buffer is None:
|
||||||
|
if not self._get_next_buffer() or self.current_buffer is None:
|
||||||
|
self.close()
|
||||||
|
return 0
|
||||||
|
ret_length = min(
|
||||||
|
len(byte_obj), len(self.current_buffer) - self.current_buffer_pos
|
||||||
|
)
|
||||||
|
byte_obj[0:ret_length] = self.current_buffer[
|
||||||
|
self.current_buffer_pos : self.current_buffer_pos + ret_length
|
||||||
|
]
|
||||||
|
self.current_buffer_pos += ret_length
|
||||||
|
if self.current_buffer_pos == len(self.current_buffer):
|
||||||
|
self.current_buffer = None
|
||||||
|
return ret_length
|
||||||
|
|
||||||
|
|
||||||
|
# check if we are in a worker or not
|
||||||
|
def is_in_browser_main_thread() -> bool:
|
||||||
|
return hasattr(js, "window") and hasattr(js, "self") and js.self == js.window
|
||||||
|
|
||||||
|
|
||||||
|
def is_cross_origin_isolated() -> bool:
|
||||||
|
return hasattr(js, "crossOriginIsolated") and js.crossOriginIsolated
|
||||||
|
|
||||||
|
|
||||||
|
def is_in_node() -> bool:
|
||||||
|
return (
|
||||||
|
hasattr(js, "process")
|
||||||
|
and hasattr(js.process, "release")
|
||||||
|
and hasattr(js.process.release, "name")
|
||||||
|
and js.process.release.name == "node"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def is_worker_available() -> bool:
|
||||||
|
return hasattr(js, "Worker") and hasattr(js, "Blob")
|
||||||
|
|
||||||
|
|
||||||
|
_fetcher: _StreamingFetcher | None = None
|
||||||
|
|
||||||
|
if is_worker_available() and (
|
||||||
|
(is_cross_origin_isolated() and not is_in_browser_main_thread())
|
||||||
|
and (not is_in_node())
|
||||||
|
):
|
||||||
|
_fetcher = _StreamingFetcher()
|
||||||
|
else:
|
||||||
|
_fetcher = None
|
||||||
|
|
||||||
|
|
||||||
|
NODE_JSPI_ERROR = (
|
||||||
|
"urllib3 only works in Node.js with pyodide.runPythonAsync"
|
||||||
|
" and requires the flag --experimental-wasm-stack-switching in "
|
||||||
|
" versions of node <24."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def send_streaming_request(request: EmscriptenRequest) -> EmscriptenResponse | None:
|
||||||
|
if has_jspi():
|
||||||
|
return send_jspi_request(request, True)
|
||||||
|
elif is_in_node():
|
||||||
|
raise _RequestError(
|
||||||
|
message=NODE_JSPI_ERROR,
|
||||||
|
request=request,
|
||||||
|
response=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
if _fetcher and streaming_ready():
|
||||||
|
return _fetcher.send(request)
|
||||||
|
else:
|
||||||
|
_show_streaming_warning()
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
_SHOWN_TIMEOUT_WARNING = False
|
||||||
|
|
||||||
|
|
||||||
|
def _show_timeout_warning() -> None:
|
||||||
|
global _SHOWN_TIMEOUT_WARNING
|
||||||
|
if not _SHOWN_TIMEOUT_WARNING:
|
||||||
|
_SHOWN_TIMEOUT_WARNING = True
|
||||||
|
message = "Warning: Timeout is not available on main browser thread"
|
||||||
|
js.console.warn(message)
|
||||||
|
|
||||||
|
|
||||||
|
_SHOWN_STREAMING_WARNING = False
|
||||||
|
|
||||||
|
|
||||||
|
def _show_streaming_warning() -> None:
|
||||||
|
global _SHOWN_STREAMING_WARNING
|
||||||
|
if not _SHOWN_STREAMING_WARNING:
|
||||||
|
_SHOWN_STREAMING_WARNING = True
|
||||||
|
message = "Can't stream HTTP requests because: \n"
|
||||||
|
if not is_cross_origin_isolated():
|
||||||
|
message += " Page is not cross-origin isolated\n"
|
||||||
|
if is_in_browser_main_thread():
|
||||||
|
message += " Python is running in main browser thread\n"
|
||||||
|
if not is_worker_available():
|
||||||
|
message += " Worker or Blob classes are not available in this environment." # Defensive: this is always False in browsers that we test in
|
||||||
|
if streaming_ready() is False:
|
||||||
|
message += """ Streaming fetch worker isn't ready. If you want to be sure that streaming fetch
|
||||||
|
is working, you need to call: 'await urllib3.contrib.emscripten.fetch.wait_for_streaming_ready()`"""
|
||||||
|
from js import console
|
||||||
|
|
||||||
|
console.warn(message)
|
||||||
|
|
||||||
|
|
||||||
|
def send_request(request: EmscriptenRequest) -> EmscriptenResponse:
|
||||||
|
if has_jspi():
|
||||||
|
return send_jspi_request(request, False)
|
||||||
|
elif is_in_node():
|
||||||
|
raise _RequestError(
|
||||||
|
message=NODE_JSPI_ERROR,
|
||||||
|
request=request,
|
||||||
|
response=None,
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
js_xhr = js.XMLHttpRequest.new()
|
||||||
|
|
||||||
|
if not is_in_browser_main_thread():
|
||||||
|
js_xhr.responseType = "arraybuffer"
|
||||||
|
if request.timeout:
|
||||||
|
js_xhr.timeout = int(request.timeout * 1000)
|
||||||
|
else:
|
||||||
|
js_xhr.overrideMimeType("text/plain; charset=ISO-8859-15")
|
||||||
|
if request.timeout:
|
||||||
|
# timeout isn't available on the main thread - show a warning in console
|
||||||
|
# if it is set
|
||||||
|
_show_timeout_warning()
|
||||||
|
|
||||||
|
js_xhr.open(request.method, request.url, False)
|
||||||
|
for name, value in request.headers.items():
|
||||||
|
if name.lower() not in HEADERS_TO_IGNORE:
|
||||||
|
js_xhr.setRequestHeader(name, value)
|
||||||
|
|
||||||
|
js_xhr.send(to_js(request.body))
|
||||||
|
|
||||||
|
headers = dict(Parser().parsestr(js_xhr.getAllResponseHeaders()))
|
||||||
|
|
||||||
|
if not is_in_browser_main_thread():
|
||||||
|
body = js_xhr.response.to_py().tobytes()
|
||||||
|
else:
|
||||||
|
body = js_xhr.response.encode("ISO-8859-15")
|
||||||
|
return EmscriptenResponse(
|
||||||
|
status_code=js_xhr.status, headers=headers, body=body, request=request
|
||||||
|
)
|
||||||
|
except JsException as err:
|
||||||
|
if err.name == "TimeoutError":
|
||||||
|
raise _TimeoutError(err.message, request=request)
|
||||||
|
elif err.name == "NetworkError":
|
||||||
|
raise _RequestError(err.message, request=request)
|
||||||
|
else:
|
||||||
|
# general http error
|
||||||
|
raise _RequestError(err.message, request=request)
|
||||||
|
|
||||||
|
|
||||||
|
def send_jspi_request(
|
||||||
|
request: EmscriptenRequest, streaming: bool
|
||||||
|
) -> EmscriptenResponse:
|
||||||
|
"""
|
||||||
|
Send a request using WebAssembly JavaScript Promise Integration
|
||||||
|
to wrap the asynchronous JavaScript fetch api (experimental).
|
||||||
|
|
||||||
|
:param request:
|
||||||
|
Request to send
|
||||||
|
|
||||||
|
:param streaming:
|
||||||
|
Whether to stream the response
|
||||||
|
|
||||||
|
:return: The response object
|
||||||
|
:rtype: EmscriptenResponse
|
||||||
|
"""
|
||||||
|
timeout = request.timeout
|
||||||
|
js_abort_controller = js.AbortController.new()
|
||||||
|
headers = {k: v for k, v in request.headers.items() if k not in HEADERS_TO_IGNORE}
|
||||||
|
req_body = request.body
|
||||||
|
fetch_data = {
|
||||||
|
"headers": headers,
|
||||||
|
"body": to_js(req_body),
|
||||||
|
"method": request.method,
|
||||||
|
"signal": js_abort_controller.signal,
|
||||||
|
}
|
||||||
|
# Node.js returns the whole response (unlike opaqueredirect in browsers),
|
||||||
|
# so urllib3 can set `redirect: manual` to control redirects itself.
|
||||||
|
# https://stackoverflow.com/a/78524615
|
||||||
|
if _is_node_js():
|
||||||
|
fetch_data["redirect"] = "manual"
|
||||||
|
# Call JavaScript fetch (async api, returns a promise)
|
||||||
|
fetcher_promise_js = js.fetch(request.url, _obj_from_dict(fetch_data))
|
||||||
|
# Now suspend WebAssembly until we resolve that promise
|
||||||
|
# or time out.
|
||||||
|
response_js = _run_sync_with_timeout(
|
||||||
|
fetcher_promise_js,
|
||||||
|
timeout,
|
||||||
|
js_abort_controller,
|
||||||
|
request=request,
|
||||||
|
response=None,
|
||||||
|
)
|
||||||
|
headers = {}
|
||||||
|
header_iter = response_js.headers.entries()
|
||||||
|
while True:
|
||||||
|
iter_value_js = header_iter.next()
|
||||||
|
if getattr(iter_value_js, "done", False):
|
||||||
|
break
|
||||||
|
else:
|
||||||
|
headers[str(iter_value_js.value[0])] = str(iter_value_js.value[1])
|
||||||
|
status_code = response_js.status
|
||||||
|
body: bytes | io.RawIOBase = b""
|
||||||
|
|
||||||
|
response = EmscriptenResponse(
|
||||||
|
status_code=status_code, headers=headers, body=b"", request=request
|
||||||
|
)
|
||||||
|
if streaming:
|
||||||
|
# get via inputstream
|
||||||
|
if response_js.body is not None:
|
||||||
|
# get a reader from the fetch response
|
||||||
|
body_stream_js = response_js.body.getReader()
|
||||||
|
body = _JSPIReadStream(
|
||||||
|
body_stream_js, timeout, request, response, js_abort_controller
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# get directly via arraybuffer
|
||||||
|
# n.b. this is another async JavaScript call.
|
||||||
|
body = _run_sync_with_timeout(
|
||||||
|
response_js.arrayBuffer(),
|
||||||
|
timeout,
|
||||||
|
js_abort_controller,
|
||||||
|
request=request,
|
||||||
|
response=response,
|
||||||
|
).to_py()
|
||||||
|
response.body = body
|
||||||
|
return response
|
||||||
|
|
||||||
|
|
||||||
|
def _run_sync_with_timeout(
|
||||||
|
promise: Any,
|
||||||
|
timeout: float,
|
||||||
|
js_abort_controller: Any,
|
||||||
|
request: EmscriptenRequest | None,
|
||||||
|
response: EmscriptenResponse | None,
|
||||||
|
) -> Any:
|
||||||
|
"""
|
||||||
|
Await a JavaScript promise synchronously with a timeout which is implemented
|
||||||
|
via the AbortController
|
||||||
|
|
||||||
|
:param promise:
|
||||||
|
Javascript promise to await
|
||||||
|
|
||||||
|
:param timeout:
|
||||||
|
Timeout in seconds
|
||||||
|
|
||||||
|
:param js_abort_controller:
|
||||||
|
A JavaScript AbortController object, used on timeout
|
||||||
|
|
||||||
|
:param request:
|
||||||
|
The request being handled
|
||||||
|
|
||||||
|
:param response:
|
||||||
|
The response being handled (if it exists yet)
|
||||||
|
|
||||||
|
:raises _TimeoutError: If the request times out
|
||||||
|
:raises _RequestError: If the request raises a JavaScript exception
|
||||||
|
|
||||||
|
:return: The result of awaiting the promise.
|
||||||
|
"""
|
||||||
|
timer_id = None
|
||||||
|
if timeout > 0:
|
||||||
|
timer_id = js.setTimeout(
|
||||||
|
js_abort_controller.abort.bind(js_abort_controller), int(timeout * 1000)
|
||||||
|
)
|
||||||
|
try:
|
||||||
|
from pyodide.ffi import run_sync
|
||||||
|
|
||||||
|
# run_sync here uses WebAssembly JavaScript Promise Integration to
|
||||||
|
# suspend python until the JavaScript promise resolves.
|
||||||
|
return run_sync(promise)
|
||||||
|
except JsException as err:
|
||||||
|
if err.name == "AbortError":
|
||||||
|
raise _TimeoutError(
|
||||||
|
message="Request timed out", request=request, response=response
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
raise _RequestError(message=err.message, request=request, response=response)
|
||||||
|
finally:
|
||||||
|
if timer_id is not None:
|
||||||
|
js.clearTimeout(timer_id)
|
||||||
|
|
||||||
|
|
||||||
|
def has_jspi() -> bool:
|
||||||
|
"""
|
||||||
|
Return true if jspi can be used.
|
||||||
|
|
||||||
|
This requires both browser support and also WebAssembly
|
||||||
|
to be in the correct state - i.e. that the javascript
|
||||||
|
call into python was async not sync.
|
||||||
|
|
||||||
|
:return: True if jspi can be used.
|
||||||
|
:rtype: bool
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
from pyodide.ffi import can_run_sync, run_sync # noqa: F401
|
||||||
|
|
||||||
|
return bool(can_run_sync())
|
||||||
|
except ImportError:
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def _is_node_js() -> bool:
|
||||||
|
"""
|
||||||
|
Check if we are in Node.js.
|
||||||
|
|
||||||
|
:return: True if we are in Node.js.
|
||||||
|
:rtype: bool
|
||||||
|
"""
|
||||||
|
return (
|
||||||
|
hasattr(js, "process")
|
||||||
|
and hasattr(js.process, "release")
|
||||||
|
# According to the Node.js documentation, the release name is always "node".
|
||||||
|
and js.process.release.name == "node"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def streaming_ready() -> bool | None:
|
||||||
|
if _fetcher:
|
||||||
|
return _fetcher.streaming_ready
|
||||||
|
else:
|
||||||
|
return None # no fetcher, return None to signify that
|
||||||
|
|
||||||
|
|
||||||
|
async def wait_for_streaming_ready() -> bool:
|
||||||
|
if _fetcher:
|
||||||
|
await _fetcher.js_worker_ready_promise
|
||||||
|
return True
|
||||||
|
else:
|
||||||
|
return False
|
||||||
+22
@@ -0,0 +1,22 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
from dataclasses import dataclass, field
|
||||||
|
|
||||||
|
from ..._base_connection import _TYPE_BODY
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class EmscriptenRequest:
|
||||||
|
method: str
|
||||||
|
url: str
|
||||||
|
params: dict[str, str] | None = None
|
||||||
|
body: _TYPE_BODY | None = None
|
||||||
|
headers: dict[str, str] = field(default_factory=dict)
|
||||||
|
timeout: float = 0
|
||||||
|
decode_content: bool = True
|
||||||
|
|
||||||
|
def set_header(self, name: str, value: str) -> None:
|
||||||
|
self.headers[name.capitalize()] = value
|
||||||
|
|
||||||
|
def set_body(self, body: _TYPE_BODY | None) -> None:
|
||||||
|
self.body = body
|
||||||
+281
@@ -0,0 +1,281 @@
|
|||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import json as _json
|
||||||
|
import logging
|
||||||
|
import typing
|
||||||
|
from contextlib import contextmanager
|
||||||
|
from dataclasses import dataclass
|
||||||
|
from http.client import HTTPException as HTTPException
|
||||||
|
from io import BytesIO, IOBase
|
||||||
|
|
||||||
|
from ...exceptions import InvalidHeader, TimeoutError
|
||||||
|
from ...response import BaseHTTPResponse
|
||||||
|
from ...util.retry import Retry
|
||||||
|
from .request import EmscriptenRequest
|
||||||
|
|
||||||
|
if typing.TYPE_CHECKING:
|
||||||
|
from ..._base_connection import BaseHTTPConnection, BaseHTTPSConnection
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class EmscriptenResponse:
|
||||||
|
status_code: int
|
||||||
|
headers: dict[str, str]
|
||||||
|
body: IOBase | bytes
|
||||||
|
request: EmscriptenRequest
|
||||||
|
|
||||||
|
|
||||||
|
class EmscriptenHttpResponseWrapper(BaseHTTPResponse):
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
internal_response: EmscriptenResponse,
|
||||||
|
url: str | None = None,
|
||||||
|
connection: BaseHTTPConnection | BaseHTTPSConnection | None = None,
|
||||||
|
):
|
||||||
|
self._pool = None # set by pool class
|
||||||
|
self._body = None
|
||||||
|
self._uncached_read_occurred = False
|
||||||
|
self._response = internal_response
|
||||||
|
self._url = url
|
||||||
|
self._connection = connection
|
||||||
|
self._closed = False
|
||||||
|
super().__init__(
|
||||||
|
headers=internal_response.headers,
|
||||||
|
status=internal_response.status_code,
|
||||||
|
request_url=url,
|
||||||
|
version=0,
|
||||||
|
version_string="HTTP/?",
|
||||||
|
reason="",
|
||||||
|
decode_content=True,
|
||||||
|
)
|
||||||
|
self.length_remaining = self._init_length(self._response.request.method)
|
||||||
|
self.length_is_certain = False
|
||||||
|
|
||||||
|
@property
|
||||||
|
def url(self) -> str | None:
|
||||||
|
return self._url
|
||||||
|
|
||||||
|
@url.setter
|
||||||
|
def url(self, url: str | None) -> None:
|
||||||
|
self._url = url
|
||||||
|
|
||||||
|
@property
|
||||||
|
def connection(self) -> BaseHTTPConnection | BaseHTTPSConnection | None:
|
||||||
|
return self._connection
|
||||||
|
|
||||||
|
@property
|
||||||
|
def retries(self) -> Retry | None:
|
||||||
|
return self._retries
|
||||||
|
|
||||||
|
@retries.setter
|
||||||
|
def retries(self, retries: Retry | None) -> None:
|
||||||
|
# Override the request_url if retries has a redirect location.
|
||||||
|
self._retries = retries
|
||||||
|
|
||||||
|
def stream(
|
||||||
|
self, amt: int | None = 2**16, decode_content: bool | None = None
|
||||||
|
) -> typing.Generator[bytes]:
|
||||||
|
"""
|
||||||
|
A generator wrapper for the read() method. A call will block until
|
||||||
|
``amt`` bytes have been read from the connection or until the
|
||||||
|
connection is closed.
|
||||||
|
|
||||||
|
:param amt:
|
||||||
|
How much of the content to read. The generator will return up to
|
||||||
|
much data per iteration, but may return less. This is particularly
|
||||||
|
likely when using compressed data. However, the empty string will
|
||||||
|
never be returned.
|
||||||
|
|
||||||
|
:param decode_content:
|
||||||
|
If True, will attempt to decode the body based on the
|
||||||
|
'content-encoding' header.
|
||||||
|
"""
|
||||||
|
while True:
|
||||||
|
data = self.read(amt=amt, decode_content=decode_content)
|
||||||
|
|
||||||
|
if data:
|
||||||
|
yield data
|
||||||
|
else:
|
||||||
|
break
|
||||||
|
|
||||||
|
def _init_length(self, request_method: str | None) -> int | None:
|
||||||
|
length: int | None
|
||||||
|
content_length: str | None = self.headers.get("content-length")
|
||||||
|
|
||||||
|
if content_length is not None:
|
||||||
|
try:
|
||||||
|
# RFC 7230 section 3.3.2 specifies multiple content lengths can
|
||||||
|
# be sent in a single Content-Length header
|
||||||
|
# (e.g. Content-Length: 42, 42). This line ensures the values
|
||||||
|
# are all valid ints and that as long as the `set` length is 1,
|
||||||
|
# all values are the same. Otherwise, the header is invalid.
|
||||||
|
lengths = {int(val) for val in content_length.split(",")}
|
||||||
|
if len(lengths) > 1:
|
||||||
|
raise InvalidHeader(
|
||||||
|
"Content-Length contained multiple "
|
||||||
|
"unmatching values (%s)" % content_length
|
||||||
|
)
|
||||||
|
length = lengths.pop()
|
||||||
|
except ValueError:
|
||||||
|
length = None
|
||||||
|
else:
|
||||||
|
if length < 0:
|
||||||
|
length = None
|
||||||
|
|
||||||
|
else: # if content_length is None
|
||||||
|
length = None
|
||||||
|
|
||||||
|
# Check for responses that shouldn't include a body
|
||||||
|
if (
|
||||||
|
self.status in (204, 304)
|
||||||
|
or 100 <= self.status < 200
|
||||||
|
or request_method == "HEAD"
|
||||||
|
):
|
||||||
|
length = 0
|
||||||
|
|
||||||
|
return length
|
||||||
|
|
||||||
|
def read(
|
||||||
|
self,
|
||||||
|
amt: int | None = None,
|
||||||
|
decode_content: bool | None = None, # ignored because browser decodes always
|
||||||
|
cache_content: bool = False,
|
||||||
|
) -> bytes:
|
||||||
|
if (
|
||||||
|
self._closed
|
||||||
|
or self._response is None
|
||||||
|
or (isinstance(self._response.body, IOBase) and self._response.body.closed)
|
||||||
|
):
|
||||||
|
return b""
|
||||||
|
|
||||||
|
with self._error_catcher():
|
||||||
|
# body has been preloaded as a string by XmlHttpRequest
|
||||||
|
if not isinstance(self._response.body, IOBase):
|
||||||
|
self.length_remaining = len(self._response.body)
|
||||||
|
self.length_is_certain = True
|
||||||
|
# wrap body in IOStream
|
||||||
|
self._response.body = BytesIO(self._response.body)
|
||||||
|
if amt is not None and amt >= 0:
|
||||||
|
# don't cache partial content
|
||||||
|
cache_content = False
|
||||||
|
data = self._response.body.read(amt)
|
||||||
|
self._uncached_read_occurred = True
|
||||||
|
else: # read all we can (and cache it)
|
||||||
|
data = self._response.body.read()
|
||||||
|
if cache_content and not self._uncached_read_occurred:
|
||||||
|
self._body = data
|
||||||
|
else:
|
||||||
|
self._uncached_read_occurred = True
|
||||||
|
if self.length_remaining is not None:
|
||||||
|
self.length_remaining = max(self.length_remaining - len(data), 0)
|
||||||
|
if len(data) == 0 or (
|
||||||
|
self.length_is_certain and self.length_remaining == 0
|
||||||
|
):
|
||||||
|
# definitely finished reading, close response stream
|
||||||
|
self._response.body.close()
|
||||||
|
return typing.cast(bytes, data)
|
||||||
|
|
||||||
|
def read_chunked(
|
||||||
|
self,
|
||||||
|
amt: int | None = None,
|
||||||
|
decode_content: bool | None = None,
|
||||||
|
) -> typing.Generator[bytes]:
|
||||||
|
# chunked is handled by browser
|
||||||
|
while True:
|
||||||
|
bytes = self.read(amt, decode_content)
|
||||||
|
if not bytes:
|
||||||
|
break
|
||||||
|
yield bytes
|
||||||
|
|
||||||
|
def release_conn(self) -> None:
|
||||||
|
if not self._pool or not self._connection:
|
||||||
|
return None
|
||||||
|
|
||||||
|
self._pool._put_conn(self._connection)
|
||||||
|
self._connection = None
|
||||||
|
|
||||||
|
def drain_conn(self) -> None:
|
||||||
|
self.close()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def data(self) -> bytes:
|
||||||
|
if self._body:
|
||||||
|
return self._body
|
||||||
|
else:
|
||||||
|
return self.read(cache_content=True)
|
||||||
|
|
||||||
|
def json(self) -> typing.Any:
|
||||||
|
"""
|
||||||
|
Deserializes the body of the HTTP response as a Python object.
|
||||||
|
|
||||||
|
The body of the HTTP response must be encoded using UTF-8, as per
|
||||||
|
`RFC 8529 Section 8.1 <https://www.rfc-editor.org/rfc/rfc8259#section-8.1>`_.
|
||||||
|
|
||||||
|
To use a custom JSON decoder pass the result of :attr:`HTTPResponse.data` to
|
||||||
|
your custom decoder instead.
|
||||||
|
|
||||||
|
If the body of the HTTP response is not decodable to UTF-8, a
|
||||||
|
`UnicodeDecodeError` will be raised. If the body of the HTTP response is not a
|
||||||
|
valid JSON document, a `json.JSONDecodeError` will be raised.
|
||||||
|
|
||||||
|
Read more :ref:`here <json_content>`.
|
||||||
|
|
||||||
|
:returns: The body of the HTTP response as a Python object.
|
||||||
|
"""
|
||||||
|
data = self.data.decode("utf-8")
|
||||||
|
return _json.loads(data)
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
if not self._closed:
|
||||||
|
if isinstance(self._response.body, IOBase):
|
||||||
|
self._response.body.close()
|
||||||
|
if self._connection:
|
||||||
|
self._connection.close()
|
||||||
|
self._connection = None
|
||||||
|
self._closed = True
|
||||||
|
|
||||||
|
@contextmanager
|
||||||
|
def _error_catcher(self) -> typing.Generator[None]:
|
||||||
|
"""
|
||||||
|
Catch Emscripten specific exceptions thrown by fetch.py,
|
||||||
|
instead re-raising urllib3 variants, so that low-level exceptions
|
||||||
|
are not leaked in the high-level api.
|
||||||
|
|
||||||
|
On exit, release the connection back to the pool.
|
||||||
|
"""
|
||||||
|
from .fetch import _RequestError, _TimeoutError # avoid circular import
|
||||||
|
|
||||||
|
clean_exit = False
|
||||||
|
|
||||||
|
try:
|
||||||
|
yield
|
||||||
|
# If no exception is thrown, we should avoid cleaning up
|
||||||
|
# unnecessarily.
|
||||||
|
clean_exit = True
|
||||||
|
except _TimeoutError as e:
|
||||||
|
raise TimeoutError(str(e))
|
||||||
|
except _RequestError as e:
|
||||||
|
raise HTTPException(str(e))
|
||||||
|
finally:
|
||||||
|
# If we didn't terminate cleanly, we need to throw away our
|
||||||
|
# connection.
|
||||||
|
if not clean_exit:
|
||||||
|
# The response may not be closed but we're not going to use it
|
||||||
|
# anymore so close it now
|
||||||
|
if (
|
||||||
|
isinstance(self._response.body, IOBase)
|
||||||
|
and not self._response.body.closed
|
||||||
|
):
|
||||||
|
self._response.body.close()
|
||||||
|
# release the connection back to the pool
|
||||||
|
self.release_conn()
|
||||||
|
else:
|
||||||
|
# If we have read everything from the response stream,
|
||||||
|
# return the connection back to the pool.
|
||||||
|
if (
|
||||||
|
isinstance(self._response.body, IOBase)
|
||||||
|
and self._response.body.closed
|
||||||
|
):
|
||||||
|
self.release_conn()
|
||||||
+563
@@ -0,0 +1,563 @@
|
|||||||
|
"""
|
||||||
|
Module for using pyOpenSSL as a TLS backend. This module was relevant before
|
||||||
|
the standard library ``ssl`` module supported SNI, but now that we've dropped
|
||||||
|
support for Python 2.7 all relevant Python versions support SNI so
|
||||||
|
**this module is no longer recommended**.
|
||||||
|
|
||||||
|
This needs the following packages installed:
|
||||||
|
|
||||||
|
* `pyOpenSSL`_ (tested with 19.0.0)
|
||||||
|
* `cryptography`_ (minimum 2.3, from pyopenssl)
|
||||||
|
* `idna`_ (minimum 2.1, from cryptography)
|
||||||
|
|
||||||
|
However, pyOpenSSL depends on cryptography, so while we use all three directly here we
|
||||||
|
end up having relatively few packages required.
|
||||||
|
|
||||||
|
You can install them with the following command:
|
||||||
|
|
||||||
|
.. code-block:: bash
|
||||||
|
|
||||||
|
$ python -m pip install pyopenssl cryptography idna
|
||||||
|
|
||||||
|
To activate certificate checking, call
|
||||||
|
:func:`~urllib3.contrib.pyopenssl.inject_into_urllib3` from your Python code
|
||||||
|
before you begin making HTTP requests. This can be done in a ``sitecustomize``
|
||||||
|
module, or at any other time before your application begins using ``urllib3``,
|
||||||
|
like this:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
try:
|
||||||
|
import urllib3.contrib.pyopenssl
|
||||||
|
urllib3.contrib.pyopenssl.inject_into_urllib3()
|
||||||
|
except ImportError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
.. _pyopenssl: https://www.pyopenssl.org
|
||||||
|
.. _cryptography: https://cryptography.io
|
||||||
|
.. _idna: https://github.com/kjd/idna
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import OpenSSL.SSL # type: ignore[import-not-found]
|
||||||
|
from cryptography import x509
|
||||||
|
|
||||||
|
try:
|
||||||
|
from cryptography.x509 import UnsupportedExtension # type: ignore[attr-defined]
|
||||||
|
except ImportError:
|
||||||
|
# UnsupportedExtension is gone in cryptography >= 2.1.0
|
||||||
|
class UnsupportedExtension(Exception): # type: ignore[no-redef]
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
import logging
|
||||||
|
import ssl
|
||||||
|
import typing
|
||||||
|
from io import BytesIO
|
||||||
|
from socket import socket as socket_cls
|
||||||
|
|
||||||
|
from .. import util
|
||||||
|
|
||||||
|
if typing.TYPE_CHECKING:
|
||||||
|
from OpenSSL.crypto import X509 # type: ignore[import-not-found]
|
||||||
|
|
||||||
|
|
||||||
|
__all__ = ["inject_into_urllib3", "extract_from_urllib3"]
|
||||||
|
|
||||||
|
# Map from urllib3 to PyOpenSSL compatible parameter-values.
|
||||||
|
_openssl_versions: dict[int, int] = {
|
||||||
|
util.ssl_.PROTOCOL_TLS: OpenSSL.SSL.SSLv23_METHOD, # type: ignore[attr-defined]
|
||||||
|
util.ssl_.PROTOCOL_TLS_CLIENT: OpenSSL.SSL.SSLv23_METHOD, # type: ignore[attr-defined]
|
||||||
|
ssl.PROTOCOL_TLSv1: OpenSSL.SSL.TLSv1_METHOD,
|
||||||
|
}
|
||||||
|
|
||||||
|
if hasattr(ssl, "PROTOCOL_TLSv1_1") and hasattr(OpenSSL.SSL, "TLSv1_1_METHOD"):
|
||||||
|
_openssl_versions[ssl.PROTOCOL_TLSv1_1] = OpenSSL.SSL.TLSv1_1_METHOD
|
||||||
|
|
||||||
|
if hasattr(ssl, "PROTOCOL_TLSv1_2") and hasattr(OpenSSL.SSL, "TLSv1_2_METHOD"):
|
||||||
|
_openssl_versions[ssl.PROTOCOL_TLSv1_2] = OpenSSL.SSL.TLSv1_2_METHOD
|
||||||
|
|
||||||
|
|
||||||
|
_stdlib_to_openssl_verify = {
|
||||||
|
ssl.CERT_NONE: OpenSSL.SSL.VERIFY_NONE,
|
||||||
|
ssl.CERT_OPTIONAL: OpenSSL.SSL.VERIFY_PEER,
|
||||||
|
ssl.CERT_REQUIRED: OpenSSL.SSL.VERIFY_PEER
|
||||||
|
+ OpenSSL.SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
|
||||||
|
}
|
||||||
|
_openssl_to_stdlib_verify = {v: k for k, v in _stdlib_to_openssl_verify.items()}
|
||||||
|
|
||||||
|
# The SSLvX values are the most likely to be missing in the future
|
||||||
|
# but we check them all just to be sure.
|
||||||
|
_OP_NO_SSLv2_OR_SSLv3: int = getattr(OpenSSL.SSL, "OP_NO_SSLv2", 0) | getattr(
|
||||||
|
OpenSSL.SSL, "OP_NO_SSLv3", 0
|
||||||
|
)
|
||||||
|
_OP_NO_TLSv1: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1", 0)
|
||||||
|
_OP_NO_TLSv1_1: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1_1", 0)
|
||||||
|
_OP_NO_TLSv1_2: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1_2", 0)
|
||||||
|
_OP_NO_TLSv1_3: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1_3", 0)
|
||||||
|
|
||||||
|
_openssl_to_ssl_minimum_version: dict[int, int] = {
|
||||||
|
ssl.TLSVersion.MINIMUM_SUPPORTED: _OP_NO_SSLv2_OR_SSLv3,
|
||||||
|
ssl.TLSVersion.TLSv1: _OP_NO_SSLv2_OR_SSLv3,
|
||||||
|
ssl.TLSVersion.TLSv1_1: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1,
|
||||||
|
ssl.TLSVersion.TLSv1_2: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1 | _OP_NO_TLSv1_1,
|
||||||
|
ssl.TLSVersion.TLSv1_3: (
|
||||||
|
_OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1 | _OP_NO_TLSv1_1 | _OP_NO_TLSv1_2
|
||||||
|
),
|
||||||
|
ssl.TLSVersion.MAXIMUM_SUPPORTED: (
|
||||||
|
_OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1 | _OP_NO_TLSv1_1 | _OP_NO_TLSv1_2
|
||||||
|
),
|
||||||
|
}
|
||||||
|
_openssl_to_ssl_maximum_version: dict[int, int] = {
|
||||||
|
ssl.TLSVersion.MINIMUM_SUPPORTED: (
|
||||||
|
_OP_NO_SSLv2_OR_SSLv3
|
||||||
|
| _OP_NO_TLSv1
|
||||||
|
| _OP_NO_TLSv1_1
|
||||||
|
| _OP_NO_TLSv1_2
|
||||||
|
| _OP_NO_TLSv1_3
|
||||||
|
),
|
||||||
|
ssl.TLSVersion.TLSv1: (
|
||||||
|
_OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1_1 | _OP_NO_TLSv1_2 | _OP_NO_TLSv1_3
|
||||||
|
),
|
||||||
|
ssl.TLSVersion.TLSv1_1: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1_2 | _OP_NO_TLSv1_3,
|
||||||
|
ssl.TLSVersion.TLSv1_2: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1_3,
|
||||||
|
ssl.TLSVersion.TLSv1_3: _OP_NO_SSLv2_OR_SSLv3,
|
||||||
|
ssl.TLSVersion.MAXIMUM_SUPPORTED: _OP_NO_SSLv2_OR_SSLv3,
|
||||||
|
}
|
||||||
|
|
||||||
|
# OpenSSL will only write 16K at a time
|
||||||
|
SSL_WRITE_BLOCKSIZE = 16384
|
||||||
|
|
||||||
|
orig_util_SSLContext = util.ssl_.SSLContext
|
||||||
|
|
||||||
|
|
||||||
|
log = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
def inject_into_urllib3() -> None:
|
||||||
|
"Monkey-patch urllib3 with PyOpenSSL-backed SSL-support."
|
||||||
|
|
||||||
|
_validate_dependencies_met()
|
||||||
|
|
||||||
|
util.SSLContext = PyOpenSSLContext # type: ignore[assignment]
|
||||||
|
util.ssl_.SSLContext = PyOpenSSLContext # type: ignore[assignment]
|
||||||
|
util.IS_PYOPENSSL = True
|
||||||
|
util.ssl_.IS_PYOPENSSL = True
|
||||||
|
|
||||||
|
|
||||||
|
def extract_from_urllib3() -> None:
|
||||||
|
"Undo monkey-patching by :func:`inject_into_urllib3`."
|
||||||
|
|
||||||
|
util.SSLContext = orig_util_SSLContext
|
||||||
|
util.ssl_.SSLContext = orig_util_SSLContext
|
||||||
|
util.IS_PYOPENSSL = False
|
||||||
|
util.ssl_.IS_PYOPENSSL = False
|
||||||
|
|
||||||
|
|
||||||
|
def _validate_dependencies_met() -> None:
|
||||||
|
"""
|
||||||
|
Verifies that PyOpenSSL's package-level dependencies have been met.
|
||||||
|
Throws `ImportError` if they are not met.
|
||||||
|
"""
|
||||||
|
# Method added in `cryptography==1.1`; not available in older versions
|
||||||
|
from cryptography.x509.extensions import Extensions
|
||||||
|
|
||||||
|
if getattr(Extensions, "get_extension_for_class", None) is None:
|
||||||
|
raise ImportError(
|
||||||
|
"'cryptography' module missing required functionality. "
|
||||||
|
"Try upgrading to v1.3.4 or newer."
|
||||||
|
)
|
||||||
|
|
||||||
|
# pyOpenSSL 0.14 and above use cryptography for OpenSSL bindings. The _x509
|
||||||
|
# attribute is only present on those versions.
|
||||||
|
from OpenSSL.crypto import X509
|
||||||
|
|
||||||
|
x509 = X509()
|
||||||
|
if getattr(x509, "_x509", None) is None:
|
||||||
|
raise ImportError(
|
||||||
|
"'pyOpenSSL' module missing required functionality. "
|
||||||
|
"Try upgrading to v0.14 or newer."
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def _dnsname_to_stdlib(name: str) -> str | None:
|
||||||
|
"""
|
||||||
|
Converts a dNSName SubjectAlternativeName field to the form used by the
|
||||||
|
standard library on the given Python version.
|
||||||
|
|
||||||
|
Cryptography produces a dNSName as a unicode string that was idna-decoded
|
||||||
|
from ASCII bytes. We need to idna-encode that string to get it back, and
|
||||||
|
then on Python 3 we also need to convert to unicode via UTF-8 (the stdlib
|
||||||
|
uses PyUnicode_FromStringAndSize on it, which decodes via UTF-8).
|
||||||
|
|
||||||
|
If the name cannot be idna-encoded then we return None signalling that
|
||||||
|
the name given should be skipped.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def idna_encode(name: str) -> bytes | None:
|
||||||
|
"""
|
||||||
|
Borrowed wholesale from the Python Cryptography Project. It turns out
|
||||||
|
that we can't just safely call `idna.encode`: it can explode for
|
||||||
|
wildcard names. This avoids that problem.
|
||||||
|
"""
|
||||||
|
import idna
|
||||||
|
|
||||||
|
try:
|
||||||
|
for prefix in ["*.", "."]:
|
||||||
|
if name.startswith(prefix):
|
||||||
|
name = name[len(prefix) :]
|
||||||
|
return prefix.encode("ascii") + idna.encode(name)
|
||||||
|
return idna.encode(name)
|
||||||
|
except idna.core.IDNAError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Don't send IPv6 addresses through the IDNA encoder.
|
||||||
|
if ":" in name:
|
||||||
|
return name
|
||||||
|
|
||||||
|
encoded_name = idna_encode(name)
|
||||||
|
if encoded_name is None:
|
||||||
|
return None
|
||||||
|
return encoded_name.decode("utf-8")
|
||||||
|
|
||||||
|
|
||||||
|
def get_subj_alt_name(peer_cert: X509) -> list[tuple[str, str]]:
|
||||||
|
"""
|
||||||
|
Given an PyOpenSSL certificate, provides all the subject alternative names.
|
||||||
|
"""
|
||||||
|
cert = peer_cert.to_cryptography()
|
||||||
|
|
||||||
|
# We want to find the SAN extension. Ask Cryptography to locate it (it's
|
||||||
|
# faster than looping in Python)
|
||||||
|
try:
|
||||||
|
ext = cert.extensions.get_extension_for_class(x509.SubjectAlternativeName).value
|
||||||
|
except x509.ExtensionNotFound:
|
||||||
|
# No such extension, return the empty list.
|
||||||
|
return []
|
||||||
|
except (
|
||||||
|
x509.DuplicateExtension,
|
||||||
|
UnsupportedExtension,
|
||||||
|
x509.UnsupportedGeneralNameType,
|
||||||
|
UnicodeError,
|
||||||
|
) as e:
|
||||||
|
# A problem has been found with the quality of the certificate. Assume
|
||||||
|
# no SAN field is present.
|
||||||
|
log.warning(
|
||||||
|
"A problem was encountered with the certificate that prevented "
|
||||||
|
"urllib3 from finding the SubjectAlternativeName field. This can "
|
||||||
|
"affect certificate validation. The error was %s",
|
||||||
|
e,
|
||||||
|
)
|
||||||
|
return []
|
||||||
|
|
||||||
|
# We want to return dNSName and iPAddress fields. We need to cast the IPs
|
||||||
|
# back to strings because the match_hostname function wants them as
|
||||||
|
# strings.
|
||||||
|
# Sadly the DNS names need to be idna encoded and then, on Python 3, UTF-8
|
||||||
|
# decoded. This is pretty frustrating, but that's what the standard library
|
||||||
|
# does with certificates, and so we need to attempt to do the same.
|
||||||
|
# We also want to skip over names which cannot be idna encoded.
|
||||||
|
names = [
|
||||||
|
("DNS", name)
|
||||||
|
for name in map(_dnsname_to_stdlib, ext.get_values_for_type(x509.DNSName))
|
||||||
|
if name is not None
|
||||||
|
]
|
||||||
|
names.extend(
|
||||||
|
("IP Address", str(name)) for name in ext.get_values_for_type(x509.IPAddress)
|
||||||
|
)
|
||||||
|
|
||||||
|
return names
|
||||||
|
|
||||||
|
|
||||||
|
class WrappedSocket:
|
||||||
|
"""API-compatibility wrapper for Python OpenSSL's Connection-class."""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
connection: OpenSSL.SSL.Connection,
|
||||||
|
socket: socket_cls,
|
||||||
|
suppress_ragged_eofs: bool = True,
|
||||||
|
) -> None:
|
||||||
|
self.connection = connection
|
||||||
|
self.socket = socket
|
||||||
|
self.suppress_ragged_eofs = suppress_ragged_eofs
|
||||||
|
self._io_refs = 0
|
||||||
|
self._closed = False
|
||||||
|
|
||||||
|
def fileno(self) -> int:
|
||||||
|
return self.socket.fileno()
|
||||||
|
|
||||||
|
# Copy-pasted from Python 3.5 source code
|
||||||
|
def _decref_socketios(self) -> None:
|
||||||
|
if self._io_refs > 0:
|
||||||
|
self._io_refs -= 1
|
||||||
|
if self._closed:
|
||||||
|
self.close()
|
||||||
|
|
||||||
|
def recv(self, *args: typing.Any, **kwargs: typing.Any) -> bytes:
|
||||||
|
try:
|
||||||
|
data = self.connection.recv(*args, **kwargs)
|
||||||
|
except OpenSSL.SSL.SysCallError as e:
|
||||||
|
if self.suppress_ragged_eofs and e.args == (-1, "Unexpected EOF"):
|
||||||
|
return b""
|
||||||
|
else:
|
||||||
|
raise OSError(e.args[0], str(e)) from e
|
||||||
|
except OpenSSL.SSL.ZeroReturnError:
|
||||||
|
if self.connection.get_shutdown() == OpenSSL.SSL.RECEIVED_SHUTDOWN:
|
||||||
|
return b""
|
||||||
|
else:
|
||||||
|
raise
|
||||||
|
except OpenSSL.SSL.WantReadError as e:
|
||||||
|
if not util.wait_for_read(self.socket, self.socket.gettimeout()):
|
||||||
|
raise TimeoutError("The read operation timed out") from e
|
||||||
|
else:
|
||||||
|
return self.recv(*args, **kwargs)
|
||||||
|
|
||||||
|
# TLS 1.3 post-handshake authentication
|
||||||
|
except OpenSSL.SSL.Error as e:
|
||||||
|
raise ssl.SSLError(f"read error: {e!r}") from e
|
||||||
|
else:
|
||||||
|
return data # type: ignore[no-any-return]
|
||||||
|
|
||||||
|
def recv_into(self, *args: typing.Any, **kwargs: typing.Any) -> int:
|
||||||
|
try:
|
||||||
|
return self.connection.recv_into(*args, **kwargs) # type: ignore[no-any-return]
|
||||||
|
except OpenSSL.SSL.SysCallError as e:
|
||||||
|
if self.suppress_ragged_eofs and e.args == (-1, "Unexpected EOF"):
|
||||||
|
return 0
|
||||||
|
else:
|
||||||
|
raise OSError(e.args[0], str(e)) from e
|
||||||
|
except OpenSSL.SSL.ZeroReturnError:
|
||||||
|
if self.connection.get_shutdown() == OpenSSL.SSL.RECEIVED_SHUTDOWN:
|
||||||
|
return 0
|
||||||
|
else:
|
||||||
|
raise
|
||||||
|
except OpenSSL.SSL.WantReadError as e:
|
||||||
|
if not util.wait_for_read(self.socket, self.socket.gettimeout()):
|
||||||
|
raise TimeoutError("The read operation timed out") from e
|
||||||
|
else:
|
||||||
|
return self.recv_into(*args, **kwargs)
|
||||||
|
|
||||||
|
# TLS 1.3 post-handshake authentication
|
||||||
|
except OpenSSL.SSL.Error as e:
|
||||||
|
raise ssl.SSLError(f"read error: {e!r}") from e
|
||||||
|
|
||||||
|
def settimeout(self, timeout: float) -> None:
|
||||||
|
return self.socket.settimeout(timeout)
|
||||||
|
|
||||||
|
def _send_until_done(self, data: bytes) -> int:
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
return self.connection.send(data) # type: ignore[no-any-return]
|
||||||
|
except OpenSSL.SSL.WantWriteError as e:
|
||||||
|
if not util.wait_for_write(self.socket, self.socket.gettimeout()):
|
||||||
|
raise TimeoutError() from e
|
||||||
|
continue
|
||||||
|
except OpenSSL.SSL.SysCallError as e:
|
||||||
|
raise OSError(e.args[0], str(e)) from e
|
||||||
|
|
||||||
|
def sendall(self, data: bytes) -> None:
|
||||||
|
total_sent = 0
|
||||||
|
while total_sent < len(data):
|
||||||
|
sent = self._send_until_done(
|
||||||
|
data[total_sent : total_sent + SSL_WRITE_BLOCKSIZE]
|
||||||
|
)
|
||||||
|
total_sent += sent
|
||||||
|
|
||||||
|
def shutdown(self, how: int) -> None:
|
||||||
|
try:
|
||||||
|
self.connection.shutdown()
|
||||||
|
except OpenSSL.SSL.Error as e:
|
||||||
|
raise ssl.SSLError(f"shutdown error: {e!r}") from e
|
||||||
|
|
||||||
|
def close(self) -> None:
|
||||||
|
self._closed = True
|
||||||
|
if self._io_refs <= 0:
|
||||||
|
self._real_close()
|
||||||
|
|
||||||
|
def _real_close(self) -> None:
|
||||||
|
try:
|
||||||
|
return self.connection.close() # type: ignore[no-any-return]
|
||||||
|
except OpenSSL.SSL.Error:
|
||||||
|
return
|
||||||
|
|
||||||
|
def getpeercert(
|
||||||
|
self, binary_form: bool = False
|
||||||
|
) -> dict[str, list[typing.Any]] | None:
|
||||||
|
x509 = self.connection.get_peer_certificate()
|
||||||
|
|
||||||
|
if not x509:
|
||||||
|
return x509 # type: ignore[no-any-return]
|
||||||
|
|
||||||
|
if binary_form:
|
||||||
|
return OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_ASN1, x509) # type: ignore[no-any-return]
|
||||||
|
|
||||||
|
return {
|
||||||
|
"subject": ((("commonName", x509.get_subject().CN),),), # type: ignore[dict-item]
|
||||||
|
"subjectAltName": get_subj_alt_name(x509),
|
||||||
|
}
|
||||||
|
|
||||||
|
def version(self) -> str:
|
||||||
|
return self.connection.get_protocol_version_name() # type: ignore[no-any-return]
|
||||||
|
|
||||||
|
def selected_alpn_protocol(self) -> str | None:
|
||||||
|
alpn_proto = self.connection.get_alpn_proto_negotiated()
|
||||||
|
return alpn_proto.decode() if alpn_proto else None
|
||||||
|
|
||||||
|
|
||||||
|
WrappedSocket.makefile = socket_cls.makefile # type: ignore[attr-defined]
|
||||||
|
|
||||||
|
|
||||||
|
class PyOpenSSLContext:
|
||||||
|
"""
|
||||||
|
I am a wrapper class for the PyOpenSSL ``Context`` object. I am responsible
|
||||||
|
for translating the interface of the standard library ``SSLContext`` object
|
||||||
|
to calls into PyOpenSSL.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, protocol: int) -> None:
|
||||||
|
self.protocol = _openssl_versions[protocol]
|
||||||
|
self._ctx = OpenSSL.SSL.Context(self.protocol)
|
||||||
|
self._options = 0
|
||||||
|
self.check_hostname = False
|
||||||
|
self._minimum_version: int = ssl.TLSVersion.MINIMUM_SUPPORTED
|
||||||
|
self._maximum_version: int = ssl.TLSVersion.MAXIMUM_SUPPORTED
|
||||||
|
self._verify_flags: int = ssl.VERIFY_X509_TRUSTED_FIRST
|
||||||
|
|
||||||
|
@property
|
||||||
|
def options(self) -> int:
|
||||||
|
return self._options
|
||||||
|
|
||||||
|
@options.setter
|
||||||
|
def options(self, value: int) -> None:
|
||||||
|
self._options = value
|
||||||
|
self._set_ctx_options()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def verify_flags(self) -> int:
|
||||||
|
return self._verify_flags
|
||||||
|
|
||||||
|
@verify_flags.setter
|
||||||
|
def verify_flags(self, value: int) -> None:
|
||||||
|
self._verify_flags = value
|
||||||
|
self._ctx.get_cert_store().set_flags(self._verify_flags)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def verify_mode(self) -> int:
|
||||||
|
return _openssl_to_stdlib_verify[self._ctx.get_verify_mode()]
|
||||||
|
|
||||||
|
@verify_mode.setter
|
||||||
|
def verify_mode(self, value: ssl.VerifyMode) -> None:
|
||||||
|
self._ctx.set_verify(_stdlib_to_openssl_verify[value], _verify_callback)
|
||||||
|
|
||||||
|
def set_default_verify_paths(self) -> None:
|
||||||
|
self._ctx.set_default_verify_paths()
|
||||||
|
|
||||||
|
def set_ciphers(self, ciphers: bytes | str) -> None:
|
||||||
|
if isinstance(ciphers, str):
|
||||||
|
ciphers = ciphers.encode("utf-8")
|
||||||
|
self._ctx.set_cipher_list(ciphers)
|
||||||
|
|
||||||
|
def load_verify_locations(
|
||||||
|
self,
|
||||||
|
cafile: str | None = None,
|
||||||
|
capath: str | None = None,
|
||||||
|
cadata: bytes | None = None,
|
||||||
|
) -> None:
|
||||||
|
if cafile is not None:
|
||||||
|
cafile = cafile.encode("utf-8") # type: ignore[assignment]
|
||||||
|
if capath is not None:
|
||||||
|
capath = capath.encode("utf-8") # type: ignore[assignment]
|
||||||
|
try:
|
||||||
|
self._ctx.load_verify_locations(cafile, capath)
|
||||||
|
if cadata is not None:
|
||||||
|
self._ctx.load_verify_locations(BytesIO(cadata))
|
||||||
|
except OpenSSL.SSL.Error as e:
|
||||||
|
raise ssl.SSLError(f"unable to load trusted certificates: {e!r}") from e
|
||||||
|
|
||||||
|
def load_cert_chain(
|
||||||
|
self,
|
||||||
|
certfile: str,
|
||||||
|
keyfile: str | None = None,
|
||||||
|
password: str | None = None,
|
||||||
|
) -> None:
|
||||||
|
try:
|
||||||
|
self._ctx.use_certificate_chain_file(certfile)
|
||||||
|
if password is not None:
|
||||||
|
if not isinstance(password, bytes):
|
||||||
|
password = password.encode("utf-8") # type: ignore[assignment]
|
||||||
|
self._ctx.set_passwd_cb(lambda *_: password)
|
||||||
|
self._ctx.use_privatekey_file(keyfile or certfile)
|
||||||
|
except OpenSSL.SSL.Error as e:
|
||||||
|
raise ssl.SSLError(f"Unable to load certificate chain: {e!r}") from e
|
||||||
|
|
||||||
|
def set_alpn_protocols(self, protocols: list[bytes | str]) -> None:
|
||||||
|
protocols = [util.util.to_bytes(p, "ascii") for p in protocols]
|
||||||
|
return self._ctx.set_alpn_protos(protocols) # type: ignore[no-any-return]
|
||||||
|
|
||||||
|
def wrap_socket(
|
||||||
|
self,
|
||||||
|
sock: socket_cls,
|
||||||
|
server_side: bool = False,
|
||||||
|
do_handshake_on_connect: bool = True,
|
||||||
|
suppress_ragged_eofs: bool = True,
|
||||||
|
server_hostname: bytes | str | None = None,
|
||||||
|
) -> WrappedSocket:
|
||||||
|
cnx = OpenSSL.SSL.Connection(self._ctx, sock)
|
||||||
|
|
||||||
|
# If server_hostname is an IP, don't use it for SNI, per RFC6066 Section 3
|
||||||
|
if server_hostname and not util.ssl_.is_ipaddress(server_hostname):
|
||||||
|
if isinstance(server_hostname, str):
|
||||||
|
server_hostname = server_hostname.encode("utf-8")
|
||||||
|
cnx.set_tlsext_host_name(server_hostname)
|
||||||
|
|
||||||
|
cnx.set_connect_state()
|
||||||
|
|
||||||
|
while True:
|
||||||
|
try:
|
||||||
|
cnx.do_handshake()
|
||||||
|
except OpenSSL.SSL.WantReadError as e:
|
||||||
|
if not util.wait_for_read(sock, sock.gettimeout()):
|
||||||
|
raise TimeoutError("select timed out") from e
|
||||||
|
continue
|
||||||
|
except OpenSSL.SSL.Error as e:
|
||||||
|
raise ssl.SSLError(f"bad handshake: {e!r}") from e
|
||||||
|
break
|
||||||
|
|
||||||
|
return WrappedSocket(cnx, sock)
|
||||||
|
|
||||||
|
def _set_ctx_options(self) -> None:
|
||||||
|
self._ctx.set_options(
|
||||||
|
self._options
|
||||||
|
| _openssl_to_ssl_minimum_version[self._minimum_version]
|
||||||
|
| _openssl_to_ssl_maximum_version[self._maximum_version]
|
||||||
|
)
|
||||||
|
|
||||||
|
@property
|
||||||
|
def minimum_version(self) -> int:
|
||||||
|
return self._minimum_version
|
||||||
|
|
||||||
|
@minimum_version.setter
|
||||||
|
def minimum_version(self, minimum_version: int) -> None:
|
||||||
|
self._minimum_version = minimum_version
|
||||||
|
self._set_ctx_options()
|
||||||
|
|
||||||
|
@property
|
||||||
|
def maximum_version(self) -> int:
|
||||||
|
return self._maximum_version
|
||||||
|
|
||||||
|
@maximum_version.setter
|
||||||
|
def maximum_version(self, maximum_version: int) -> None:
|
||||||
|
self._maximum_version = maximum_version
|
||||||
|
self._set_ctx_options()
|
||||||
|
|
||||||
|
|
||||||
|
def _verify_callback(
|
||||||
|
cnx: OpenSSL.SSL.Connection,
|
||||||
|
x509: X509,
|
||||||
|
err_no: int,
|
||||||
|
err_depth: int,
|
||||||
|
return_code: int,
|
||||||
|
) -> bool:
|
||||||
|
return err_no == 0
|
||||||
+228
@@ -0,0 +1,228 @@
|
|||||||
|
"""
|
||||||
|
This module contains provisional support for SOCKS proxies from within
|
||||||
|
urllib3. This module supports SOCKS4, SOCKS4A (an extension of SOCKS4), and
|
||||||
|
SOCKS5. To enable its functionality, either install PySocks or install this
|
||||||
|
module with the ``socks`` extra.
|
||||||
|
|
||||||
|
The SOCKS implementation supports the full range of urllib3 features. It also
|
||||||
|
supports the following SOCKS features:
|
||||||
|
|
||||||
|
- SOCKS4A (``proxy_url='socks4a://...``)
|
||||||
|
- SOCKS4 (``proxy_url='socks4://...``)
|
||||||
|
- SOCKS5 with remote DNS (``proxy_url='socks5h://...``)
|
||||||
|
- SOCKS5 with local DNS (``proxy_url='socks5://...``)
|
||||||
|
- Usernames and passwords for the SOCKS proxy
|
||||||
|
|
||||||
|
.. note::
|
||||||
|
It is recommended to use ``socks5h://`` or ``socks4a://`` schemes in
|
||||||
|
your ``proxy_url`` to ensure that DNS resolution is done from the remote
|
||||||
|
server instead of client-side when connecting to a domain name.
|
||||||
|
|
||||||
|
SOCKS4 supports IPv4 and domain names with the SOCKS4A extension. SOCKS5
|
||||||
|
supports IPv4, IPv6, and domain names.
|
||||||
|
|
||||||
|
When connecting to a SOCKS4 proxy the ``username`` portion of the ``proxy_url``
|
||||||
|
will be sent as the ``userid`` section of the SOCKS request:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
proxy_url="socks4a://<userid>@proxy-host"
|
||||||
|
|
||||||
|
When connecting to a SOCKS5 proxy the ``username`` and ``password`` portion
|
||||||
|
of the ``proxy_url`` will be sent as the username/password to authenticate
|
||||||
|
with the proxy:
|
||||||
|
|
||||||
|
.. code-block:: python
|
||||||
|
|
||||||
|
proxy_url="socks5h://<username>:<password>@proxy-host"
|
||||||
|
|
||||||
|
"""
|
||||||
|
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
try:
|
||||||
|
import socks # type: ignore[import-untyped]
|
||||||
|
except ImportError:
|
||||||
|
import warnings
|
||||||
|
|
||||||
|
from ..exceptions import DependencyWarning
|
||||||
|
|
||||||
|
warnings.warn(
|
||||||
|
(
|
||||||
|
"SOCKS support in urllib3 requires the installation of optional "
|
||||||
|
"dependencies: specifically, PySocks. For more information, see "
|
||||||
|
"https://urllib3.readthedocs.io/en/latest/advanced-usage.html#socks-proxies"
|
||||||
|
),
|
||||||
|
DependencyWarning,
|
||||||
|
)
|
||||||
|
raise
|
||||||
|
|
||||||
|
import typing
|
||||||
|
from socket import timeout as SocketTimeout
|
||||||
|
|
||||||
|
from ..connection import HTTPConnection, HTTPSConnection
|
||||||
|
from ..connectionpool import HTTPConnectionPool, HTTPSConnectionPool
|
||||||
|
from ..exceptions import ConnectTimeoutError, NewConnectionError
|
||||||
|
from ..poolmanager import PoolManager
|
||||||
|
from ..util.url import parse_url
|
||||||
|
|
||||||
|
try:
|
||||||
|
import ssl
|
||||||
|
except ImportError:
|
||||||
|
ssl = None # type: ignore[assignment]
|
||||||
|
|
||||||
|
|
||||||
|
class _TYPE_SOCKS_OPTIONS(typing.TypedDict):
|
||||||
|
socks_version: int
|
||||||
|
proxy_host: str | None
|
||||||
|
proxy_port: str | None
|
||||||
|
username: str | None
|
||||||
|
password: str | None
|
||||||
|
rdns: bool
|
||||||
|
|
||||||
|
|
||||||
|
class SOCKSConnection(HTTPConnection):
|
||||||
|
"""
|
||||||
|
A plain-text HTTP connection that connects via a SOCKS proxy.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
_socks_options: _TYPE_SOCKS_OPTIONS,
|
||||||
|
*args: typing.Any,
|
||||||
|
**kwargs: typing.Any,
|
||||||
|
) -> None:
|
||||||
|
self._socks_options = _socks_options
|
||||||
|
super().__init__(*args, **kwargs)
|
||||||
|
|
||||||
|
def _new_conn(self) -> socks.socksocket:
|
||||||
|
"""
|
||||||
|
Establish a new connection via the SOCKS proxy.
|
||||||
|
"""
|
||||||
|
extra_kw: dict[str, typing.Any] = {}
|
||||||
|
if self.source_address:
|
||||||
|
extra_kw["source_address"] = self.source_address
|
||||||
|
|
||||||
|
if self.socket_options:
|
||||||
|
extra_kw["socket_options"] = self.socket_options
|
||||||
|
|
||||||
|
try:
|
||||||
|
conn = socks.create_connection(
|
||||||
|
(self.host, self.port),
|
||||||
|
proxy_type=self._socks_options["socks_version"],
|
||||||
|
proxy_addr=self._socks_options["proxy_host"],
|
||||||
|
proxy_port=self._socks_options["proxy_port"],
|
||||||
|
proxy_username=self._socks_options["username"],
|
||||||
|
proxy_password=self._socks_options["password"],
|
||||||
|
proxy_rdns=self._socks_options["rdns"],
|
||||||
|
timeout=self.timeout,
|
||||||
|
**extra_kw,
|
||||||
|
)
|
||||||
|
|
||||||
|
except SocketTimeout as e:
|
||||||
|
raise ConnectTimeoutError(
|
||||||
|
self,
|
||||||
|
f"Connection to {self.host} timed out. (connect timeout={self.timeout})",
|
||||||
|
) from e
|
||||||
|
|
||||||
|
except socks.ProxyError as e:
|
||||||
|
# This is fragile as hell, but it seems to be the only way to raise
|
||||||
|
# useful errors here.
|
||||||
|
if e.socket_err:
|
||||||
|
error = e.socket_err
|
||||||
|
if isinstance(error, SocketTimeout):
|
||||||
|
raise ConnectTimeoutError(
|
||||||
|
self,
|
||||||
|
f"Connection to {self.host} timed out. (connect timeout={self.timeout})",
|
||||||
|
) from e
|
||||||
|
else:
|
||||||
|
# Adding `from e` messes with coverage somehow, so it's omitted.
|
||||||
|
# See #2386.
|
||||||
|
raise NewConnectionError(
|
||||||
|
self, f"Failed to establish a new connection: {error}"
|
||||||
|
)
|
||||||
|
else: # Defensive: see https://github.com/urllib3/urllib3/pull/3728#pullrequestreview-3816302703
|
||||||
|
raise NewConnectionError(
|
||||||
|
self, f"Failed to establish a new connection: {e}"
|
||||||
|
) from e
|
||||||
|
|
||||||
|
except OSError as e: # Defensive: PySocks should catch all these.
|
||||||
|
raise NewConnectionError(
|
||||||
|
self, f"Failed to establish a new connection: {e}"
|
||||||
|
) from e
|
||||||
|
|
||||||
|
return conn
|
||||||
|
|
||||||
|
|
||||||
|
# We don't need to duplicate the Verified/Unverified distinction from
|
||||||
|
# urllib3/connection.py here because the HTTPSConnection will already have been
|
||||||
|
# correctly set to either the Verified or Unverified form by that module. This
|
||||||
|
# means the SOCKSHTTPSConnection will automatically be the correct type.
|
||||||
|
class SOCKSHTTPSConnection(SOCKSConnection, HTTPSConnection):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class SOCKSHTTPConnectionPool(HTTPConnectionPool):
|
||||||
|
ConnectionCls = SOCKSConnection
|
||||||
|
|
||||||
|
|
||||||
|
class SOCKSHTTPSConnectionPool(HTTPSConnectionPool):
|
||||||
|
ConnectionCls = SOCKSHTTPSConnection
|
||||||
|
|
||||||
|
|
||||||
|
class SOCKSProxyManager(PoolManager):
|
||||||
|
"""
|
||||||
|
A version of the urllib3 ProxyManager that routes connections via the
|
||||||
|
defined SOCKS proxy.
|
||||||
|
"""
|
||||||
|
|
||||||
|
pool_classes_by_scheme = {
|
||||||
|
"http": SOCKSHTTPConnectionPool,
|
||||||
|
"https": SOCKSHTTPSConnectionPool,
|
||||||
|
}
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
proxy_url: str,
|
||||||
|
username: str | None = None,
|
||||||
|
password: str | None = None,
|
||||||
|
num_pools: int = 10,
|
||||||
|
headers: typing.Mapping[str, str] | None = None,
|
||||||
|
**connection_pool_kw: typing.Any,
|
||||||
|
):
|
||||||
|
parsed = parse_url(proxy_url)
|
||||||
|
|
||||||
|
if username is None and password is None and parsed.auth is not None:
|
||||||
|
split = parsed.auth.split(":")
|
||||||
|
if len(split) == 2:
|
||||||
|
username, password = split
|
||||||
|
if parsed.scheme == "socks5":
|
||||||
|
socks_version = socks.PROXY_TYPE_SOCKS5
|
||||||
|
rdns = False
|
||||||
|
elif parsed.scheme == "socks5h":
|
||||||
|
socks_version = socks.PROXY_TYPE_SOCKS5
|
||||||
|
rdns = True
|
||||||
|
elif parsed.scheme == "socks4":
|
||||||
|
socks_version = socks.PROXY_TYPE_SOCKS4
|
||||||
|
rdns = False
|
||||||
|
elif parsed.scheme == "socks4a":
|
||||||
|
socks_version = socks.PROXY_TYPE_SOCKS4
|
||||||
|
rdns = True
|
||||||
|
else:
|
||||||
|
raise ValueError(f"Unable to determine SOCKS version from {proxy_url}")
|
||||||
|
|
||||||
|
self.proxy_url = proxy_url
|
||||||
|
|
||||||
|
socks_options = {
|
||||||
|
"socks_version": socks_version,
|
||||||
|
"proxy_host": parsed.host,
|
||||||
|
"proxy_port": parsed.port,
|
||||||
|
"username": username,
|
||||||
|
"password": password,
|
||||||
|
"rdns": rdns,
|
||||||
|
}
|
||||||
|
connection_pool_kw["_socks_options"] = socks_options
|
||||||
|
|
||||||
|
super().__init__(num_pools, headers, **connection_pool_kw)
|
||||||
|
|
||||||
|
self.pool_classes_by_scheme = SOCKSProxyManager.pool_classes_by_scheme
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user