Initial commit: 25 scripts Python sauvegardés

Date: 2026-08-28 11:43:34 EDT
This commit is contained in:
2026-08-28 11:43:34 -04:00
parent 0ba9701e0f
commit 994ea86d47
204 changed files with 70104 additions and 2 deletions
+24 -2
View File
@@ -1,3 +1,25 @@
# hermes-scripts-backup # Hermes Scripts Backup
Backup des scripts Python de Hermes skills — auto-commit automatique ## Purpose
Sauvegarde automatique de tous les scripts Python des compétences Hermes Agent.
## Structure
```
skills/
productivity/
ocr-and-documents/scripts/extract_pymupdf.py
google-workspace/scripts/google_api.py
linear/scripts/linear_api.py
creative/
comfyui/tests/test_extract_schema.py
bin/
restore-skill-scripts.sh — Restaure les scripts depuis ce backup
scripts/
validate-scripts.py — Valide l'intégrité des scripts
```
## Auto-Commit
Les scripts sont sauvegardés automatiquement à chaque modification via un cron job Hermes.
## Last Updated
$(date '+%Y-%m-%d %H:%M:%S %Z')
@@ -0,0 +1,171 @@
---
name: heartmula
description: "HeartMuLa: Suno-like song generation from lyrics + tags."
version: 1.0.0
platforms: [linux, macos, windows]
metadata:
hermes:
tags: [music, audio, generation, ai, heartmula, heartcodec, lyrics, songs]
related_skills: [audiocraft]
---
# HeartMuLa - Open-Source Music Generation
## Overview
HeartMuLa is a family of open-source music foundation models (Apache-2.0) that generates music conditioned on lyrics and tags, with multilingual support. Generates full songs from lyrics + tags. Comparable to Suno for open-source. Includes:
- **HeartMuLa** - Music language model (3B/7B) for generation from lyrics + tags
- **HeartCodec** - 12.5Hz music codec for high-fidelity audio reconstruction
- **HeartTranscriptor** - Whisper-based lyrics transcription
- **HeartCLAP** - Audio-text alignment model
## When to Use
- User wants to generate music/songs from text descriptions
- User wants an open-source Suno alternative
- User wants local/offline music generation
- User asks about HeartMuLa, heartlib, or AI music generation
## Hardware Requirements
- **Minimum**: 8GB VRAM with `--lazy_load true` (loads/unloads models sequentially)
- **Recommended**: 16GB+ VRAM for comfortable single-GPU usage
- **Multi-GPU**: Use `--mula_device cuda:0 --codec_device cuda:1` to split across GPUs
- 3B model with lazy_load peaks at ~6.2GB VRAM
## Installation Steps
### 1. Clone Repository
```bash
cd ~/ # or desired directory
git clone https://github.com/HeartMuLa/heartlib.git
cd heartlib
```
### 2. Create Virtual Environment (Python 3.10 required)
```bash
uv venv --python 3.10 .venv
. .venv/bin/activate
uv pip install -e .
```
### 3. Fix Dependency Compatibility Issues
**IMPORTANT**: As of Feb 2026, the pinned dependencies have conflicts with newer packages. Apply these fixes:
```bash
# Upgrade datasets (old version incompatible with current pyarrow)
uv pip install --upgrade datasets
# Upgrade transformers (needed for huggingface-hub 1.x compatibility)
uv pip install --upgrade transformers
```
### 4. Patch Source Code (Required for transformers 5.x)
**Patch 1 - RoPE cache fix** in `src/heartlib/heartmula/modeling_heartmula.py`:
In the `setup_caches` method of the `HeartMuLa` class, add RoPE reinitialization after the `reset_caches` try/except block and before the `with device:` block:
```python
# Re-initialize RoPE caches that were skipped during meta-device loading
from torchtune.models.llama3_1._position_embeddings import Llama3ScaledRoPE
for module in self.modules():
if isinstance(module, Llama3ScaledRoPE) and not module.is_cache_built:
module.rope_init()
module.to(device)
```
**Why**: `from_pretrained` creates model on meta device first; `Llama3ScaledRoPE.rope_init()` skips cache building on meta tensors, then never rebuilds after weights are loaded to real device.
**Patch 2 - HeartCodec loading fix** in `src/heartlib/pipelines/music_generation.py`:
Add `ignore_mismatched_sizes=True` to ALL `HeartCodec.from_pretrained()` calls (there are 2: the eager load in `__init__` and the lazy load in the `codec` property).
**Why**: VQ codebook `initted` buffers have shape `[1]` in checkpoint vs `[]` in model. Same data, just scalar vs 0-d tensor. Safe to ignore.
### 5. Download Model Checkpoints
```bash
cd heartlib # project root
hf download --local-dir './ckpt' 'HeartMuLa/HeartMuLaGen'
hf download --local-dir './ckpt/HeartMuLa-oss-3B' 'HeartMuLa/HeartMuLa-oss-3B-happy-new-year'
hf download --local-dir './ckpt/HeartCodec-oss' 'HeartMuLa/HeartCodec-oss-20260123'
```
All 3 can be downloaded in parallel. Total size is several GB.
## GPU / CUDA
HeartMuLa uses CUDA by default (`--mula_device cuda --codec_device cuda`). No extra setup needed if the user has an NVIDIA GPU with PyTorch CUDA support installed.
- The installed `torch==2.4.1` includes CUDA 12.1 support out of the box
- `torchtune` may report version `0.4.0+cpu` — this is just package metadata, it still uses CUDA via PyTorch
- To verify GPU is being used, look for "CUDA memory" lines in the output (e.g. "CUDA memory before unloading: 6.20 GB")
- **No GPU?** You can run on CPU with `--mula_device cpu --codec_device cpu`, but expect generation to be **extremely slow** (potentially 30-60+ minutes for a single song vs ~4 minutes on GPU). CPU mode also requires significant RAM (~12GB+ free). If the user has no NVIDIA GPU, recommend using a cloud GPU service (Google Colab free tier with T4, Lambda Labs, etc.) or the online demo at https://heartmula.github.io/ instead.
## Usage
### Basic Generation
```bash
cd heartlib
. .venv/bin/activate
python ./examples/run_music_generation.py \
--model_path=./ckpt \
--version="3B" \
--lyrics="./assets/lyrics.txt" \
--tags="./assets/tags.txt" \
--save_path="./assets/output.mp3" \
--lazy_load true
```
### Input Formatting
**Tags** (comma-separated, no spaces):
```
piano,happy,wedding,synthesizer,romantic
```
or
```
rock,energetic,guitar,drums,male-vocal
```
**Lyrics** (use bracketed structural tags):
```
[Intro]
[Verse]
Your lyrics here...
[Chorus]
Chorus lyrics...
[Bridge]
Bridge lyrics...
[Outro]
```
### Key Parameters
| Parameter | Default | Description |
|-----------|---------|-------------|
| `--max_audio_length_ms` | 240000 | Max length in ms (240s = 4 min) |
| `--topk` | 50 | Top-k sampling |
| `--temperature` | 1.0 | Sampling temperature |
| `--cfg_scale` | 1.5 | Classifier-free guidance scale |
| `--lazy_load` | false | Load/unload models on demand (saves VRAM) |
| `--mula_dtype` | bfloat16 | Dtype for HeartMuLa (bf16 recommended) |
| `--codec_dtype` | float32 | Dtype for HeartCodec (fp32 recommended for quality) |
### Performance
- RTF (Real-Time Factor) ≈ 1.0 — a 4-minute song takes ~4 minutes to generate
- Output: MP3, 48kHz stereo, 128kbps
## Pitfalls
1. **Do NOT use bf16 for HeartCodec** — degrades audio quality. Use fp32 (default).
2. **Tags may be ignored** — known issue (#90). Lyrics tend to dominate; experiment with tag ordering.
3. **Triton not available on macOS** — Linux/CUDA only for GPU acceleration.
4. **RTX 5080 incompatibility** reported in upstream issues.
5. The dependency pin conflicts require the manual upgrades and patches described above.
## Links
- Repo: https://github.com/HeartMuLa/heartlib
- Models: https://huggingface.co/HeartMuLa
- Paper: https://arxiv.org/abs/2601.10547
- License: Apache-2.0
+193
View File
@@ -0,0 +1,193 @@
---
name: kanban-orchestrator
description: Decomposition playbook + anti-temptation rules for an orchestrator profile routing work through Kanban. The "don't do the work yourself" rule and the basic lifecycle are auto-injected into every kanban worker's system prompt; this skill is the deeper playbook when you're specifically playing the orchestrator role.
version: 3.0.0
platforms: [linux, macos, windows]
metadata:
hermes:
tags: [kanban, multi-agent, orchestration, routing]
related_skills: [kanban-worker]
---
# Kanban Orchestrator — Decomposition Playbook
> The **core worker lifecycle** (including the `kanban_create` fan-out pattern and the "decompose, don't execute" rule) is auto-injected into every kanban process via the `KANBAN_GUIDANCE` system-prompt block. This skill is the deeper playbook when you're an orchestrator profile whose whole job is routing.
## Profiles are user-configured — not a fixed roster
Hermes setups vary widely. Some users run a single profile that does everything; some run a small fleet (`docker-worker`, `cron-worker`); some run a curated specialist team they've named themselves. There is **no default specialist roster** — the orchestrator skill does not know what profiles exist on this machine.
Before fanning out, you must ground the decomposition in the profiles that actually exist. The dispatcher silently fails to spawn unknown assignee names — it doesn't autocorrect, doesn't suggest, doesn't fall back. So a card assigned to `researcher` on a setup that only has `docker-worker` just sits in `ready` forever.
**Step 0: discover available profiles before planning.**
Use one of these:
- `hermes profile list` — prints the table of profiles configured on this machine. Run it through your terminal tool if you have one; otherwise ask the user.
- `kanban_list(assignee="<some-name>")` — sanity-check a single name. Returns an empty list (rather than an error) for an unknown assignee, so this only confirms a name you're already considering.
- **Just ask the user.** "What profiles do you have set up?" is a fine first turn when the goal needs more than one specialist.
Cache the result in your working memory for the rest of the conversation. Re-asking every turn wastes a tool call.
## When to use the board (vs. just doing the work)
Create Kanban tasks when any of these are true:
1. **Multiple specialists are needed.** Research + analysis + writing is three profiles.
2. **The work should survive a crash or restart.** Long-running, recurring, or important.
3. **The user might want to interject.** Human-in-the-loop at any step.
4. **Multiple subtasks can run in parallel.** Fan-out for speed.
5. **Review / iteration is expected.** A reviewer profile loops on drafter output.
6. **The audit trail matters.** Board rows persist in SQLite forever.
If *none* of those apply — it's a small one-shot reasoning task — use `delegate_task` instead or answer the user directly.
## The anti-temptation rules
Your job description says "route, don't execute." The rules that enforce that:
- **Do not execute the work yourself.** Your restricted toolset usually doesn't even include terminal/file/code/web for implementation. If you find yourself "just fixing this quickly" — stop and create a task for the right specialist.
- **For any concrete task, create a Kanban task and assign it.** Every single time.
- **Split multi-lane requests before creating cards.** A user prompt can contain several independent workstreams. Extract those lanes first, then create one card per lane instead of bundling unrelated work into a single implementer card.
- **Run independent lanes in parallel.** If two cards do not need each other's output, leave them unlinked so the dispatcher can fan them out. Link only true data dependencies.
- **Never create dependent work as independent ready cards.** If a card must wait for another card, pass `parents=[...]` in the original `kanban_create` call. Do not create it first and link it later, and do not rely on prose like "wait for T1" inside the body.
- **If no specialist fits the available profiles, ask the user which profile to create or which existing profile to use.** Do not invent profile names; the dispatcher will silently drop unknown assignees.
- **Decompose, route, and summarize — that's the whole job.**
## Decomposition playbook
### Step 1 — Understand the goal
Ask clarifying questions if the goal is ambiguous. Cheap to ask; expensive to spawn the wrong fleet.
### Step 2 — Sketch the task graph
Before creating anything, draft the graph out loud (in your response to the user). Treat every concrete workstream as a candidate card:
1. Extract the lanes from the request.
2. Map each lane to one of the profiles you discovered in Step 0. If a lane doesn't fit any existing profile, ask the user which to use or create.
3. Decide whether each lane is independent or gated by another lane.
4. Create independent lanes as parallel cards with no parent links.
5. Create synthesis/review/integration cards with parent links to the lanes they depend on. A child created with unfinished parents starts in `todo`; the dispatcher promotes it to `ready` only after every parent is done.
Examples of prompts that should fan out (using placeholder profile names — substitute whatever exists on the user's setup):
- "Build an app" → one card to a design-oriented profile for product/UI direction, one or two cards to engineering profiles for implementation, plus a later integration/review card if the user has a reviewer profile.
- "Fix blockers and check model variants" → one implementation card for the blocker fixes plus one discovery/research card for config/source verification. A final reviewer card can depend on both.
- "Research docs and implement" → a docs-research card can run in parallel with a codebase-discovery card; implementation waits only if it truly needs those findings.
- "Analyze this screenshot and find the related code" → one card to a vision-capable profile for the visual analysis while another searches the codebase.
Words like "also," "finally," or "and" do not automatically imply a dependency. They often mean "make sure this is covered before reporting back." Only link tasks when one card cannot start until another card's output exists.
Show the graph to the user before creating cards. Let them correct it — including which actual profile name should own each lane.
### Step 3 — Create tasks and link
Use the profile names from Step 0. The example below uses placeholders `<profile-A>`, `<profile-B>`, `<profile-C>` — replace them with what the user actually has.
```python
t1 = kanban_create(
title="research: Postgres cost vs current",
assignee="<profile-A>", # whichever profile handles research on this setup
body="Compare estimated infrastructure costs, migration costs, and ongoing ops costs over a 3-year window. Sources: AWS/GCP pricing, team time estimates, current Postgres bills from peers.",
tenant=os.environ.get("HERMES_TENANT"),
)["task_id"]
t2 = kanban_create(
title="research: Postgres performance vs current",
assignee="<profile-A>", # same profile, run in parallel
body="Compare query latency, throughput, and scaling characteristics at our expected data volume (~500GB, 10k QPS peak). Sources: benchmark papers, public case studies, pgbench results if easy.",
)["task_id"]
t3 = kanban_create(
title="synthesize migration recommendation",
assignee="<profile-B>", # whichever profile does synthesis/analysis
body="Read the findings from T1 (cost) and T2 (performance). Produce a 1-page recommendation with explicit trade-offs and a go/no-go call.",
parents=[t1, t2],
)["task_id"]
t4 = kanban_create(
title="draft decision memo",
assignee="<profile-C>", # whichever profile drafts user-facing prose
body="Turn the analyst's recommendation into a 2-page memo for the CTO. Match the tone of previous decision memos in the team's knowledge base.",
parents=[t3],
)["task_id"]
```
`parents=[...]` gates promotion — children stay in `todo` until every parent reaches `done`, then auto-promote to `ready`. No manual coordination needed; the dispatcher and dependency engine handle it.
If the task graph has dependencies, create the parent cards first, capture their returned ids, and include those ids in the child card's `parents` list during the child `kanban_create` call. Avoid creating all cards in parallel and linking them afterward; that creates a window where the dispatcher can claim a child before its inputs exist.
### Step 4 — Complete your own task
If you were spawned as a task yourself (e.g. a planner profile was assigned `T0: "investigate Postgres migration"`), mark it done with a summary of what you created:
```python
kanban_complete(
summary="decomposed into T1-T4: 2 research lanes in parallel, 1 synthesis on their outputs, 1 prose draft on the recommendation",
metadata={
"task_graph": {
"T1": {"assignee": "<profile-A>", "parents": []},
"T2": {"assignee": "<profile-A>", "parents": []},
"T3": {"assignee": "<profile-B>", "parents": ["T1", "T2"]},
"T4": {"assignee": "<profile-C>", "parents": ["T3"]},
},
},
)
```
### Step 5 — Report back to the user
Tell them what you created in plain prose, naming the actual profiles you used:
> I've queued 4 tasks:
> - **T1** (`<profile-A>`): cost comparison
> - **T2** (`<profile-A>`): performance comparison, in parallel with T1
> - **T3** (`<profile-B>`): synthesizes T1 + T2 into a recommendation
> - **T4** (`<profile-C>`): turns T3 into a CTO memo
>
> The dispatcher will pick up T1 and T2 now. T3 starts when both finish. You'll get a gateway ping when T4 completes. Use the dashboard or `hermes kanban tail <id>` to follow along.
## Common patterns
**Fan-out + fan-in (research → synthesize):** N research-style cards with no parents, one synthesis card with all of them as parents.
**Parallel implementation + validation:** one implementer card makes the change while one explorer/researcher card verifies config, docs, or source mapping. A reviewer card can depend on both. Do not make the implementer own unrelated verification just because the user mentioned both in one sentence.
**Pipeline with gates:** `planner → implementer → reviewer`. Each stage's `parents=[previous_task]`. Reviewer blocks or completes; if reviewer blocks, the operator unblocks with feedback and respawns.
**Same-profile queue:** N tasks, all assigned to the same profile, no dependencies between them. Dispatcher serializes — that profile processes them in priority order, accumulating experience in its own memory.
**Human-in-the-loop:** Any task can `kanban_block()` to wait for input. Dispatcher respawns after `/unblock`. The comment thread carries the full context.
## Pitfalls
**Inventing profile names that don't exist.** The dispatcher silently fails to spawn unknown assignees — the card just sits in `ready` forever. Always assign to a profile from your Step 0 discovery; ask the user if you're unsure.
**Unassigned tasks (assignee is None).** Same symptom as above — the dispatcher only picks up tasks with a valid assignee matching an active profile. A task created without `assignee=`, or one whose assignee was cleared to `None`, will never leave `ready`. Always verify with `kanban_show` or a direct DB query (`SELECT assignee FROM tasks WHERE id=?`) that the task actually has a non-null assignee. See `references/cli-recovery.md` for the fix recipe.
**CLI `kanban assign` can fail silently.** Outside the hermes-agent venv, `python3 -m hermes_cli.kanban assign <id> <profile>` may return exit 0 with no output and no effect — the module import fails but the error is swallowed. Prefer using `execute_code` with a direct `kanban_db.assign_task()` call, setting `HERMES_KANBAN_DB` explicitly. See `references/cli-recovery.md` for the canonical snippet.
**Bundling independent lanes into one card.** If the user asks for two independent outcomes, create two cards. Example: "fix blockers and check model variants" is not one fixer task; create a fixer/engineer card for the fixes and an explorer/researcher card for the variant check, then optionally gate review on both.
**Over-linking because of wording.** "Finally check X" may still be parallel with implementation if X is static config, docs, or source discovery. Link it after implementation only when the check depends on the implementation result.
**Forgetting dependency links.** If the task graph says `research -> implement -> review`, do not create all tasks as independent ready cards. Use parent links so implement/review cannot run before their inputs exist.
**Reassignment vs. new task.** If a reviewer blocks with "needs changes," create a NEW task linked from the reviewer's task — don't re-run the same task with a stern look. The new task is assigned to the original implementer profile.
**Argument order for links.** `kanban_link(parent_id=..., child_id=...)` — parent first. Mixing them up demotes the wrong task to `todo`.
**Don't pre-create the whole graph if the shape depends on intermediate findings.** If T3's structure depends on what T1 and T2 find, let T3 exist as a "synthesize findings" task whose own first step is to read parent handoffs and plan the rest. Orchestrators can spawn orchestrators.
**Tenant inheritance.** If `HERMES_TENANT` is set in your env, pass `tenant=os.environ.get("HERMES_TENANT")` on every `kanban_create` call so child tasks stay in the same namespace.
## Recovering stuck workers
When a worker profile keeps crashing, hallucinating, or getting blocked by its own mistakes (usually: wrong model, missing skill, broken credential), the kanban dashboard flags the task with a ⚠ badge and opens a **Recovery** section in the drawer. Three primary actions:
1. **Reclaim** (or `hermes kanban reclaim <task_id>`) — abort the running worker immediately and reset the task to `ready`. The existing claim TTL is ~15 min; this is the fast path out.
2. **Reassign** (or `hermes kanban reassign <task_id> <new-profile> --reclaim`) — switch the task to a different profile (one that exists on this setup) and let the dispatcher pick it up with a fresh worker.
3. **Change profile model** — the dashboard prints a copy-paste hint for `hermes -p <profile> model` since profile config lives on disk; edit it in a terminal, then Reclaim to retry with the new model.
Hallucination warnings appear on tasks where a worker's `kanban_complete(created_cards=[...])` claim included card ids that don't exist or weren't created by the worker's profile (the gate blocks the completion), or where the free-form summary references `t_<hex>` ids that don't resolve (advisory prose scan, non-blocking). Both produce audit events that persist even after recovery actions — the trail stays for debugging.
@@ -0,0 +1,326 @@
# Instructions for *ACL Proceedings
The following instructions are for authors of papers submitted for review to ACL conferences (hereafter, "review version") or paper accepted for publication in its proceedings (hereafter, "final version").
All authors are required to adhere to these specifications.
## Style Files
*ACL provides style files for LaTeX and Microsoft Word that meet these requirements. They can be found at:
> https://acl-org.github.io/ACLPUB/
We strongly recommend the use of these style files, which have been appropriately tailored for the *ACL proceedings.
## Paper Length
The conference accepts submissions of long papers and short papers.
Review versions of long papers may have up to eight (8) pages of content plus unlimited pages for references.
Upon acceptance, final versions of long papers will be given one additional page -- up to nine (9) pages of content plus unlimited pages for acknowledgements and references -- so that reviewers' comments can be taken into account.
Review versions of short papers may have up to four (4) pages of content, plus unlimited pages for references.
Final versions of short papers may have up to five (5) pages, plus unlimited pages for acknowledgements and references.
For both long and short papers, all figures and tables that are part of the main text must fit within these page limits.
The conference encourages submission of appendices and supplementary material, which are not required to fit within these page limits. However, review versions of papers must be self-contained: it is optional for reviewers to look at appendices or supplementary material. Please see [Appendices](#Appendices) and [Supplementary](#Supplementary Material) for more information.
Review versions should not refer, for further detail, to documents, code or data resources that are not available to the reviewers.
Papers that do not conform to these requirements may be rejected without review.
Workshop chairs may have different rules for allowed length and whether appendices or supplementary materials are welcome.
As always, the respective call for papers is the authoritative source.
## Anonymity
As reviewing will be double-blind, review versions must not include any identifying information about the authors (such as names, affiliations, or URLs).
Self-references that reveal the author's identity, e.g.,
> We previously showed (Gusfield, 1997)...
must be avoided, and anonymous citations, e.g.,
> We previously showed (Anonymous, 1997)...
should also be avoided. Instead, use citations such as
> Gusfield (1997) previously showed...
Review versions must not include acknowledgements.
**Papers that do not conform to these requirements may be rejected without review.**
Any preliminary non-archival versions of submitted papers should be listed in the submission form but not in the review version of the paper.
Reviewers are generally aware that authors may present preliminary versions of their work in other venues, but will not be provided the list of previous presentations from the submission form.
Once a paper has been accepted to the conference, the final version should include the author's names and affiliations, and is allowed to use self-references.
## Multiple Submission
Papers that have been or will be submitted to other meetings or publications must indicate this at submission time in the START submission form, and must be withdrawn from the other venues if accepted by *ACL.
Authors of papers accepted for presentation at *ACL must notify the program chairs by the deadline for final versions ("camera-ready deadline") whether the paper will be presented.
We will not accept for publication or presentation any papers that overlap significantly in content or results with papers that will be (or have been) published elsewhere.
Authors submitting more than one paper to *ACL must ensure that submissions do not overlap significantly (>25%) with each other in content or results.
## Formatting Instructions
### File Format
Papers must be in Adobe Portable Document Format (PDF).
Please make sure that your PDF file embeds all necessary fonts (especially for tree diagrams, symbols, and Asian languages).
When you print or create the PDF file, there is usually an option in your printer setup to include none, all or just non-standard fonts.
Please make sure that you select the option of including *all* the fonts.
**Before sending it, test your PDF by printing it from a computer different from the one where it was created.**
Some word processors may generate very large PDF files, where each page is rendered as an image.
Such images may reproduce poorly.
In this case, try alternative ways to obtain the PDF.
All papers must use **A4 paper format** (21 cm x 29.7 cm).
Papers must not be submitted with any other paper size.
If you cannot meet the above requirements, please contact the publication chairs as soon as possible.
### Layout
All text except for page numbers must fit within the margins.
Review versions should have page numbers, centered in the bottom margin, but **pages should not be numbered in the final version.**
Manuscripts must be set in two columns.
Exceptions to the two-column format include the title, authors' names and complete addresses, which must be centered at the top of the first page, and any full-width figures or tables.
The exact dimensions for a page on A4 paper are:
* Left margin: 2.5 cm
* Right margin: 2.5 cm
* Top margin: 2.5 cm
* Bottom margin: 2.5 cm
* Column width: 7.7 cm
* Column height: 24.7 cm
* Gap between columns: 0.6 cm
In the review version, a ruler (line numbers in the left and right margins of the article) should be printed, so that reviewers may comment on particular lines in the paper.
The ruler should not change the appearance of any other content on the page.
The final version should not contain a ruler.
### Fonts
All text (except non-Latin scripts and mathematical formulas) should be set in **Times Roman**.
If Times Roman is unavailable, you may use **Times New Roman** or **Computer Modern Roman.**
The following table specifies what font sizes and styles must be used for each type of text in the manuscript.
| Type of Text | Font Size | Style |
| --------------------- | --------- | ----- |
| paper title | 15 pt | bold |
| author names | 12 pt | bold |
| author affiliation | 12 pt | |
| the word ``Abstract'' | 12 pt | bold |
| section titles | 12 pt | bold |
| subsection titles | 11 pt | bold |
| document text | 11 pt | |
| captions | 10 pt | |
| abstract text | 10 pt | |
| bibliography | 10 pt | |
| footnotes | 9 pt | |
### Title and Authors
Center the title, author's name(s) and affiliation(s) across both columns.
Place the title centered at the top of the first page, in 15-point bold.
Long titles should be typed on two lines without a blank line intervening.
Put the title 2.5 cm from the top of the page.
Write the title in [title case](https://apastyle.apa.org/style-grammar-guidelines/capitalization/title-case); do not write the title in all capital letters, except for acronyms (e.g., "BLEU") or proper nouns ("English") that are normally uppercased or capitalized.
Place the author name(s) and affiliation(s) under the title.
Write authors' full names; do not abbreviate given names to initials, unless they are normally written as initials ("Margaret Mitchell", not "M. Mitchell").
Do not format surnames in all capitals ("Mitchell", not "MITCHELL").
Do not use footnotes for affiliations.
The affiliation should contain the author's complete address, and if possible, an electronic mail address.
The title, author names and addresses should be completely identical to those entered to the paper submission website in order to maintain the consistency of author information among all publications of the conference.
If they are different, the publication chairs may resolve the difference without consulting with you; so it is in your own interest to double-check that the information is consistent.
Start the body of the first page 7.5 cm from the top of the page.
**Even in the review version of the paper, you should maintain space for names and addresses so that they will fit in the final version.**
### Abstract
Type the abstract at the beginning of the first column.
Center the word **Abstract** in 12 point bold above the body of the abstract.
The width of the abstract should be smaller than the
normal column width by 0.6 cm on each side.
The abstract text should be 10 point roman, single-spaced.
The abstract should be a concise summary of the general thesis and conclusions of the paper.
It should be no longer than 200 words.
### Text
Begin typing the main body of the text immediately after the abstract, continuing in two columns.
The text should be 11 point roman, single-spaced.
Indent 0.4 cm when starting a new paragraph, except for the first paragraph in a section.
### Sections
Use numbered sections (Arabic numerals) to facilitate cross references.
Number subsections with the section number and the subsection number separated by a dot, in Arabic numerals, e.g.,
> 1 Introduction
or
> 6.1 File Format
### Footnotes
Put footnotes at the bottom of the page and use 9 point font.
They may be numbered or referred to by asterisks or other symbols.
Footnotes should be separated from the text by a line.
### Figures and tables
Place figures and tables in the paper near where they are first discussed, rather than at the end, if possible.
Wide figures/tables may run across both columns.
To accommodate people who are color-blind (as well as those printing with black-and-white printers), grayscale readability is strongly encouraged.
Color is not forbidden, but authors should ensure that tables and figures do not rely solely on color to convey critical distinctions.
**Captions:**
Provide a caption for every figure/table; number each one sequentially in the form:
> Figure 1: Caption of the Figure.
and
> Table 1: Caption of the Table.
Captions should be placed below figures/tables, in 10 point roman type.
Captions that are one line are centered.
Captions longer than one line are left-aligned.
### Hyperlinks
Within-document and external hyperlinks should be dark blue (hex #000099), not underlined or boxed.
### Non-English Text
Text in languages other than English should be accompanied by translations into English, and text in scripts other than Latin should \emph{also} be accompanied by transliterations into Latin script, since not all readers can recognize non-Latin characters easily.
For example, παράδειγμα *paradeigma* ‘example’ is a Greek word, and this is a Greek sentence:
> Αυτό είναι ένα παράδειγμα.
> auto einai ena paradeigma.
> ‘This is an example.’
### Citations
Citations within the text appear in parentheses (Gusfield, 1997), or, if the author's name appears in the text itself: Gusfield (1997).
Append lowercase letters to the year in cases of ambiguities.
Cite papers with two authors using both authors' names (Aho and Ullman, 1972), but cite papers with more than two authors by the first author's name and ``et al.'' (Chandra et al., 1981).
Collapse multiple citations into a single pair of parentheses (Gusfield, 1997; Aho and Ullman, 1972).
Refrain from using full citations as sentence constituents.
Instead of
> (Gusfield, 1997) showed that ...
> In (Gusfield, 1997), ...''
write
> Gusfield (1997) showed that ...
> In Gusfield (1997), ...
Submissions should accurately reference prior and related work, including code and data.
If a piece of prior work appeared in multiple venues, the version that appeared in a refereed, archival venue should be referenced.
If multiple versions of a piece of prior work exist, the one used by the authors should be referenced.
### Acknowledgments
The acknowledgments should go immediately before the references.
Do not number the acknowledgments section.
Do not include this section in the review version.
### References
Gather the full set of references together under the unnumbered section heading **References**.
Place the References section before any Appendices.
Arrange the references alphabetically by first author, rather than by order of occurrence in the text.
Provide as complete a citation as possible, using a consistent format, such as the [one for Computational Linguistics](http://cljournal.org/style_guide_refs.html) or the one in the [Publication Manual of the American Psychological Association](https://apastyle.apa.org/products/publication-manual-7th-edition).
Use full names for authors, not just initials.
Authors should not rely on automated citation indices to provide accurate references for prior and related work.
As part of our work to make ACL materials more widely used and cited outside of our discipline, ACL has registered as a CrossRef member, as a registrant of Digital Object Identifiers (DOIs), the standard for registering permanent URNs for referencing scholarly materials.
All references are required to contain DOIs of all cited works when possible, or, as a second resort, links to ACL Anthology pages.
Appropriate records should be found for most materials in the current [ACL Anthology](https://aclweb.org/anthology/).
Example article in a journal:
> Rie Kubota Ando and Tong Zhang. 2005. [A framework for learning predictive structures from multiple tasks and unlabeled data](https://www.jmlr.org/papers/v6/ando05a.html). *Journal of Machine Learning Research*, 6:1817–1853.
Example paper in non-ACL proceedings, with DOI:
> Galen Andrew and Jianfeng Gao. 2007. [Scalable training of L1-regularized log-linear models](https://doi.org/10.1145/1273496.1273501). In *Proceedings of the 24th International Conference on Machine Learning*, pages 33–40.
Example ACL Anthology paper with DOI:
> James Goodman, Andreas Vlachos, and Jason Naradowsky. 2016. [Noise reduction and targeted exploration in imitation learning for Abstract Meaning Representation parsing](http://dx.doi.org/10.18653/v1/P16-1001). In *Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 1–45711, Berlin, Germany. Association for Computational Linguistics.
Example ACL Anthology paper without DOI:
> Benjamin Börschinger and Mark Johnson. 2011. [A particle filter algorithm for Bayesian word segmentation](https://www.aclweb.org/anthology/U11-1004/). In *Proceedings of the Australasian Language Technology Association Workshop 2011*, pages 10–44718, Canberra, Australia.
Example arXiv paper:
> Mohammad Sadegh Rasooli and Joel R. Tetreault. 2015. [Yara parser: A fast and accurate dependency parser](http://arxiv.org/abs/1503.06733). *Computing Research Repository*, arXiv:1503.06733. Version 2.
## Appendices
Appendices are material that can be read, and include lemmas, formulas, proofs, and tables that are not critical to the reading and understanding of the paper.
Letter them in sequence and provide an informative title:
> Appendix A. Title of Appendix
The appendices come after the references.
Review versions of appendices must follow the same anonymity guidelines as the main paper.
## Supplementary Material
Submissions may include non-readable supplementary material used in the work and described in the paper.
Any accompanying software and/or data should include licenses and documentation of research review as appropriate.
Supplementary material may report preprocessing decisions, model parameters, and other details necessary for the replication of the experiments reported in the paper.
Seemingly small preprocessing decisions can sometimes make a large difference in performance, so it is crucial to record such decisions to precisely characterize state-of-the-art methods.
Nonetheless, supplementary material should be supplementary (rather than central) to the paper.
**Submissions that misuse the supplementary material may be rejected without review.**
Supplementary material may include explanations or details of proofs or derivations that do not fit into the paper, lists of features or feature templates, sample inputs and outputs for a system, pseudo-code or source code, and data.
(Source code and data should be separate uploads, rather than part of the paper).
The paper should not rely on the supplementary material: while the paper may refer to and cite the supplementary material and the supplementary material will be available to the reviewers, they will not be asked to review the supplementary material.
Review versions of supplementary material must follow the same anonymity guidelines as the main paper.
## Credits
This document has been adapted from the instructions for earlier ACL and NAACL proceedings, including those for
ACL 2020 by Steven Bethard, Ryan Cotterell and Rui Yan,
ACL 2019 by Douwe Kiela and Ivan Ivan Vulić,
NAACL 2019 by Stephanie Lukin and Alla Roskovskaya,
ACL 2018 by Shay Cohen, Kevin Gimpel, and Wei Lu,
NAACL 2018 by Margaret Mitchell and Stephanie Lukin,
BibTeX suggestions for (NA)ACL 2017/2018 from Jason Eisner,
ACL 2017 by Dan Gildea and Min-Yen Kan,
NAACL 2017 by Margaret Mitchell,
ACL 2012 by Maggie Li and Michael White,
ACL 2010 by Jing-Shin Chang and Philipp Koehn,
ACL 2008 by Johanna D. Moore, Simone Teufel, James Allan, and Sadaoki Furui,
ACL 2005 by Hwee Tou Ng and Kemal Oflazer,
ACL 2002 by Eugene Charniak and Dekang Lin,
and earlier ACL and EACL formats written by several people, including
John Chen, Henry S. Thompson and Donald Walker.
Additional elements were taken from the formatting instructions of the *International Joint Conference on Artificial Intelligence* and the *Conference on Computer Vision and Pattern Recognition*.
+437
View File
@@ -0,0 +1,437 @@
#!/usr/bin/env python3
"""
check_deps.py — Verify a ComfyUI workflow's dependencies (custom nodes, models,
embeddings) against a running server.
Improvements over v1:
- Cloud-aware endpoint mapping (handles `/api/experiment/models/{folder}` and
`/api/object_info` variants verified against live cloud API)
- Distinguishes 200-empty (genuinely no models in folder) vs 404
(folder doesn't exist) vs 403 (auth/tier issue) — no silent passes
- Outputs concrete remediation commands (e.g. `comfy node install <name>`)
when nodes are missing
- Detects embedding references inside prompt strings as model deps
- Skips check on cloud free tier `/api/object_info` (403) without false alarm
- Accepts API key from CLI flag OR $COMFY_CLOUD_API_KEY env var
Usage:
python3 check_deps.py workflow_api.json
python3 check_deps.py workflow_api.json --host 127.0.0.1 --port 8188
python3 check_deps.py workflow_api.json --host https://cloud.comfy.org
Stdlib-only. Python 3.10+.
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from _common import ( # noqa: E402
DEFAULT_LOCAL_HOST, ENV_API_KEY,
emit_json, folder_aliases_for, http_get, is_cloud_host,
iter_embedding_refs, iter_model_deps, iter_nodes, parse_model_list,
resolve_api_key, resolve_url, unwrap_workflow,
)
# Known node → custom-node-package map. When a workflow needs a node we don't
# recognize, suggesting the right `comfy node install ...` makes the difference
# between a working agent and a stuck one.
NODE_TO_PACKAGE: dict[str, str] = {
# rgthree (Reroute is JS-only and doesn't appear in /object_info)
"Power Lora Loader (rgthree)": "rgthree-comfy",
"Image Comparer (rgthree)": "rgthree-comfy",
"Seed (rgthree)": "rgthree-comfy",
"Display Any (rgthree)": "rgthree-comfy",
"Display Int (rgthree)": "rgthree-comfy",
# Impact pack
"FaceDetailer": "comfyui-impact-pack",
"DetailerForEach": "comfyui-impact-pack",
"BboxDetectorSEGS": "comfyui-impact-pack",
"SAMLoader": "comfyui-impact-pack",
"ImpactWildcardProcessor": "comfyui-impact-pack",
# Impact subpack (separate package)
"UltralyticsDetectorProvider": "comfyui-impact-subpack",
# Was Node Suite
"Image Save": "was-node-suite-comfyui",
"Number Counter": "was-node-suite-comfyui",
"Text String": "was-node-suite-comfyui",
# easy-use
"easy fullLoader": "comfyui-easy-use",
"easy positive": "comfyui-easy-use",
"easy negative": "comfyui-easy-use",
"easy seed": "comfyui-easy-use",
"easy imageSave": "comfyui-easy-use",
# Video Helper Suite
"VHS_VideoCombine": "comfyui-videohelpersuite",
"VHS_LoadVideo": "comfyui-videohelpersuite",
"VHS_LoadAudio": "comfyui-videohelpersuite",
# AnimateDiff
"ADE_AnimateDiffLoaderWithContext": "comfyui-animatediff-evolved",
"ADE_AnimateDiffLoaderGen1": "comfyui-animatediff-evolved",
"ADE_LoadAnimateDiffModel": "comfyui-animatediff-evolved",
# ControlNet aux preprocessors (full class names)
"CannyEdgePreprocessor": "comfyui_controlnet_aux",
"DWPreprocessor": "comfyui_controlnet_aux",
"OpenposePreprocessor": "comfyui_controlnet_aux",
"DepthAnythingPreprocessor": "comfyui_controlnet_aux",
"Zoe_DepthAnythingPreprocessor": "comfyui_controlnet_aux",
"AnimalPosePreprocessor": "comfyui_controlnet_aux",
# IPAdapter Plus
"IPAdapterAdvanced": "comfyui_ipadapter_plus",
"IPAdapterUnifiedLoader": "comfyui_ipadapter_plus",
"IPAdapterModelLoader": "comfyui_ipadapter_plus",
"IPAdapterInsightFaceLoader": "comfyui_ipadapter_plus",
# InstantID
"InstantIDModelLoader": "comfyui_instantid",
"ApplyInstantID": "comfyui_instantid",
# Comfy essentials (note: registry slug uses underscore, not hyphen)
"GetImageSize+": "comfyui_essentials",
"ImageBatchMultiple+": "comfyui_essentials",
# pysssss
"ShowText|pysssss": "comfyui-custom-scripts",
"PreviewImage|pysssss": "comfyui-custom-scripts",
# SUPIR
"SUPIR_Upscale": "comfyui-supir",
"SUPIR_first_stage": "comfyui-supir",
# GGUF (case-sensitive registry slug)
"UNETLoaderGGUF": "ComfyUI-GGUF",
"DualCLIPLoaderGGUF": "ComfyUI-GGUF",
# Florence2
"Florence2Run": "comfyui-florence2",
# WAS
"Image Filter Adjustments": "was-node-suite-comfyui",
# Photomaker (case-sensitive)
"PhotoMakerLoader": "ComfyUI-PhotoMaker-Plus",
# Wan video (case-sensitive)
"WanVideoSampler": "ComfyUI-WanVideoWrapper",
"WanVideoModelLoader": "ComfyUI-WanVideoWrapper",
}
# Nodes whose package isn't on the comfy registry — need git-URL install via
# ComfyUI-Manager. We surface a helpful hint instead of an unrunnable command.
NODE_TO_GIT_URL: dict[str, str] = {
"HunyuanVideoSampler": "https://github.com/kijai/ComfyUI-HunyuanVideoWrapper",
"HunyuanVideoModelLoader": "https://github.com/kijai/ComfyUI-HunyuanVideoWrapper",
}
def fetch_object_info(url: str, headers: dict) -> tuple[set[str] | None, dict | None]:
"""Returns (installed_node_set, error_info). Error info is a dict if we
couldn't query (e.g. cloud free tier), else None.
"""
r = http_get(url, headers=headers, retries=2, timeout=30)
if r.status == 200:
try:
data = r.json()
if isinstance(data, dict):
return set(data.keys()), None
except Exception:
pass
return None, {"http_status": 200, "reason": "non-dict response"}
if r.status == 403:
try:
body = r.json()
except Exception:
body = {"raw": r.text()[:200]}
return None, {"http_status": 403, "reason": "forbidden", "body": body}
if r.status == 404:
return None, {"http_status": 404, "reason": "endpoint not found"}
return None, {"http_status": r.status, "reason": "unexpected", "body": r.text()[:200]}
def _fetch_one_folder(
base: str, folder: str, headers: dict, *, is_cloud: bool,
) -> tuple[set[str] | None, dict | None]:
"""Single-folder fetch, no aliasing. Returns (installed_set, error_info)."""
url = resolve_url(base, f"/models/{folder}", is_cloud=is_cloud)
r = http_get(url, headers=headers, retries=2, timeout=30)
if r.status == 200:
try:
return parse_model_list(r.json()), None
except Exception:
return set(), {"http_status": 200, "reason": "non-list response"}
if r.status == 404:
body_text = r.text()
try:
body = r.json()
except Exception:
body = {"raw": body_text[:200]}
code = body.get("code") if isinstance(body, dict) else None
if code == "folder_not_found":
# Folder is genuinely empty/missing on server — not the same as
# "endpoint missing". Return empty set with informational error.
return set(), {"http_status": 404, "reason": "folder_empty_or_unknown", "body": body}
return None, {"http_status": 404, "reason": "endpoint not found", "body": body}
if r.status == 403:
try:
body = r.json()
except Exception:
body = {}
return None, {"http_status": 403, "reason": "forbidden", "body": body}
return None, {"http_status": r.status, "reason": "unexpected"}
def fetch_models_for_folder(
base: str, folder: str, headers: dict, *, is_cloud: bool,
) -> tuple[set[str] | None, dict | None]:
"""Fetch installed models for a folder, trying aliases.
Folder renames over time (e.g. unet → diffusion_models, clip → text_encoders)
mean a workflow asking for a model in `unet` may need to look in
`diffusion_models`. We union models from every reachable alias.
Returns (combined_set | None, last_error | None).
"""
aliases = folder_aliases_for(folder)
combined: set[str] = set()
any_success = False
last_err: dict | None = None
for alias in aliases:
models, err = _fetch_one_folder(base, alias, headers, is_cloud=is_cloud)
if models is not None:
combined.update(models)
any_success = True
last_err = None
else:
last_err = err
if not any_success:
return None, last_err
return combined, None
def fetch_embeddings(base: str, headers: dict, *, is_cloud: bool) -> tuple[set[str] | None, dict | None]:
"""Local ComfyUI exposes /embeddings; cloud uses /experiment/models/embeddings."""
if is_cloud:
return fetch_models_for_folder(base, "embeddings", headers, is_cloud=True)
# Local: dedicated /embeddings returns a flat list of names
r = http_get(resolve_url(base, "/embeddings", is_cloud=False), headers=headers, retries=2)
if r.status == 200:
try:
data = r.json()
if isinstance(data, list):
# Strip extensions from the registered names since prompt syntax
# usually omits them ("embedding:goodvibes" vs "goodvibes.pt")
names = set()
for n in data:
if isinstance(n, str):
names.add(n)
# Also store stem for fuzzy matching
names.add(Path(n).stem)
return names, None
except Exception:
pass
return None, {"http_status": r.status, "reason": "unexpected"}
def normalize_for_match(name: str) -> set[str]:
"""Generate matching variants of a model name (with/without extension, slashes, etc.)"""
s = {name}
s.add(Path(name).stem)
s.add(Path(name).name)
# ComfyUI sometimes strips/keeps the leading folder
if "/" in name or "\\" in name:
flat = name.replace("\\", "/").split("/")[-1]
s.add(flat)
s.add(Path(flat).stem)
return {x for x in s if x}
def model_present(needed: str, installed: set[str]) -> bool:
if not installed:
return False
needed_variants = normalize_for_match(needed)
installed_norm: set[str] = set()
for inst in installed:
installed_norm.update(normalize_for_match(inst))
return bool(needed_variants & installed_norm)
def suggest_install_command(node_class: str) -> str | None:
pkg = NODE_TO_PACKAGE.get(node_class)
if pkg:
return f"comfy node install {pkg}"
return None
def suggest_git_url(node_class: str) -> str | None:
"""For nodes not on the registry, return a git URL the user can hand to
ComfyUI-Manager's `/manager/queue/install` endpoint."""
return NODE_TO_GIT_URL.get(node_class)
def check_deps(
workflow: dict, host: str, *, api_key: str | None = None,
) -> dict:
headers: dict[str, str] = {}
if api_key:
headers["X-API-Key"] = api_key
is_cloud = is_cloud_host(host)
base = host.rstrip("/")
# ---- 1. Required nodes ----
required_nodes: set[str] = set()
for _, node in iter_nodes(workflow):
required_nodes.add(node["class_type"])
object_info_url = resolve_url(base, "/object_info", is_cloud=is_cloud)
installed_nodes, obj_err = fetch_object_info(object_info_url, headers)
missing_nodes: list[dict] = []
node_check_skipped = False
if installed_nodes is None:
# Couldn't query (e.g. cloud free tier). Don't false-alarm; mark skipped.
node_check_skipped = True
else:
for cls in sorted(required_nodes):
if cls not in installed_nodes:
entry = {"class_type": cls}
cmd = suggest_install_command(cls)
git_url = suggest_git_url(cls)
if cmd:
entry["fix_command"] = cmd
elif git_url:
entry["fix_git_url"] = git_url
entry["fix_hint"] = (
f"Not on registry. Install via Manager with this git URL: {git_url}"
)
else:
entry["fix_hint"] = (
"Search https://registry.comfy.org or "
"use ComfyUI-Manager UI to find the package providing this node."
)
missing_nodes.append(entry)
# ---- 2. Required models ----
model_cache: dict[str, tuple[set[str] | None, dict | None]] = {}
missing_models: list[dict] = []
folder_errors: dict[str, dict] = {}
for dep in iter_model_deps(workflow):
folder = dep["folder"]
if folder not in model_cache:
model_cache[folder] = fetch_models_for_folder(
base, folder, headers, is_cloud=is_cloud,
)
installed, err = model_cache[folder]
if installed is None:
# Couldn't enumerate this folder — record once
folder_errors.setdefault(folder, err or {})
# Don't flag as missing (we don't know); the folder_errors block surfaces this
continue
if not model_present(dep["value"], installed):
entry = dict(dep)
entry["fix_hint"] = (
f"comfy model download --url <URL> --relative-path models/{folder} "
f"--filename {dep['value']!r}"
)
missing_models.append(entry)
# ---- 3. Embedding refs in prompts ----
emb_installed, emb_err = fetch_embeddings(base, headers, is_cloud=is_cloud)
missing_embeddings: list[dict] = []
seen_emb: set[tuple[str, str]] = set()
for nid, emb_name in iter_embedding_refs(workflow):
if (nid, emb_name) in seen_emb:
continue
seen_emb.add((nid, emb_name))
if emb_installed is None:
# Couldn't enumerate — skip silently here, surface the error in the
# folder_errors block
continue
if not model_present(emb_name, emb_installed):
missing_embeddings.append({
"node_id": nid,
"embedding_name": emb_name,
"folder": "embeddings",
"fix_hint": (
f"Download {emb_name}.pt or .safetensors and place in "
f"models/embeddings/, or `comfy model download --url <URL> "
f"--relative-path models/embeddings`"
),
})
if emb_err and emb_installed is None:
folder_errors.setdefault("embeddings", emb_err)
is_ready = (
not node_check_skipped
and not missing_nodes
and not missing_models
and not missing_embeddings
)
return {
"is_ready": is_ready,
"node_check_skipped": node_check_skipped,
"node_check_skip_reason": obj_err if node_check_skipped else None,
"missing_nodes": missing_nodes,
"missing_models": missing_models,
"missing_embeddings": missing_embeddings,
"folder_errors": folder_errors,
# 0 is a legitimate count (e.g. empty server). Use None only when not queried.
"installed_node_count": len(installed_nodes) if installed_nodes is not None else None,
"required_node_count": len(required_nodes),
"required_nodes": sorted(required_nodes),
"host": base,
"is_cloud": is_cloud,
}
def main(argv: list[str] | None = None) -> int:
p = argparse.ArgumentParser(description="Check ComfyUI workflow dependencies against a running server")
p.add_argument("workflow", help="Path to workflow API JSON file")
p.add_argument("--host", default=DEFAULT_LOCAL_HOST, help="ComfyUI server URL")
p.add_argument("--port", type=int, help="Server port (overrides --host port)")
p.add_argument("--api-key", help=f"API key for cloud (or set ${ENV_API_KEY} env var)")
p.add_argument("--strict", action="store_true",
help="Exit non-zero if node check is skipped (e.g. on cloud free tier)")
args = p.parse_args(argv)
host = args.host
if args.port is not None:
# Strip any port from host and append --port
from urllib.parse import urlparse, urlunparse
parsed = urlparse(host if "://" in host else f"http://{host}")
new_netloc = f"{parsed.hostname}:{args.port}"
host = urlunparse(parsed._replace(netloc=new_netloc))
api_key = resolve_api_key(args.api_key)
wf_path = Path(args.workflow).expanduser()
if not wf_path.exists():
emit_json({"error": f"Workflow file not found: {args.workflow}"})
return 1
try:
with wf_path.open() as f:
payload = json.load(f)
workflow = unwrap_workflow(payload)
except ValueError as e:
emit_json({"error": str(e)})
return 1
except json.JSONDecodeError as e:
emit_json({"error": f"Invalid JSON: {e}"})
return 1
try:
result = check_deps(workflow, host=host, api_key=api_key)
except Exception as e:
emit_json({"error": f"Dep check failed: {e}", "host": host})
return 1
emit_json(result)
if not result["is_ready"]:
return 1
if args.strict and result["node_check_skipped"]:
return 1
return 0
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,315 @@
#!/usr/bin/env python3
"""
extract_schema.py — Analyze a ComfyUI API-format workflow and extract
controllable parameters.
Improvements over v1:
- Catalogs live in `_common.py`, shared with `check_deps.py`
- Coverage expanded for Flux / SD3 / Wan / Hunyuan / LTX / IPAdapter / rgthree
- Symmetric duplicate-name resolution: ALL duplicates get a node-id suffix
(instead of "first wins, second renamed"), so callers see consistent names
- Negative prompt detected by tracing `KSampler.negative` connections back to
the source CLIPTextEncode (more reliable than meta-title heuristic)
- Embedding references in prompt text are extracted as model dependencies
- Detects Primitive nodes that drive other nodes' inputs (and surfaces them
as the user-facing parameter)
- Reroutes are followed when tracing connections
Usage:
python3 extract_schema.py workflow_api.json
python3 extract_schema.py workflow_api.json --output schema.json
Stdlib-only. Python 3.10+.
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
from typing import Any
sys.path.insert(0, str(Path(__file__).resolve().parent))
from _common import ( # noqa: E402
OUTPUT_NODES, PARAM_PATTERNS, PROMPT_FIELDS,
is_link, iter_embedding_refs, iter_model_deps, iter_nodes, unwrap_workflow,
)
# Sampler nodes whose `positive` / `negative` connections we trace
SAMPLER_NODE_FAMILY = {
"KSampler", "KSamplerAdvanced",
"SamplerCustom", "SamplerCustomAdvanced",
"BasicGuider", "CFGGuider", "DualCFGGuider",
}
def infer_type(value: Any) -> str:
if isinstance(value, bool):
return "bool"
if isinstance(value, int):
return "int"
if isinstance(value, float):
return "float"
if isinstance(value, str):
return "string"
if isinstance(value, list):
return "link"
if isinstance(value, dict):
return "object"
return "unknown"
def trace_to_node(workflow: dict, link: list, *, max_hops: int = 8) -> str | None:
"""Follow a [node_id, slot] link, hopping through Reroute / Primitive nodes
if needed, to find the *upstream* node id that holds the actual value/input.
Bounded by both `max_hops` AND a visited-set to prevent infinite loops on
pathological graphs.
"""
if not is_link(link):
return None
nid: str | None = link[0]
visited: set[str] = set()
for _ in range(max_hops):
if nid is None or nid in visited:
return nid
visited.add(nid)
node = workflow.get(nid)
if not isinstance(node, dict):
return None
cls = node.get("class_type", "")
# Reroute / Primitive / passthrough wrappers
if cls in {"Reroute", "PrimitiveNode", "Note", "easy showAnything"}:
inputs = node.get("inputs", {}) or {}
# Find first link-shaped input and follow it
next_link = next((v for v in inputs.values() if is_link(v)), None)
if next_link is None:
return nid
nid = next_link[0]
continue
return nid
return nid
def find_negative_prompt_node(workflow: dict) -> str | None:
"""Trace `negative` input of a sampler back to the source text encoder."""
for nid, node in iter_nodes(workflow):
if node["class_type"] not in SAMPLER_NODE_FAMILY:
continue
inputs = node.get("inputs", {}) or {}
neg = inputs.get("negative")
if not is_link(neg):
continue
src = trace_to_node(workflow, neg)
if src and isinstance(workflow.get(src), dict):
cls = workflow[src].get("class_type", "")
if cls.startswith("CLIPTextEncode") or cls in {"smZ CLIPTextEncode", "BNK_CLIPTextEncodeAdvanced"}:
return src
return None
def find_positive_prompt_node(workflow: dict) -> str | None:
for nid, node in iter_nodes(workflow):
if node["class_type"] not in SAMPLER_NODE_FAMILY:
continue
inputs = node.get("inputs", {}) or {}
pos = inputs.get("positive")
if not is_link(pos):
continue
src = trace_to_node(workflow, pos)
if src and isinstance(workflow.get(src), dict):
cls = workflow[src].get("class_type", "")
if cls.startswith("CLIPTextEncode") or cls in {"smZ CLIPTextEncode", "BNK_CLIPTextEncodeAdvanced"}:
return src
return None
def extract_schema(workflow: dict) -> dict:
"""Extract controllable parameters from a workflow.
Returns:
{
"parameters": { friendly_name: {node_id, field, type, value, ...} },
"output_nodes": [node_id, ...],
"model_dependencies": [{node_id, class_type, field, value, folder}],
"embedding_dependencies": [{node_id, embedding_name, found_in_field, value_excerpt}],
"summary": {...}
}
"""
output_nodes: list[str] = []
# First pass: identify positive / negative prompt nodes via connection tracing
pos_node = find_positive_prompt_node(workflow)
neg_node = find_negative_prompt_node(workflow)
# ----- collect raw parameter candidates -----
# Each candidate = (friendly_name, node_id, field, value)
# We resolve duplicate friendly_names AFTER the loop so dedup is symmetric.
raw_params: list[dict] = []
for node_id, node in iter_nodes(workflow):
cls = node["class_type"]
inputs = node.get("inputs", {}) or {}
if cls in OUTPUT_NODES:
output_nodes.append(node_id)
# Match this node against PARAM_PATTERNS
for p_class, p_field, friendly in PARAM_PATTERNS:
if cls != p_class:
continue
if p_field not in inputs:
continue
value = inputs[p_field]
t = infer_type(value)
if t == "link":
continue # connections aren't directly controllable
actual_name = friendly
# Disambiguate prompt vs negative_prompt by connection tracing
if friendly == "prompt":
if node_id == neg_node and pos_node != neg_node:
actual_name = "negative_prompt"
elif node_id == pos_node:
actual_name = "prompt"
else:
# Fallback: use _meta.title hints if present
meta_title = (node.get("_meta") or {}).get("title", "").lower()
if any(t_ in meta_title for t_ in ("negative", "neg", "-prompt", "anti")):
actual_name = "negative_prompt"
raw_params.append({
"name_hint": actual_name,
"node_id": node_id,
"field": p_field,
"type": t,
"value": value,
"class_type": cls,
})
# ----- symmetric duplicate-name resolution -----
# Group by name_hint. If a hint appears once, keep it. If multiple, suffix
# ALL with their node_id. Always-stable, always-uniquely-addressable.
by_name: dict[str, list[dict]] = {}
for r in raw_params:
by_name.setdefault(r["name_hint"], []).append(r)
parameters: dict[str, dict] = {}
for name, entries in by_name.items():
if len(entries) == 1:
r = entries[0]
parameters[name] = {
"node_id": r["node_id"], "field": r["field"],
"type": r["type"], "value": r["value"],
"class_type": r["class_type"],
}
else:
# Sort by node_id (string-natural) for stability
entries.sort(key=lambda x: (str(x["node_id"]).zfill(8), x["field"]))
for r in entries:
full_name = f"{name}_{r['node_id']}"
parameters[full_name] = {
"node_id": r["node_id"], "field": r["field"],
"type": r["type"], "value": r["value"],
"class_type": r["class_type"],
"alias_of": name,
}
# ----- model dependencies -----
model_deps = list(iter_model_deps(workflow))
# ----- embedding dependencies (in prompt text) -----
embedding_deps: list[dict] = []
seen_emb: set[tuple[str, str]] = set()
for nid, emb_name in iter_embedding_refs(workflow):
key = (nid, emb_name)
if key in seen_emb:
continue
seen_emb.add(key)
# Find which field had the reference, for context
node = workflow.get(nid, {})
inputs = node.get("inputs", {}) or {}
found_field = None
excerpt = None
for fname, fval in inputs.items():
if isinstance(fval, str) and fname in PROMPT_FIELDS and emb_name in fval:
found_field = fname
excerpt = fval[:120]
break
embedding_deps.append({
"node_id": nid,
"embedding_name": emb_name,
"field": found_field,
"value_excerpt": excerpt,
"folder": "embeddings",
})
# ----- summary -----
summary = {
"parameter_count": len(parameters),
"output_node_count": len(output_nodes),
"model_dep_count": len(model_deps),
"embedding_dep_count": len(embedding_deps),
"has_negative_prompt": "negative_prompt" in parameters,
"has_seed": "seed" in parameters or any(p.startswith("seed_") for p in parameters),
"is_video_workflow": any(
workflow.get(n, {}).get("class_type", "") in {
"VHS_VideoCombine", "SaveVideo", "SaveAnimatedWEBP", "SaveAnimatedPNG",
} for n in output_nodes
),
}
return {
"parameters": parameters,
"output_nodes": output_nodes,
"model_dependencies": model_deps,
"embedding_dependencies": embedding_deps,
"summary": summary,
}
def main(argv: list[str] | None = None) -> int:
p = argparse.ArgumentParser(description="Extract controllable parameters from a ComfyUI workflow")
p.add_argument("workflow", help="Path to workflow API JSON file")
p.add_argument("--output", "-o", help="Output file (default: stdout)")
p.add_argument("--summary-only", action="store_true",
help="Only print the summary block")
args = p.parse_args(argv)
wf_path = Path(args.workflow).expanduser()
if not wf_path.exists():
print(f"Error: {wf_path} not found", file=sys.stderr)
return 1
try:
with wf_path.open() as f:
payload = json.load(f)
workflow = unwrap_workflow(payload)
except ValueError as e:
print(f"Error: {e}", file=sys.stderr)
return 1
except json.JSONDecodeError as e:
print(f"Error: invalid JSON — {e}", file=sys.stderr)
return 1
schema = extract_schema(workflow)
if args.summary_only:
out = json.dumps(schema["summary"], indent=2)
else:
out = json.dumps(schema, indent=2, default=str)
if args.output:
Path(args.output).write_text(out)
print(f"Schema written to {args.output}", file=sys.stderr)
else:
print(out)
return 0
if __name__ == "__main__":
sys.exit(main())
+158
View File
@@ -0,0 +1,158 @@
#!/usr/bin/env python3
"""
fetch_logs.py — Retrieve workflow execution diagnostics from a ComfyUI server.
When a workflow errors, the server's /history (local) or /jobs (cloud) entry
contains the full Python traceback. This script makes it easy to fetch by
prompt_id, with sensible formatting.
Usage:
python3 fetch_logs.py <prompt_id>
python3 fetch_logs.py <prompt_id> --host https://cloud.comfy.org
python3 fetch_logs.py --tail-queue # show currently queued/running jobs
"""
from __future__ import annotations
import argparse
import json
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
from _common import ( # noqa: E402
DEFAULT_LOCAL_HOST, ENV_API_KEY, emit_json, http_get, is_cloud_host,
resolve_api_key, resolve_url,
)
def fetch_history_entry(host: str, headers: dict, prompt_id: str, *, is_cloud: bool) -> dict:
if is_cloud:
# Try /jobs/{id} first
url = resolve_url(host, f"/jobs/{prompt_id}", is_cloud=True)
r = http_get(url, headers=headers, retries=2, timeout=30)
if r.status == 200:
try:
return {"ok": True, "entry": r.json(), "source": "/api/jobs"}
except Exception:
pass
# Fallback to history_v2
url = resolve_url(host, f"/history/{prompt_id}", is_cloud=True)
r = http_get(url, headers=headers, retries=2, timeout=30)
try:
data = r.json()
except Exception:
data = None
if r.status == 200 and data:
return {"ok": True, "entry": data, "source": "/api/history_v2"}
return {"ok": False, "http_status": r.status, "body": r.text()[:500]}
url = resolve_url(host, f"/history/{prompt_id}", is_cloud=False)
r = http_get(url, headers=headers, retries=2, timeout=30)
if r.status != 200:
return {"ok": False, "http_status": r.status, "body": r.text()[:500]}
try:
data = r.json()
except Exception:
return {"ok": False, "reason": "non-JSON response"}
if not isinstance(data, dict) or prompt_id not in data:
return {"ok": False, "reason": "prompt_id not found in history",
"history_keys": list(data.keys())[:5] if isinstance(data, dict) else []}
return {"ok": True, "entry": data[prompt_id], "source": "/history"}
def fetch_queue(host: str, headers: dict) -> dict:
url = resolve_url(host, "/queue")
r = http_get(url, headers=headers, retries=2, timeout=15)
try:
data = r.json()
except Exception:
data = {"raw": r.text()[:500]}
return {"http_status": r.status, "data": data}
def extract_diagnostics(entry: dict) -> dict:
"""Pull out the parts a human cares about: status, errors, traceback, timing."""
diag: dict = {}
status = entry.get("status") or {}
diag["status_str"] = status.get("status_str")
diag["completed"] = status.get("completed")
messages = status.get("messages") or []
diag["execution_log"] = []
for msg in messages:
if isinstance(msg, list) and len(msg) >= 2:
mtype, mdata = msg[0], msg[1]
diag["execution_log"].append({"type": mtype, "data": mdata})
else:
diag["execution_log"].append(msg)
# Look for execution_error inside messages
errors = []
for msg in messages:
if isinstance(msg, list) and len(msg) >= 2 and msg[0] == "execution_error":
errors.append(msg[1])
if errors:
diag["errors"] = errors
# Cloud's /jobs response shape: top-level outputs / status / etc.
if "outputs" in entry:
out = entry["outputs"] or {}
if isinstance(out, dict):
diag["output_node_ids"] = list(out.keys())
# Count file refs across all output buckets (images / video / etc.)
total = 0
for node_output in out.values():
if not isinstance(node_output, dict):
continue
for v in node_output.values():
if isinstance(v, list):
total += len(v)
diag["output_count"] = total
else:
diag["output_node_ids"] = []
diag["output_count"] = 0
return diag
def main(argv: list[str] | None = None) -> int:
p = argparse.ArgumentParser(description="Fetch workflow execution diagnostics")
p.add_argument("prompt_id", nargs="?", help="prompt_id to look up")
p.add_argument("--host", default=DEFAULT_LOCAL_HOST)
p.add_argument("--api-key", help=f"or set ${ENV_API_KEY}")
p.add_argument("--raw", action="store_true",
help="Print the full history entry instead of the digest")
p.add_argument("--tail-queue", action="store_true",
help="Show currently running/pending jobs instead")
args = p.parse_args(argv)
api_key = resolve_api_key(args.api_key)
headers = {"X-API-Key": api_key} if api_key else {}
is_cloud = is_cloud_host(args.host)
if args.tail_queue:
emit_json(fetch_queue(args.host, headers))
return 0
if not args.prompt_id:
print("Error: prompt_id is required (or use --tail-queue)", file=sys.stderr)
return 1
res = fetch_history_entry(args.host, headers, args.prompt_id, is_cloud=is_cloud)
if not res.get("ok"):
emit_json(res)
return 1
if args.raw:
emit_json(res)
return 0
diag = extract_diagnostics(res["entry"])
diag["source"] = res.get("source")
diag["prompt_id"] = args.prompt_id
emit_json(diag)
return 0 if diag.get("status_str") not in {"error",} else 1
if __name__ == "__main__":
sys.exit(main())
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,198 @@
# Design System: Revolut
> **Hermes Agent — Implementation Notes**
>
> The original site uses proprietary fonts. For self-contained HTML output, use these CDN substitutes:
> - **Primary:** `Inter` | **Mono:** `system monospace stack`
> - **Font stack (CSS):** `font-family: 'Inter', system-ui, -apple-system, 'Segoe UI', Roboto, sans-serif;`
> - **Mono stack (CSS):** `font-family: ui-monospace, SFMono-Regular, Menlo, Monaco, Consolas, 'Liberation Mono', 'Courier New', monospace;`
> ```html
> <link href="https://fonts.googleapis.com/css2?family=Inter:wght@300;400;500;600;700&display=swap" rel="stylesheet">
> ```
> Use `write_file` to create HTML, serve via `generative-widgets` skill (cloudflared tunnel).
> Verify visual accuracy with `browser_vision` after generating.
## 1. Visual Theme & Atmosphere
Revolut's website is fintech confidence distilled into pixels — a design system that communicates "your money is in capable hands" through massive typography, generous whitespace, and a disciplined neutral palette. The visual language is built on Aeonik Pro, a geometric grotesque that creates billboard-scale headlines at 136px with weight 500 and aggressive negative tracking (-2.72px). This isn't subtle branding; it's fintech at stadium scale.
The color system is built on a comprehensive `--rui-*` (Revolut UI) token architecture with semantic naming for every state: danger (`#e23b4a`), warning (`#ec7e00`), teal (`#00a87e`), blue (`#494fdf`), deep-pink (`#e61e49`), and more. But the marketing surface itself is remarkably restrained — near-black (`#191c1f`) and pure white (`#ffffff`) dominate, with the colorful semantic tokens reserved for the product interface, not the marketing page.
What distinguishes Revolut is its pill-everything button system. Every button uses 9999px radius — primary dark (`#191c1f`), secondary light (`#f4f4f4`), outlined (`transparent + 2px solid`), and ghost on dark (`rgba(244,244,244,0.1) + 2px solid`). The padding is generous (14px 32px–34px), creating large, confident touch targets. Combined with Inter for body text at various weights and positive letter-spacing (0.16px–0.24px), the result is a design that feels both premium and accessible — banking for the modern era.
**Key Characteristics:**
- Aeonik Pro display at 136px weight 500 — billboard-scale fintech headlines
- Near-black (`#191c1f`) + white binary with comprehensive `--rui-*` semantic tokens
- Universal pill buttons (9999px radius) with generous padding (14px 32px)
- Inter for body text with positive letter-spacing (0.16px–0.24px)
- Rich semantic color system: blue, teal, pink, yellow, green, brown, danger, warning
- Zero shadows detected — depth through color contrast only
- Tight display line-heights (1.00) with relaxed body (1.50–1.56)
## 2. Color Palette & Roles
### Primary
- **Revolut Dark** (`#191c1f`): Primary dark surface, button background, near-black text
- **Pure White** (`#ffffff`): `--rui-color-action-label`, primary light surface
- **Light Surface** (`#f4f4f4`): Secondary button background, subtle surface
### Brand / Interactive
- **Revolut Blue** (`#494fdf`): `--rui-color-blue`, primary brand blue
- **Action Blue** (`#4f55f1`): `--rui-color-action-photo-header-text`, header accent
- **Blue Text** (`#376cd5`): `--website-color-blue-text`, link blue
### Semantic
- **Danger Red** (`#e23b4a`): `--rui-color-danger`, error/destructive
- **Deep Pink** (`#e61e49`): `--rui-color-deep-pink`, critical accent
- **Warning Orange** (`#ec7e00`): `--rui-color-warning`, warning states
- **Yellow** (`#b09000`): `--rui-color-yellow`, attention
- **Teal** (`#00a87e`): `--rui-color-teal`, success/positive
- **Light Green** (`#428619`): `--rui-color-light-green`, secondary success
- **Green Text** (`#006400`): `--website-color-green-text`, green text
- **Light Blue** (`#007bc2`): `--rui-color-light-blue`, informational
- **Brown** (`#936d62`): `--rui-color-brown`, warm neutral accent
- **Red Text** (`#8b0000`): `--website-color-red-text`, dark red text
### Neutral Scale
- **Mid Slate** (`#505a63`): Secondary text
- **Cool Gray** (`#8d969e`): Muted text, tertiary
- **Gray Tone** (`#c9c9cd`): `--rui-color-grey-tone-20`, borders/dividers
## 3. Typography Rules
### Font Families
- **Display**: `Aeonik Pro` — geometric grotesque, no detected fallbacks
- **Body / UI**: `Inter` — standard system sans
- **Fallback**: `Arial` for specific button contexts
### Hierarchy
| Role | Font | Size | Weight | Line Height | Letter Spacing | Notes |
|------|------|------|--------|-------------|----------------|-------|
| Display Mega | Aeonik Pro | 136px (8.50rem) | 500 | 1.00 (tight) | -2.72px | Stadium-scale hero |
| Display Hero | Aeonik Pro | 80px (5.00rem) | 500 | 1.00 (tight) | -0.8px | Primary hero |
| Section Heading | Aeonik Pro | 48px (3.00rem) | 500 | 1.21 (tight) | -0.48px | Feature sections |
| Sub-heading | Aeonik Pro | 40px (2.50rem) | 500 | 1.20 (tight) | -0.4px | Sub-sections |
| Card Title | Aeonik Pro | 32px (2.00rem) | 500 | 1.19 (tight) | -0.32px | Card headings |
| Feature Title | Aeonik Pro | 24px (1.50rem) | 400 | 1.33 | normal | Light headings |
| Nav / UI | Aeonik Pro | 20px (1.25rem) | 500 | 1.40 | normal | Navigation, buttons |
| Body Large | Inter | 18px (1.13rem) | 400 | 1.56 | -0.09px | Introductions |
| Body | Inter | 16px (1.00rem) | 400 | 1.50 | 0.24px | Standard reading |
| Body Semibold | Inter | 16px (1.00rem) | 600 | 1.50 | 0.16px | Emphasized body |
| Body Bold Link | Inter | 16px (1.00rem) | 700 | 1.50 | 0.24px | Bold links |
### Principles
- **Weight 500 as display default**: Aeonik Pro uses medium (500) for ALL headings — no bold. This creates authority through size and tracking, not weight.
- **Billboard tracking**: -2.72px at 136px is extremely compressed — text designed to be read at a glance, like airport signage.
- **Positive tracking on body**: Inter uses +0.16px to +0.24px, creating airy, well-spaced reading text that contrasts with the compressed headings.
## 4. Component Stylings
### Buttons
**Primary Dark Pill**
- Background: `#191c1f`
- Text: `#ffffff`
- Padding: 14px 32px
- Radius: 9999px (full pill)
- Hover: opacity 0.85
- Focus: `0 0 0 0.125rem` ring
**Secondary Light Pill**
- Background: `#f4f4f4`
- Text: `#000000`
- Padding: 14px 34px
- Radius: 9999px
- Hover: opacity 0.85
**Outlined Pill**
- Background: transparent
- Text: `#191c1f`
- Border: `2px solid #191c1f`
- Padding: 14px 32px
- Radius: 9999px
**Ghost on Dark**
- Background: `rgba(244, 244, 244, 0.1)`
- Text: `#f4f4f4`
- Border: `2px solid #f4f4f4`
- Padding: 14px 32px
- Radius: 9999px
### Cards & Containers
- Radius: 12px (small), 20px (cards)
- No shadows — flat surfaces with color contrast
- Dark and light section alternation
### Navigation
- Aeonik Pro 20px weight 500
- Clean header, hamburger toggle at 12px radius
- Pill CTAs right-aligned
## 5. Layout Principles
### Spacing System
- Base unit: 8px
- Scale: 4px, 6px, 8px, 14px, 16px, 20px, 24px, 32px, 40px, 48px, 80px, 88px, 120px
- Large section spacing: 80px–120px
### Border Radius Scale
- Standard (12px): Navigation, small buttons
- Card (20px): Feature cards
- Pill (9999px): All buttons
## 6. Depth & Elevation
| Level | Treatment | Use |
|-------|-----------|-----|
| Flat (Level 0) | No shadow | Everything — Revolut uses zero shadows |
| Focus | `0 0 0 0.125rem` ring | Accessibility focus |
**Shadow Philosophy**: Revolut uses ZERO shadows. Depth comes entirely from the dark/light section contrast and the generous whitespace between elements.
## 7. Do's and Don'ts
### Do
- Use Aeonik Pro weight 500 for all display headings
- Apply 9999px radius to all buttons — pill shape is universal
- Use generous button padding (14px 32px)
- Keep the palette to near-black + white for marketing surfaces
- Apply positive letter-spacing on Inter body text
### Don't
- Don't use shadows — Revolut is flat by design
- Don't use bold (700) for Aeonik Pro headings — 500 is the weight
- Don't use small buttons — the generous padding is intentional
- Don't apply semantic colors to marketing surfaces — they're for the product
## 8. Responsive Behavior
### Breakpoints
| Name | Width | Key Changes |
|------|-------|-------------|
| Mobile Small | <400px | Compact, single column |
| Mobile | 400–720px | Standard mobile |
| Tablet | 720–1024px | 2-column layouts |
| Desktop | 1024–1280px | Standard desktop |
| Large | 1280–1920px | Full layout |
## 9. Agent Prompt Guide
### Quick Color Reference
- Dark: Revolut Dark (`#191c1f`)
- Light: White (`#ffffff`)
- Surface: Light (`#f4f4f4`)
- Blue: Revolut Blue (`#494fdf`)
- Danger: Red (`#e23b4a`)
- Success: Teal (`#00a87e`)
### Example Component Prompts
- "Create a hero: white background. Headline at 136px Aeonik Pro weight 500, line-height 1.00, letter-spacing -2.72px, #191c1f text. Dark pill CTA (#191c1f, 9999px, 14px 32px). Outlined pill secondary (transparent, 2px solid #191c1f)."
- "Build a pill button: #191c1f background, white text, 9999px radius, 14px 32px padding, 20px Aeonik Pro weight 500. Hover: opacity 0.85."
### Iteration Guide
1. Aeonik Pro 500 for headings — never bold
2. All buttons are pills (9999px) with generous padding
3. Zero shadows — flat is the Revolut identity
4. Near-black + white for marketing, semantic colors for product
@@ -0,0 +1,3 @@
---
description: Skills for searching, downloading, and working with GIFs and short-form animated media.
---
@@ -0,0 +1,19 @@
# inference.sh
Run 150+ AI applications in the cloud via the [inference.sh](https://inference.sh) platform.
**One API key for everything** — access image generation, video creation, LLMs, search, 3D, and more through a single account. No need to manage separate API keys for each provider.
## Available Skills
- **cli**: Use the inference.sh CLI (`infsh`) via the terminal tool
## What's Included
- **Image Generation**: FLUX, Reve, Seedream, Grok Imagine, Gemini
- **Video Generation**: Veo, Wan, Seedance, OmniHuman, HunyuanVideo
- **LLMs**: Claude, Gemini, Kimi, GLM-4 (via OpenRouter)
- **Search**: Tavily, Exa
- **3D**: Rodin
- **Social**: Twitter/X automation
- **Audio**: TTS, voice cloning
+267
View File
@@ -0,0 +1,267 @@
#!/usr/bin/env python3
"""
ws_monitor.py — Real-time ComfyUI WebSocket monitor.
Connects to /ws and pretty-prints execution events: node start/finish, sampling
progress, cached nodes, errors. Optionally writes preview frames to disk.
Useful for:
- Watching a long-running job in real time without parsing JSON yourself
- Saving in-progress preview frames for video / animation workflows
- Debugging "why is this hanging?" — see exactly which node is stuck
Usage:
# Local — watch all jobs from this client_id
python3 ws_monitor.py
# Cloud — watch a specific prompt_id
python3 ws_monitor.py --host https://cloud.comfy.org \
--prompt-id abc-123-def
# Save preview frames to ./previews/
python3 ws_monitor.py --previews ./previews
Requires: websocket-client (`pip install websocket-client`).
Falls back to a clear error message when not installed.
"""
from __future__ import annotations
import argparse
import json
import struct
import sys
from pathlib import Path
from urllib.parse import urlparse
sys.path.insert(0, str(Path(__file__).resolve().parent))
from _common import ( # noqa: E402
DEFAULT_LOCAL_HOST, ENV_API_KEY, log, new_client_id, resolve_api_key, is_cloud_host,
)
# Binary frame types from ComfyUI WebSocket protocol
BINARY_PREVIEW_IMAGE = 1
BINARY_TEXT = 3
BINARY_PREVIEW_IMAGE_WITH_METADATA = 4
# Image type codes inside PREVIEW_IMAGE
IMAGE_TYPE_JPEG = 1
IMAGE_TYPE_PNG = 2
# ANSI escape codes (works on most modern terminals)
RESET = "\033[0m"
DIM = "\033[2m"
BOLD = "\033[1m"
GREEN = "\033[32m"
YELLOW = "\033[33m"
RED = "\033[31m"
CYAN = "\033[36m"
def fmt_color(s: str, color: str, *, color_on: bool = True) -> str:
return f"{color}{s}{RESET}" if color_on else s
def parse_binary_frame(data: bytes) -> dict | None:
if len(data) < 8:
return None
type_code = struct.unpack(">I", data[0:4])[0]
if type_code == BINARY_PREVIEW_IMAGE:
image_type = struct.unpack(">I", data[4:8])[0]
ext = "jpg" if image_type == IMAGE_TYPE_JPEG else "png" if image_type == IMAGE_TYPE_PNG else "bin"
return {
"kind": "preview",
"image_type": image_type,
"ext": ext,
"image_bytes": data[8:],
}
if type_code == BINARY_PREVIEW_IMAGE_WITH_METADATA:
if len(data) < 12:
return None
meta_len = struct.unpack(">I", data[4:8])[0]
meta_end = 8 + meta_len
if len(data) < meta_end:
return None
try:
meta = json.loads(data[8:meta_end].decode("utf-8"))
except Exception:
meta = {"raw": data[8:meta_end][:200].decode("utf-8", "replace")}
return {
"kind": "preview_with_metadata",
"metadata": meta,
"image_bytes": data[meta_end:],
"ext": "png",
}
if type_code == BINARY_TEXT:
if len(data) < 8:
return None
nid_len = struct.unpack(">I", data[4:8])[0]
nid_end = 8 + nid_len
if len(data) < nid_end:
return None
return {
"kind": "text",
"node_id": data[8:nid_end].decode("utf-8", "replace"),
"text": data[nid_end:].decode("utf-8", "replace"),
}
return {"kind": "unknown", "type_code": type_code, "size": len(data)}
def main(argv: list[str] | None = None) -> int:
p = argparse.ArgumentParser(description="Real-time ComfyUI WebSocket monitor")
p.add_argument("--host", default=DEFAULT_LOCAL_HOST, help="ComfyUI server URL")
p.add_argument("--api-key", help=f"API key for cloud (or set ${ENV_API_KEY} env var)")
p.add_argument("--client-id", default=None, help="Client ID (default: random UUID)")
p.add_argument("--prompt-id", default=None,
help="Filter to a specific prompt_id (default: all jobs)")
p.add_argument("--previews", default=None,
help="Directory to save in-progress preview frames")
p.add_argument("--no-color", action="store_true", help="Disable ANSI colour")
p.add_argument("--timeout", type=float, default=600.0,
help="Hard cap on monitor duration (default 600s)")
args = p.parse_args(argv)
try:
import websocket # type: ignore[import-not-found]
except ImportError:
print(json.dumps({
"error": "websocket-client not installed",
"install": "pip install websocket-client",
}))
return 1
api_key = resolve_api_key(args.api_key)
cloud = is_cloud_host(args.host)
client_id = args.client_id or new_client_id()
# Build WS URL preserving any base-path component (e.g. behind reverse proxy).
parsed = urlparse(args.host if "://" in args.host else f"http://{args.host}")
scheme = "wss" if parsed.scheme == "https" else "ws"
netloc = parsed.netloc
base_path = parsed.path.rstrip("/")
ws_url = f"{scheme}://{netloc}{base_path}/ws?clientId={client_id}"
if cloud and api_key:
ws_url += f"&token={api_key}"
color_on = not args.no_color and sys.stdout.isatty()
preview_dir = Path(args.previews).expanduser() if args.previews else None
if preview_dir:
preview_dir.mkdir(parents=True, exist_ok=True)
log(f"Saving previews to {preview_dir}")
log(f"Connecting to {ws_url} (client_id={client_id})")
if args.prompt_id:
log(f"Filtering messages to prompt_id={args.prompt_id}")
ws = websocket.create_connection(ws_url, timeout=args.timeout)
ws.settimeout(args.timeout)
preview_counter = 0
try:
while True:
try:
msg = ws.recv()
except websocket.WebSocketTimeoutException:
log(f"Idle for {args.timeout}s — exiting")
return 0
if isinstance(msg, bytes):
parsed = parse_binary_frame(msg)
if parsed is None:
continue
if parsed["kind"] in {"preview", "preview_with_metadata"} and preview_dir:
img_bytes = parsed.get("image_bytes", b"")
if img_bytes:
ext = parsed.get("ext", "png")
out = preview_dir / f"preview_{preview_counter:05d}.{ext}"
out.write_bytes(img_bytes)
preview_counter += 1
log(f" [preview] saved {out.name} ({len(img_bytes)} bytes)")
continue
try:
payload = json.loads(msg)
except Exception:
continue
mtype = payload.get("type", "")
mdata = payload.get("data", {}) or {}
pid = mdata.get("prompt_id")
if args.prompt_id and pid and pid != args.prompt_id:
continue
if mtype == "status":
qr = mdata.get("status", {}).get("exec_info", {}).get("queue_remaining", "?")
print(fmt_color(f"[status] queue_remaining={qr}", DIM, color_on=color_on))
elif mtype == "execution_start":
print(fmt_color(f"[start] prompt_id={pid}", BOLD, color_on=color_on))
elif mtype == "executing":
node = mdata.get("node")
if node:
print(fmt_color(f" [executing] node={node}", CYAN, color_on=color_on))
else:
print(fmt_color(f" [executing] (workflow done) prompt_id={pid}", DIM, color_on=color_on))
elif mtype == "progress":
v, m = mdata.get("value", 0), mdata.get("max", 0)
pct = (v / m * 100) if m else 0
print(f" [progress] {v}/{m} ({pct:5.1f}%) node={mdata.get('node')}")
elif mtype == "progress_state":
# Newer extended progress message
nodes = mdata.get("nodes") or {}
running = [k for k, v in nodes.items() if v.get("running")]
if running:
print(fmt_color(f" [progress_state] running={running}", DIM, color_on=color_on))
elif mtype == "executed":
node = mdata.get("node")
out = mdata.get("output") or {}
summary_parts = []
for key in ("images", "video", "videos", "gifs", "audio", "files"):
if out.get(key):
summary_parts.append(f"{key}={len(out[key])}")
summary = ", ".join(summary_parts) if summary_parts else "(no files)"
print(fmt_color(f" [executed] node={node} {summary}", GREEN, color_on=color_on))
elif mtype == "execution_cached":
cached = mdata.get("nodes") or []
if cached:
print(fmt_color(f" [cached] {len(cached)} nodes skipped", DIM, color_on=color_on))
elif mtype == "execution_success":
print(fmt_color(f"[success] prompt_id={pid}", GREEN + BOLD, color_on=color_on))
if args.prompt_id:
return 0
elif mtype == "execution_error":
exc_type = mdata.get("exception_type", "?")
exc_msg = mdata.get("exception_message", "?")
print(fmt_color(f"[error] {exc_type}: {exc_msg}", RED + BOLD, color_on=color_on))
tb = mdata.get("traceback")
if tb:
if isinstance(tb, list):
for line in tb:
print(fmt_color(f" {line}", RED, color_on=color_on))
else:
print(fmt_color(f" {tb}", RED, color_on=color_on))
if args.prompt_id:
return 1
elif mtype == "execution_interrupted":
print(fmt_color(f"[interrupted] prompt_id={pid}", YELLOW, color_on=color_on))
if args.prompt_id:
return 1
elif mtype == "notification":
v = mdata.get("value", "")
print(fmt_color(f"[notification] {v}", DIM, color_on=color_on))
else:
# Unknown / lightly-used types: print compactly
print(fmt_color(f"[{mtype}] {json.dumps(mdata, default=str)[:200]}", DIM, color_on=color_on))
except KeyboardInterrupt:
log("Interrupted")
return 130
finally:
try:
ws.close()
except Exception:
pass
if __name__ == "__main__":
sys.exit(main())
+64
View File
@@ -0,0 +1,64 @@
"""Pytest configuration for the comfyui skill test suite.
Adds `scripts/` to sys.path so tests can `from _common import ...`, and
provides a few common fixtures.
"""
from __future__ import annotations
import json
import os
import sys
from pathlib import Path
import pytest
ROOT = Path(__file__).resolve().parent.parent
SCRIPTS = ROOT / "scripts"
WORKFLOWS = ROOT / "workflows"
sys.path.insert(0, str(SCRIPTS))
@pytest.fixture
def sd15_workflow() -> dict:
return json.loads((WORKFLOWS / "sd15_txt2img.json").read_text())
@pytest.fixture
def flux_workflow() -> dict:
return json.loads((WORKFLOWS / "flux_dev_txt2img.json").read_text())
@pytest.fixture
def video_workflow() -> dict:
return json.loads((WORKFLOWS / "wan_video_t2v.json").read_text())
@pytest.fixture
def workflows_dir() -> Path:
return WORKFLOWS
@pytest.fixture
def scripts_dir() -> Path:
return SCRIPTS
@pytest.fixture
def cloud_key() -> str | None:
"""Cloud API key if set, otherwise None.
Tests that need cloud connectivity should skip when this is None.
"""
return os.environ.get("COMFY_CLOUD_API_KEY")
def pytest_collection_modifyitems(config, items):
"""Auto-skip cloud tests when no API key is set."""
if os.environ.get("COMFY_CLOUD_API_KEY"):
return
skip_cloud = pytest.mark.skip(reason="Set COMFY_CLOUD_API_KEY to run cloud tests")
for item in items:
if "cloud" in item.keywords:
item.add_marker(skip_cloud)
@@ -0,0 +1,162 @@
---
name: dogfood
description: "Exploratory QA of web apps: find bugs, evidence, reports."
version: 1.0.0
platforms: [linux, macos, windows]
metadata:
hermes:
tags: [qa, testing, browser, web, dogfood]
related_skills: []
---
# Dogfood: Systematic Web Application QA Testing
## Overview
This skill guides you through systematic exploratory QA testing of web applications using the browser toolset. You will navigate the application, interact with elements, capture evidence of issues, and produce a structured bug report.
## Prerequisites
- Browser toolset must be available (`browser_navigate`, `browser_snapshot`, `browser_click`, `browser_type`, `browser_vision`, `browser_console`, `browser_scroll`, `browser_back`, `browser_press`)
- A target URL and testing scope from the user
## Inputs
The user provides:
1. **Target URL** — the entry point for testing
2. **Scope** — what areas/features to focus on (or "full site" for comprehensive testing)
3. **Output directory** (optional) — where to save screenshots and the report (default: `./dogfood-output`)
## Workflow
Follow this 5-phase systematic workflow:
### Phase 1: Plan
1. Create the output directory structure:
```
{output_dir}/
├── screenshots/ # Evidence screenshots
└── report.md # Final report (generated in Phase 5)
```
2. Identify the testing scope based on user input.
3. Build a rough sitemap by planning which pages and features to test:
- Landing/home page
- Navigation links (header, footer, sidebar)
- Key user flows (sign up, login, search, checkout, etc.)
- Forms and interactive elements
- Edge cases (empty states, error pages, 404s)
### Phase 2: Explore
For each page or feature in your plan:
1. **Navigate** to the page:
```
browser_navigate(url="https://example.com/page")
```
2. **Take a snapshot** to understand the DOM structure:
```
browser_snapshot()
```
3. **Check the console** for JavaScript errors:
```
browser_console(clear=true)
```
Do this after every navigation and after every significant interaction. Silent JS errors are high-value findings.
4. **Take an annotated screenshot** to visually assess the page and identify interactive elements:
```
browser_vision(question="Describe the page layout, identify any visual issues, broken elements, or accessibility concerns", annotate=true)
```
The `annotate=true` flag overlays numbered `[N]` labels on interactive elements. Each `[N]` maps to ref `@eN` for subsequent browser commands.
5. **Test interactive elements** systematically:
- Click buttons and links: `browser_click(ref="@eN")`
- Fill forms: `browser_type(ref="@eN", text="test input")`
- Test keyboard navigation: `browser_press(key="Tab")`, `browser_press(key="Enter")`
- Scroll through content: `browser_scroll(direction="down")`
- Test form validation with invalid inputs
- Test empty submissions
6. **After each interaction**, check for:
- Console errors: `browser_console()`
- Visual changes: `browser_vision(question="What changed after the interaction?")`
- Expected vs actual behavior
### Phase 3: Collect Evidence
For every issue found:
1. **Take a screenshot** showing the issue:
```
browser_vision(question="Capture and describe the issue visible on this page", annotate=false)
```
Save the `screenshot_path` from the response — you will reference it in the report.
2. **Record the details**:
- URL where the issue occurs
- Steps to reproduce
- Expected behavior
- Actual behavior
- Console errors (if any)
- Screenshot path
3. **Classify the issue** using the issue taxonomy (see `references/issue-taxonomy.md`):
- Severity: Critical / High / Medium / Low
- Category: Functional / Visual / Accessibility / Console / UX / Content
### Phase 4: Categorize
1. Review all collected issues.
2. De-duplicate — merge issues that are the same bug manifesting in different places.
3. Assign final severity and category to each issue.
4. Sort by severity (Critical first, then High, Medium, Low).
5. Count issues by severity and category for the executive summary.
### Phase 5: Report
Generate the final report using the template at `templates/dogfood-report-template.md`.
The report must include:
1. **Executive summary** with total issue count, breakdown by severity, and testing scope
2. **Per-issue sections** with:
- Issue number and title
- Severity and category badges
- URL where observed
- Description of the issue
- Steps to reproduce
- Expected vs actual behavior
- Screenshot references (use `MEDIA:<screenshot_path>` for inline images)
- Console errors if relevant
3. **Summary table** of all issues
4. **Testing notes** — what was tested, what was not, any blockers
Save the report to `{output_dir}/report.md`.
## Tools Reference
| Tool | Purpose |
|------|---------|
| `browser_navigate` | Go to a URL |
| `browser_snapshot` | Get DOM text snapshot (accessibility tree) |
| `browser_click` | Click an element by ref (`@eN`) or text |
| `browser_type` | Type into an input field |
| `browser_scroll` | Scroll up/down on the page |
| `browser_back` | Go back in browser history |
| `browser_press` | Press a keyboard key |
| `browser_vision` | Screenshot + AI analysis; use `annotate=true` for element labels |
| `browser_console` | Get JS console output and errors |
## Tips
- **Always check `browser_console()` after navigating and after significant interactions.** Silent JS errors are among the most valuable findings.
- **Use `annotate=true` with `browser_vision`** when you need to reason about interactive element positions or when the snapshot refs are unclear.
- **Test with both valid and invalid inputs** — form validation bugs are common.
- **Scroll through long pages** — content below the fold may have rendering issues.
- **Test navigation flows** — click through multi-step processes end-to-end.
- **Check responsive behavior** by noting any layout issues visible in screenshots.
- **Don't forget edge cases**: empty states, very long text, special characters, rapid clicking.
- When reporting screenshots to the user, include `MEDIA:<screenshot_path>` so they can see the evidence inline.
@@ -0,0 +1,226 @@
# Citation Management & Hallucination Prevention
This reference provides a complete workflow for managing citations programmatically, preventing AI-generated citation hallucinations, and maintaining clean bibliographies.
---
## Contents
- [Why Citation Verification Matters](#why-citation-verification-matters)
- [Citation APIs Overview](#citation-apis-overview)
- [Verified Citation Workflow](#verified-citation-workflow)
- [Python Implementation](#python-implementation)
- [BibTeX Management](#bibtex-management)
- [Common Citation Formats](#common-citation-formats)
- [Troubleshooting](#troubleshooting)
---
## Why Citation Verification Matters
### The Hallucination Problem
Research has documented significant issues with AI-generated citations:
- **~40% error rate** in AI-generated citations (Enago Academy research)
- NeurIPS 2025 found **100+ hallucinated citations** slipped through review
- Common errors include:
- Fabricated paper titles with real author names
- Wrong publication venues or years
- Non-existent papers with plausible metadata
- Incorrect DOIs or arXiv IDs
### Consequences
- Desk rejection at some venues
- Loss of credibility with reviewers
- Potential retraction if published
- Wasted time chasing non-existent sources
### Solution
**Never generate citations from memory—always verify programmatically.**
---
## Citation APIs Overview
### Primary APIs
| API | Coverage | Rate Limits | Best For |
|-----|----------|-------------|----------|
| **Semantic Scholar** | 214M papers | 1 RPS (free key) | ML/AI papers, citation graphs |
| **CrossRef** | 140M+ DOIs | Polite pool with mailto | DOI lookup, BibTeX retrieval |
| **arXiv** | Preprints | 3-second delays | ML preprints, PDF access |
| **OpenAlex** | 240M+ works | 100K/day, 10 RPS | Open alternative to MAG |
### API Selection Guide
```
Need ML paper search? → Semantic Scholar
Have DOI, need BibTeX? → CrossRef content negotiation
Looking for preprint? → arXiv API
Need open data, bulk access? → OpenAlex
```
### No Official Google Scholar API
Google Scholar has no official API. Scraping violates ToS. Use SerpApi ($75-275/month) only if Semantic Scholar coverage is insufficient.
---
## Verified Citation Workflow
### 5-Step Process
```
1. SEARCH → Query Semantic Scholar with specific keywords
↓
2. VERIFY → Confirm paper exists in 2+ sources
↓
3. RETRIEVE → Get BibTeX via DOI content negotiation
↓
4. VALIDATE → Confirm the claim appears in source
↓
5. ADD → Add verified entry to .bib file
```
### Step 1: Search
Use Semantic Scholar for ML/AI papers:
```python
from semanticscholar import SemanticScholar
sch = SemanticScholar()
results = sch.search_paper("transformer attention mechanism", limit=10)
for paper in results:
print(f"Title: {paper.title}")
print(f"Year: {paper.year}")
print(f"DOI: {paper.externalIds.get('DOI', 'N/A')}")
print(f"arXiv: {paper.externalIds.get('ArXiv', 'N/A')}")
print(f"Citation count: {paper.citationCount}")
print("---")
```
### Step 2: Verify Existence
Confirm paper exists in at least two sources:
```python
import requests
def verify_paper(doi=None, arxiv_id=None, title=None):
"""Verify paper exists in multiple sources."""
sources_found = []
# Check Semantic Scholar
sch = SemanticScholar()
if doi:
paper = sch.get_paper(f"DOI:{doi}")
if paper:
sources_found.append("Semantic Scholar")
# Check CrossRef (via DOI)
if doi:
resp = requests.get(f"https://api.crossref.org/works/{doi}")
if resp.status_code == 200:
sources_found.append("CrossRef")
# Check arXiv
if arxiv_id:
resp = requests.get(
f"http://export.arxiv.org/api/query?id_list={arxiv_id}"
)
if "<entry>" in resp.text:
sources_found.append("arXiv")
return len(sources_found) >= 2, sources_found
```
### Step 3: Retrieve BibTeX
Use DOI content negotiation for guaranteed accuracy:
```python
import requests
def doi_to_bibtex(doi: str) -> str:
"""Get verified BibTeX from DOI via CrossRef content negotiation."""
response = requests.get(
f"https://doi.org/{doi}",
headers={"Accept": "application/x-bibtex"},
allow_redirects=True
)
response.raise_for_status()
return response.text
# Example: "Attention Is All You Need"
bibtex = doi_to_bibtex("10.48550/arXiv.1706.03762")
print(bibtex)
```
### Step 4: Validate Claims
Before citing a paper for a specific claim, verify the claim exists:
```python
def get_paper_abstract(doi):
"""Get abstract to verify claims."""
sch = SemanticScholar()
paper = sch.get_paper(f"DOI:{doi}")
return paper.abstract if paper else None
# Verify claim appears in abstract
abstract = get_paper_abstract("10.48550/arXiv.1706.03762")
claim = "attention mechanism"
if claim.lower() in abstract.lower():
print("Claim appears in paper")
```
### Step 5: Add to Bibliography
Add verified entry to your .bib file with consistent key format:
```python
def generate_citation_key(bibtex: str) -> str:
"""Generate consistent citation key: author_year_firstword."""
import re
# Extract author
author_match = re.search(r'author\s*=\s*\{([^}]+)\}', bibtex, re.I)
if author_match:
first_author = author_match.group(1).split(',')[0].split()[-1]
else:
first_author = "unknown"
# Extract year
year_match = re.search(r'year\s*=\s*\{?(\d{4})\}?', bibtex, re.I)
year = year_match.group(1) if year_match else "0000"
# Extract title first word
title_match = re.search(r'title\s*=\s*\{([^}]+)\}', bibtex, re.I)
if title_match:
first_word = title_match.group(1).split()[0].lower()
first_word = re.sub(r'[^a-z]', '', first_word)
else:
first_word = "paper"
return f"{first_author.lower()}_{year}_{first_word}"
```
---
## Python Implementation
### Complete Citation Manager Class
{% raw %}
```python
"""
Citation Manager - Verified citation workflow for ML papers.
"""
import requests
import time
from typing import Opt
@@ -0,0 +1,24 @@
---
name: domain-intel
description: Passive domain reconnaissance using Python stdlib. Use this skill for subdomain discovery, SSL certificate inspection, WHOIS lookups, DNS records, domain availability checks, and bulk multi-domain analysis. No API keys required. Triggers on requests like "find subdomains", "check ssl cert", "whois lookup", "is this domain available", "bulk check these domains".
license: MIT
---
Passive domain intelligence using only Python stdlib and public data sources.
Zero dependencies. Zero API keys. Works out of the box.
## Capabilities
- Subdomain discovery via crt.sh certificate transparency logs
- Live SSL/TLS certificate inspection (expiry, cipher, SANs, TLS version)
- WHOIS lookup — supports 100+ TLDs via direct TCP queries
- DNS records: A, AAAA, MX, NS, TXT, CNAME
- Domain availability check (DNS + WHOIS + SSL signals)
- Bulk multi-domain analysis in parallel (up to 20 domains)
## Data Sources
- crt.sh — Certificate Transparency logs
- WHOIS servers — Direct TCP to 100+ authoritative TLD servers
- Google DNS-over-HTTPS — MX/NS/TXT/CNAME resolution
- System DNS — A/AAAA records
@@ -0,0 +1,184 @@
"""Tests for extract_schema.py."""
from __future__ import annotations
from extract_schema import (
extract_schema,
find_negative_prompt_node,
find_positive_prompt_node,
trace_to_node,
)
# =============================================================================
# Connection tracing
# =============================================================================
class TestConnectionTracing:
def test_direct_link(self):
wf = {
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "x"}},
"2": {"class_type": "KSampler",
"inputs": {"positive": ["1", 0], "negative": ["1", 0]}},
}
assert trace_to_node(wf, ["1", 0]) == "1"
def test_through_reroute(self):
wf = {
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "x"}},
"2": {"class_type": "Reroute", "inputs": {"input": ["1", 0]}},
"3": {"class_type": "Reroute", "inputs": {"input": ["2", 0]}},
}
assert trace_to_node(wf, ["3", 0]) == "1"
def test_circular_safe(self):
wf = {
"1": {"class_type": "Reroute", "inputs": {"input": ["2", 0]}},
"2": {"class_type": "Reroute", "inputs": {"input": ["1", 0]}},
}
# Should hit max_hops without infinite loop
result = trace_to_node(wf, ["1", 0], max_hops=5)
assert result in {"1", "2"} # any node, just don't hang
class TestPositiveNegativeDetection:
def test_basic(self, sd15_workflow):
# In sd15_workflow.json node 6 is positive, node 7 is negative
assert find_positive_prompt_node(sd15_workflow) == "6"
assert find_negative_prompt_node(sd15_workflow) == "7"
def test_swapped_order(self):
wf = {
"3": {"class_type": "KSampler",
"inputs": {
"positive": ["7", 0], "negative": ["6", 0],
"model": ["4", 0], "latent_image": ["5", 0],
"seed": 1, "steps": 20, "cfg": 7.5,
"sampler_name": "euler", "scheduler": "normal", "denoise": 1.0,
}},
"4": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "x"}},
"5": {"class_type": "EmptyLatentImage", "inputs": {"width": 512, "height": 512, "batch_size": 1}},
"6": {"class_type": "CLIPTextEncode", "inputs": {"text": "ugly", "clip": ["4", 1]}},
"7": {"class_type": "CLIPTextEncode", "inputs": {"text": "beautiful", "clip": ["4", 1]}},
}
# Now 7 is the positive (despite higher node ID)
assert find_positive_prompt_node(wf) == "7"
assert find_negative_prompt_node(wf) == "6"
# =============================================================================
# Schema extraction
# =============================================================================
class TestExtractSchema:
def test_basic_sd15(self, sd15_workflow):
schema = extract_schema(sd15_workflow)
params = schema["parameters"]
assert "prompt" in params
assert "negative_prompt" in params
assert "seed" in params
assert "steps" in params
assert "cfg" in params
assert "width" in params
assert "height" in params
def test_prompt_value_correct(self, sd15_workflow):
schema = extract_schema(sd15_workflow)
# The positive prompt in the example is the landscape one
assert "landscape" in schema["parameters"]["prompt"]["value"]
assert "ugly" in schema["parameters"]["negative_prompt"]["value"]
def test_model_dependencies(self, sd15_workflow):
schema = extract_schema(sd15_workflow)
deps = schema["model_dependencies"]
ckpts = [d["value"] for d in deps if d["folder"] == "checkpoints"]
assert "v1-5-pruned-emaonly.safetensors" in ckpts
def test_output_nodes(self, sd15_workflow):
schema = extract_schema(sd15_workflow)
assert "9" in schema["output_nodes"]
def test_summary(self, sd15_workflow):
schema = extract_schema(sd15_workflow)
s = schema["summary"]
assert s["has_negative_prompt"] is True
assert s["has_seed"] is True
assert s["is_video_workflow"] is False
assert s["parameter_count"] > 5
def test_flux_workflow(self, flux_workflow):
schema = extract_schema(flux_workflow)
# Flux uses RandomNoise for seed
assert schema["summary"]["has_seed"] is True
# Flux has only positive prompt (no negative encoder)
assert schema["summary"]["has_negative_prompt"] is False
def test_video_detected(self, video_workflow):
schema = extract_schema(video_workflow)
assert schema["summary"]["is_video_workflow"] is True
class TestEmbeddingDeps:
def test_extract_from_prompt(self):
wf = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "x"}},
"5": {"class_type": "EmptyLatentImage",
"inputs": {"width": 512, "height": 512, "batch_size": 1}},
"6": {"class_type": "CLIPTextEncode",
"inputs": {
"text": "a cat, embedding:goodvibes, embedding:art:1.2",
"clip": ["1", 1]
}},
"7": {"class_type": "CLIPTextEncode",
"inputs": {
"text": "ugly, embedding:badhands",
"clip": ["1", 1]
}},
"3": {"class_type": "KSampler",
"inputs": {
"positive": ["6", 0], "negative": ["7", 0],
"model": ["1", 0], "latent_image": ["5", 0],
"seed": 1, "steps": 20, "cfg": 7.5,
"sampler_name": "euler", "scheduler": "normal", "denoise": 1.0,
}},
"9": {"class_type": "SaveImage", "inputs": {"filename_prefix": "x", "images": ["3", 0]}},
}
schema = extract_schema(wf)
names = [d["embedding_name"] for d in schema["embedding_dependencies"]]
assert sorted(names) == ["art", "badhands", "goodvibes"]
class TestDuplicateDeduplication:
def test_two_ksamplers_get_unique_names(self):
wf = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "x"}},
"5": {"class_type": "EmptyLatentImage",
"inputs": {"width": 512, "height": 512, "batch_size": 1}},
"6": {"class_type": "CLIPTextEncode", "inputs": {"text": "a", "clip": ["1", 1]}},
"7": {"class_type": "CLIPTextEncode", "inputs": {"text": "b", "clip": ["1", 1]}},
"3": {"class_type": "KSampler",
"inputs": {
"positive": ["6", 0], "negative": ["7", 0],
"model": ["1", 0], "latent_image": ["5", 0],
"seed": 42, "steps": 20, "cfg": 7.5,
"sampler_name": "euler", "scheduler": "normal", "denoise": 1.0,
}},
"4": {"class_type": "KSampler",
"inputs": {
"positive": ["6", 0], "negative": ["7", 0],
"model": ["1", 0], "latent_image": ["5", 0],
"seed": 99, "steps": 30, "cfg": 8.0,
"sampler_name": "euler", "scheduler": "normal", "denoise": 0.6,
}},
"9": {"class_type": "SaveImage", "inputs": {"filename_prefix": "x", "images": ["3", 0]}},
}
schema = extract_schema(wf)
params = schema["parameters"]
# Both seeds present with disambiguated names
seed_keys = [k for k in params if "seed" in k]
# Symmetric: both renamed (no bare "seed")
assert "seed" not in params
assert "seed_3" in params and "seed_4" in params
assert params["seed_3"]["value"] == 42
assert params["seed_4"]["value"] == 99
@@ -0,0 +1,36 @@
FIGURES_FOLDER := figures
PDFS := \
$(filter-out $(wildcard $(FIGURES_FOLDER)/*-crop.pdf),$(wildcard $(FIGURES_FOLDER)/*.pdf)) \
$(filter-out $(wildcard $(FIGURES_FOLDER)/**/*-crop.pdf),$(wildcard $(FIGURES_FOLDER)/**/*.pdf))
CROPPED_PDFS := $(PDFS:.pdf=-crop.pdf)
all: main.pdf
%.pdf: %.tex Makefile $(CROPPED_PDFS)
pdflatex -synctex=1 -interaction=nonstopmode $<
-bibtex $*.aux
pdflatex -synctex=1 -interaction=nonstopmode $<
pdflatex -synctex=1 -interaction=nonstopmode $<
.PHONY: figures
figures: $(CROPPED_PDFS)
.PRECIOUS: $(CROPPED_PDFS)
%-crop.pdf: %.pdf Makefile
pdfcrop $<
.PHONY: clean upgrade
clean:
find . -maxdepth 1 \
\( -name "*.aux" -o -name "*.bbl" -o -name "*.blg" -o \
-name "*.log" -o -name "*.out" -o -name "*.pdf" -o \
-name "*.synctex.gz" \) | xargs $(RM)
find $(FIGURES_FOLDER) -name "*-crop.pdf" | xargs $(RM)
YEAR := 2025
upgrade:
curl -O https://media.neurips.cc/Conferences/NeurIPS$(YEAR)/Styles.zip
unzip -u Styles.zip
mv Styles/neurips_${YEAR}.sty neurips.sty
$(RM) -r Styles.zip Styles
@@ -0,0 +1,185 @@
"""Tests for extract_schema.py."""
from __future__ import annotations
import pytest
from extract_schema import (
extract_schema,
find_negative_prompt_node,
find_positive_prompt_node,
trace_to_node,
)
# =============================================================================
# Connection tracing
# =============================================================================
class TestConnectionTracing:
def test_direct_link(self):
wf = {
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "x"}},
"2": {"class_type": "KSampler",
"inputs": {"positive": ["1", 0], "negative": ["1", 0]}},
}
assert trace_to_node(wf, ["1", 0]) == "1"
def test_through_reroute(self):
wf = {
"1": {"class_type": "CLIPTextEncode", "inputs": {"text": "x"}},
"2": {"class_type": "Reroute", "inputs": {"input": ["1", 0]}},
"3": {"class_type": "Reroute", "inputs": {"input": ["2", 0]}},
}
assert trace_to_node(wf, ["3", 0]) == "1"
def test_circular_safe(self):
wf = {
"1": {"class_type": "Reroute", "inputs": {"input": ["2", 0]}},
"2": {"class_type": "Reroute", "inputs": {"input": ["1", 0]}},
}
# Should hit max_hops without infinite loop
result = trace_to_node(wf, ["1", 0], max_hops=5)
assert result in {"1", "2"} # any node, just don't hang
class TestPositiveNegativeDetection:
def test_basic(self, sd15_workflow):
# In sd15_workflow.json node 6 is positive, node 7 is negative
assert find_positive_prompt_node(sd15_workflow) == "6"
assert find_negative_prompt_node(sd15_workflow) == "7"
def test_swapped_order(self):
wf = {
"3": {"class_type": "KSampler",
"inputs": {
"positive": ["7", 0], "negative": ["6", 0],
"model": ["4", 0], "latent_image": ["5", 0],
"seed": 1, "steps": 20, "cfg": 7.5,
"sampler_name": "euler", "scheduler": "normal", "denoise": 1.0,
}},
"4": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "x"}},
"5": {"class_type": "EmptyLatentImage", "inputs": {"width": 512, "height": 512, "batch_size": 1}},
"6": {"class_type": "CLIPTextEncode", "inputs": {"text": "ugly", "clip": ["4", 1]}},
"7": {"class_type": "CLIPTextEncode", "inputs": {"text": "beautiful", "clip": ["4", 1]}},
}
# Now 7 is the positive (despite higher node ID)
assert find_positive_prompt_node(wf) == "7"
assert find_negative_prompt_node(wf) == "6"
# =============================================================================
# Schema extraction
# =============================================================================
class TestExtractSchema:
def test_basic_sd15(self, sd15_workflow):
schema = extract_schema(sd15_workflow)
params = schema["parameters"]
assert "prompt" in params
assert "negative_prompt" in params
assert "seed" in params
assert "steps" in params
assert "cfg" in params
assert "width" in params
assert "height" in params
def test_prompt_value_correct(self, sd15_workflow):
schema = extract_schema(sd15_workflow)
# The positive prompt in the example is the landscape one
assert "landscape" in schema["parameters"]["prompt"]["value"]
assert "ugly" in schema["parameters"]["negative_prompt"]["value"]
def test_model_dependencies(self, sd15_workflow):
schema = extract_schema(sd15_workflow)
deps = schema["model_dependencies"]
ckpts = [d["value"] for d in deps if d["folder"] == "checkpoints"]
assert "v1-5-pruned-emaonly.safetensors" in ckpts
def test_output_nodes(self, sd15_workflow):
schema = extract_schema(sd15_workflow)
assert "9" in schema["output_nodes"]
def test_summary(self, sd15_workflow):
schema = extract_schema(sd15_workflow)
s = schema["summary"]
assert s["has_negative_prompt"] is True
assert s["has_seed"] is True
assert s["is_video_workflow"] is False
assert s["parameter_count"] > 5
def test_flux_workflow(self, flux_workflow):
schema = extract_schema(flux_workflow)
# Flux uses RandomNoise for seed
assert schema["summary"]["has_seed"] is True
# Flux has only positive prompt (no negative encoder)
assert schema["summary"]["has_negative_prompt"] is False
def test_video_detected(self, video_workflow):
schema = extract_schema(video_workflow)
assert schema["summary"]["is_video_workflow"] is True
class TestEmbeddingDeps:
def test_extract_from_prompt(self):
wf = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "x"}},
"5": {"class_type": "EmptyLatentImage",
"inputs": {"width": 512, "height": 512, "batch_size": 1}},
"6": {"class_type": "CLIPTextEncode",
"inputs": {
"text": "a cat, embedding:goodvibes, embedding:art:1.2",
"clip": ["1", 1]
}},
"7": {"class_type": "CLIPTextEncode",
"inputs": {
"text": "ugly, embedding:badhands",
"clip": ["1", 1]
}},
"3": {"class_type": "KSampler",
"inputs": {
"positive": ["6", 0], "negative": ["7", 0],
"model": ["1", 0], "latent_image": ["5", 0],
"seed": 1, "steps": 20, "cfg": 7.5,
"sampler_name": "euler", "scheduler": "normal", "denoise": 1.0,
}},
"9": {"class_type": "SaveImage", "inputs": {"filename_prefix": "x", "images": ["3", 0]}},
}
schema = extract_schema(wf)
names = [d["embedding_name"] for d in schema["embedding_dependencies"]]
assert sorted(names) == ["art", "badhands", "goodvibes"]
class TestDuplicateDeduplication:
def test_two_ksamplers_get_unique_names(self):
wf = {
"1": {"class_type": "CheckpointLoaderSimple", "inputs": {"ckpt_name": "x"}},
"5": {"class_type": "EmptyLatentImage",
"inputs": {"width": 512, "height": 512, "batch_size": 1}},
"6": {"class_type": "CLIPTextEncode", "inputs": {"text": "a", "clip": ["1", 1]}},
"7": {"class_type": "CLIPTextEncode", "inputs": {"text": "b", "clip": ["1", 1]}},
"3": {"class_type": "KSampler",
"inputs": {
"positive": ["6", 0], "negative": ["7", 0],
"model": ["1", 0], "latent_image": ["5", 0],
"seed": 42, "steps": 20, "cfg": 7.5,
"sampler_name": "euler", "scheduler": "normal", "denoise": 1.0,
}},
"4": {"class_type": "KSampler",
"inputs": {
"positive": ["6", 0], "negative": ["7", 0],
"model": ["1", 0], "latent_image": ["5", 0],
"seed": 99, "steps": 30, "cfg": 8.0,
"sampler_name": "euler", "scheduler": "normal", "denoise": 0.6,
}},
"9": {"class_type": "SaveImage", "inputs": {"filename_prefix": "x", "images": ["3", 0]}},
}
schema = extract_schema(wf)
params = schema["parameters"]
# Both seeds present with disambiguated names
seed_keys = [k for k in params if "seed" in k]
# Symmetric: both renamed (no bare "seed")
assert "seed" not in params
assert "seed_3" in params and "seed_4" in params
assert params["seed_3"]["value"] == 42
assert params["seed_4"]["value"] == 99
@@ -0,0 +1,167 @@
"""Named RGB palettes for pixel_art() and pixel_art_video().
Palette RGB values sourced from pixel-art-studio (MIT License)
https://github.com/Synero/pixel-art-studio — see ATTRIBUTION.md.
"""
PALETTES = {
# ── Hardware palettes ───────────────────────────────────────────────
"NES": [
(0, 0, 0), (124, 124, 124), (0, 0, 252), (0, 0, 188), (68, 40, 188),
(148, 0, 132), (168, 0, 32), (168, 16, 0), (136, 20, 0), (0, 116, 0),
(0, 148, 0), (0, 120, 0), (0, 88, 0), (0, 64, 88), (188, 188, 188),
(0, 120, 248), (0, 88, 248), (104, 68, 252), (216, 0, 204), (228, 0, 88),
(248, 56, 0), (228, 92, 16), (172, 124, 0), (0, 184, 0), (0, 168, 0),
(0, 168, 68), (0, 136, 136), (248, 248, 248), (60, 188, 252),
(104, 136, 252), (152, 120, 248), (248, 120, 248), (248, 88, 152),
(248, 120, 88), (252, 160, 68), (248, 184, 0), (184, 248, 24),
(88, 216, 84), (88, 248, 152), (0, 232, 216), (120, 120, 120),
(252, 252, 252), (164, 228, 252), (184, 184, 248), (216, 184, 248),
(248, 184, 248), (248, 164, 192), (240, 208, 176), (252, 224, 168),
(248, 216, 120), (216, 248, 120), (184, 248, 184), (184, 248, 216),
(0, 252, 252), (216, 216, 216),
],
"C64": [
(0, 0, 0), (255, 255, 255), (161, 77, 67), (106, 191, 199),
(161, 87, 164), (92, 172, 95), (64, 64, 223), (191, 206, 137),
(161, 104, 60), (108, 80, 21), (203, 126, 117), (98, 98, 98),
(137, 137, 137), (154, 226, 155), (124, 124, 255), (173, 173, 173),
],
"COMMODORE_64": [
(0, 0, 0), (255, 255, 255), (161, 77, 67), (106, 192, 200),
(161, 87, 165), (92, 172, 95), (64, 68, 227), (203, 214, 137),
(163, 104, 58), (110, 84, 11), (204, 127, 118), (99, 99, 99),
(139, 139, 139), (154, 227, 157), (139, 127, 205), (175, 175, 175),
],
"ZX_SPECTRUM": [
(0, 0, 0), (0, 39, 251), (252, 48, 22), (255, 63, 252),
(0, 249, 44), (0, 252, 254), (255, 253, 51), (255, 255, 255),
],
"APPLE_II_LO": [
(0, 0, 0), (133, 59, 81), (80, 71, 137), (234, 93, 240),
(0, 104, 82), (146, 146, 146), (0, 168, 241), (202, 195, 248),
(81, 92, 15), (235, 127, 35), (146, 146, 146), (246, 185, 202),
(0, 202, 41), (203, 211, 155), (155, 220, 203), (255, 255, 255),
],
"APPLE_II_HI": [
(0, 0, 0), (255, 0, 255), (0, 255, 0), (255, 255, 255),
(0, 175, 255), (255, 80, 0),
],
"GAMEBOY_ORIGINAL": [
(0, 63, 0), (46, 115, 32), (140, 191, 10), (160, 207, 10),
],
"GAMEBOY_POCKET": [
(0, 0, 0), (85, 85, 85), (170, 170, 170), (255, 255, 255),
],
"GAMEBOY_VIRTUALBOY": [
(239, 0, 0), (164, 0, 0), (85, 0, 0), (0, 0, 0),
],
"PICO_8": [
(0, 0, 0), (29, 43, 83), (126, 37, 83), (0, 135, 81), (171, 82, 54),
(95, 87, 79), (194, 195, 199), (255, 241, 232), (255, 0, 77),
(255, 163, 0), (255, 236, 39), (0, 228, 54), (41, 173, 255),
(131, 118, 156), (255, 119, 168), (255, 204, 170),
],
"TELETEXT": [
(0, 0, 0), (255, 0, 0), (0, 128, 0), (255, 255, 0),
(0, 0, 255), (255, 0, 255), (0, 255, 255), (255, 255, 255),
],
"CGA_MODE4_PAL1": [
(0, 0, 0), (255, 255, 255), (0, 255, 255), (255, 0, 255),
],
"MSX": [
(0, 0, 0), (62, 184, 73), (116, 208, 125), (89, 85, 224),
(128, 118, 241), (185, 94, 81), (101, 219, 239), (219, 101, 89),
(255, 137, 125), (204, 195, 94), (222, 208, 135), (58, 162, 65),
(183, 102, 181), (204, 204, 204), (255, 255, 255),
],
"MICROSOFT_WINDOWS_16": [
(0, 0, 0), (128, 0, 0), (0, 128, 0), (128, 128, 0), (0, 0, 128),
(128, 0, 128), (0, 128, 128), (192, 192, 192), (128, 128, 128),
(255, 0, 0), (0, 255, 0), (255, 255, 0), (0, 0, 255),
(255, 0, 255), (0, 255, 255), (255, 255, 255),
],
"MICROSOFT_WINDOWS_PAINT": [
(0, 0, 0), (255, 255, 255), (123, 123, 123), (189, 189, 189),
(123, 12, 2), (255, 37, 0), (123, 123, 2), (255, 251, 2),
(0, 123, 2), (2, 249, 2), (0, 123, 122), (2, 253, 254),
(2, 19, 122), (5, 50, 255), (123, 25, 122), (255, 64, 254),
(122, 57, 2), (255, 122, 57), (123, 123, 56), (255, 252, 122),
(2, 57, 57), (5, 250, 123), (0, 123, 255), (255, 44, 123),
],
"MONO_BW": [(0, 0, 0), (255, 255, 255)],
"MONO_AMBER": [(40, 40, 40), (255, 176, 0)],
"MONO_GREEN": [(40, 40, 40), (51, 255, 51)],
# ── Artistic palettes ───────────────────────────────────────────────
"PASTEL_DREAM": [
(255, 218, 233), (255, 229, 204), (255, 255, 204), (204, 255, 229),
(204, 229, 255), (229, 204, 255), (255, 204, 229), (204, 255, 255),
(255, 245, 220), (230, 230, 250),
],
"NEON_CYBER": [
(0, 0, 0), (255, 0, 128), (0, 255, 255), (255, 0, 255),
(0, 255, 128), (255, 255, 0), (128, 0, 255), (255, 128, 0),
(0, 128, 255), (255, 255, 255),
],
"RETRO_WARM": [
(62, 39, 35), (139, 69, 19), (210, 105, 30), (244, 164, 96),
(255, 218, 185), (255, 245, 238), (178, 34, 34), (205, 92, 92),
(255, 99, 71), (255, 160, 122),
],
"OCEAN_DEEP": [
(0, 25, 51), (0, 51, 102), (0, 76, 153), (0, 102, 178),
(0, 128, 204), (51, 153, 204), (102, 178, 204), (153, 204, 229),
(204, 229, 255), (229, 245, 255),
],
"FOREST_MOSS": [
(34, 51, 34), (51, 76, 51), (68, 102, 51), (85, 128, 68),
(102, 153, 85), (136, 170, 102), (170, 196, 136), (204, 221, 170),
(238, 238, 204), (245, 245, 220),
],
"SUNSET_FIRE": [
(51, 0, 0), (102, 0, 0), (153, 0, 0), (204, 0, 0), (255, 0, 0),
(255, 51, 0), (255, 102, 0), (255, 153, 0), (255, 204, 0),
(255, 255, 51),
],
"ARCTIC_ICE": [
(0, 0, 51), (0, 0, 102), (0, 51, 153), (0, 102, 153),
(51, 153, 204), (102, 204, 255), (153, 229, 255), (204, 242, 255),
(229, 247, 255), (255, 255, 255),
],
"VINTAGE_ROSE": [
(103, 58, 63), (137, 72, 81), (170, 91, 102), (196, 113, 122),
(219, 139, 147), (232, 168, 175), (240, 196, 199), (245, 215, 217),
(249, 232, 233), (255, 245, 245),
],
"EARTH_CLAY": [
(62, 39, 35), (89, 56, 47), (116, 73, 59), (143, 90, 71),
(170, 107, 83), (197, 124, 95), (210, 155, 126), (222, 186, 160),
(235, 217, 196), (248, 248, 232),
],
"ELECTRIC_VIOLET": [
(26, 0, 51), (51, 0, 102), (76, 0, 153), (102, 0, 204),
(128, 0, 255), (153, 51, 255), (178, 102, 255), (204, 153, 255),
(229, 204, 255), (245, 229, 255),
],
}
def build_palette_image(palette_name):
"""Build a 1x1 PIL 'P'-mode image with the named palette for Image.quantize(palette=...)."""
from PIL import Image
if palette_name not in PALETTES:
raise ValueError(
f"Unknown palette {palette_name!r}. "
f"Choose from: {sorted(PALETTES)}"
)
flat = []
for (r, g, b) in PALETTES[palette_name]:
flat.extend([r, g, b])
# Pad to 768 bytes (256 colors) as PIL requires
while len(flat) < 768:
flat.append(0)
pal_img = Image.new("P", (1, 1))
pal_img.putpalette(flat)
return pal_img
@@ -0,0 +1,3 @@
---
description: Skills for academic research, paper discovery, literature review, domain reconnaissance, market data, content monitoring, and scientific knowledge retrieval.
---
@@ -0,0 +1,345 @@
"""Pixel art video — overlay procedural animations onto a source image.
Takes any image (typically pre-processed with pixel_art()) and overlays
animated pixel effects (stars, rain, fireflies, etc.), then encodes to MP4
(and optionally GIF) via ffmpeg.
Scene animations ported from pixel-art-studio (MIT) — see ATTRIBUTION.md.
The generative/Pollinations code is intentionally dropped — Hermes uses
`image_generate` + `pixel_art()` for base frames instead.
Usage (import):
from pixel_art_video import pixel_art_video
pixel_art_video("frame.png", "out.mp4", scene="night", duration=6)
Usage (CLI):
python pixel_art_video.py frame.png out.mp4 --scene night --duration 6 --gif
"""
import math
import os
import random
import shutil
import subprocess
import tempfile
from PIL import Image, ImageDraw
# ── Pixel drawing helpers ──────────────────────────────────────────────
def _px(draw, x, y, color, size=2):
x, y = int(x), int(y)
W, H = draw.im.size
if 0 <= x < W and 0 <= y < H:
draw.rectangle([x, y, x + size - 1, y + size - 1], fill=color)
def _pixel_cross(draw, x, y, color, arm=2):
x, y = int(x), int(y)
for i in range(-arm, arm + 1):
_px(draw, x + i, y, color, 1)
_px(draw, x, y + i, color, 1)
# ── Animation init/draw pairs ──────────────────────────────────────────
def init_stars(rng, W, H):
return [(rng.randint(0, W), rng.randint(0, H // 2)) for _ in range(15)]
def draw_stars(draw, stars, t, W, H):
for i, (sx, sy) in enumerate(stars):
if math.sin(t * 2.0 + i * 0.7) > 0.65:
_pixel_cross(draw, sx, sy, (255, 255, 220), arm=2)
def init_fireflies(rng, W, H):
return [{"x": rng.randint(20, W - 20), "y": rng.randint(H // 4, H - 20),
"phase": rng.uniform(0, 6.28), "speed": rng.uniform(0.3, 0.8)}
for _ in range(10)]
def draw_fireflies(draw, ff, t, W, H):
for f in ff:
if math.sin(t * 1.5 + f["phase"]) < 0.15:
continue
_px(draw,
f["x"] + math.sin(t * f["speed"] + f["phase"]) * 3,
f["y"] + math.cos(t * f["speed"] * 0.7) * 2,
(200, 255, 100), 2)
def init_leaves(rng, W, H):
return [{"x": rng.randint(0, W), "y": rng.randint(-H, 0),
"speed": rng.uniform(0.5, 1.5), "wobble": rng.uniform(0.02, 0.05),
"phase": rng.uniform(0, 6.28),
"color": rng.choice([(180, 120, 50), (160, 100, 40), (200, 140, 60)])}
for _ in range(12)]
def draw_leaves(draw, leaves, t, W, H):
for leaf in leaves:
_px(draw,
leaf["x"] + math.sin(t * leaf["wobble"] + leaf["phase"]) * 15,
(leaf["y"] + t * leaf["speed"] * 20) % (H + 40) - 20,
leaf["color"], 2)
def init_dust_motes(rng, W, H):
return [{"x": rng.randint(30, W - 30), "y": rng.randint(30, H - 30),
"phase": rng.uniform(0, 6.28), "speed": rng.uniform(0.2, 0.5),
"amp": rng.uniform(2, 6)} for _ in range(20)]
def draw_dust_motes(draw, motes, t, W, H):
for m in motes:
if math.sin(t * 2.0 + m["phase"]) > 0.3:
_px(draw,
m["x"] + math.sin(t * 0.3 + m["phase"]) * m["amp"],
m["y"] - (m["speed"] * t * 15) % H,
(255, 210, 100), 1)
def init_sparkles(rng, W, H):
return [(rng.randint(W // 4, 3 * W // 4), rng.randint(H // 4, 3 * H // 4),
rng.uniform(0, 6.28),
rng.choice([(180, 200, 255), (255, 220, 150), (200, 180, 255)]))
for _ in range(10)]
def draw_sparkles(draw, sparkles, t, W, H):
for sx, sy, phase, color in sparkles:
if math.sin(t * 1.8 + phase) > 0.6:
_pixel_cross(draw, sx, sy, color, arm=2)
def init_rain(rng, W, H):
return [{"x": rng.randint(0, W), "y": rng.randint(0, H),
"speed": rng.uniform(4, 8)} for _ in range(30)]
def draw_rain(draw, rain, t, W, H):
for r in rain:
y = (r["y"] + t * r["speed"] * 20) % H
_px(draw, r["x"], y, (120, 150, 200), 1)
_px(draw, r["x"], y + 4, (100, 130, 180), 1)
def init_lightning(rng, W, H):
return {"timer": 0, "flash": False, "rng": rng}
def draw_lightning(draw, state, t, W, H):
state["timer"] += 1
if state["timer"] > 45 and state["rng"].random() < 0.04:
state["flash"] = True
state["timer"] = 0
if state["flash"]:
for x in range(0, W, 4):
for y in range(0, H // 3, 3):
if state["rng"].random() < 0.12:
_px(draw, x, y, (255, 255, 240), 2)
state["flash"] = False
def init_bubbles(rng, W, H):
return [{"x": rng.randint(20, W - 20), "y": rng.randint(H, H * 2),
"speed": rng.uniform(0.3, 0.8), "size": rng.choice([1, 2, 2])}
for _ in range(15)]
def draw_bubbles(draw, bubbles, t, W, H):
for b in bubbles:
x = b["x"] + math.sin(t * 0.5 + b["x"]) * 3
y = b["y"] - (t * b["speed"] * 20) % (H + 40)
if 0 < y < H:
_px(draw, x, y, (150, 200, 255), b["size"])
def init_embers(rng, W, H):
return [{"x": rng.randint(0, W), "y": rng.randint(0, H),
"speed": rng.uniform(0.3, 0.9), "phase": rng.uniform(0, 6.28),
"color": rng.choice([(255, 150, 30), (255, 100, 20), (255, 200, 50)])}
for _ in range(18)]
def draw_embers(draw, embers, t, W, H):
for e in embers:
x = e["x"] + math.sin(t * 0.4 + e["phase"]) * 5
y = e["y"] - (t * e["speed"] * 15) % H
if math.sin(t * 2.5 + e["phase"]) > 0.2:
_px(draw, x, y, e["color"], 2)
def init_snowflakes(rng, W, H):
return [{"x": rng.randint(0, W), "y": rng.randint(-H, 0),
"speed": rng.uniform(0.3, 0.6), "wobble": rng.uniform(0.04, 0.09),
"size": rng.choice([2, 2, 3])}
for _ in range(40)]
def draw_snowflakes(draw, flakes, t, W, H):
for f in flakes:
x = f["x"] + math.sin(t * f["wobble"] + f["x"]) * 20
y = (f["y"] + t * f["speed"] * 8) % (H + 20) - 10
if f["size"] >= 3:
_pixel_cross(draw, x, y, (230, 235, 255), arm=1)
else:
_px(draw, x, y, (230, 235, 255), 2)
def init_neon_pulse(rng, W, H):
return [(rng.randint(0, W), rng.randint(0, H), rng.uniform(0, 6.28),
rng.choice([(255, 0, 200), (0, 255, 255), (255, 50, 150)]))
for _ in range(8)]
def draw_neon_pulse(draw, points, t, W, H):
for x, y, phase, color in points:
if math.sin(t * 2.5 + phase) > 0.5:
_pixel_cross(draw, x, y, color, arm=3)
def init_heat_shimmer(rng, W, H):
return [{"x": rng.randint(0, W), "y": rng.randint(H // 2, H),
"phase": rng.uniform(0, 6.28)} for _ in range(12)]
def draw_heat_shimmer(draw, points, t, W, H):
for p in points:
x = p["x"] + math.sin(t * 0.8 + p["phase"]) * 2
y = p["y"] + math.sin(t * 1.2 + p["phase"]) * 1
if abs(math.sin(t * 1.5 + p["phase"])) > 0.6:
_px(draw, x, y, (255, 200, 100), 1)
# ── Scene → animation mapping ──────────────────────────────────────────
SCENES = {
"night": ["stars", "fireflies", "leaves"],
"dusk": ["fireflies", "sparkles"],
"tavern": ["dust_motes", "sparkles"],
"indoor": ["dust_motes"],
"urban": ["rain", "neon_pulse"],
"nature": ["leaves", "fireflies"],
"magic": ["sparkles", "fireflies"],
"storm": ["rain", "lightning"],
"underwater": ["bubbles", "sparkles"],
"fire": ["embers", "sparkles"],
"snow": ["snowflakes", "sparkles"],
"desert": ["heat_shimmer", "dust_motes"],
}
# Map scene layer name to (init_fn, draw_fn).
_LAYERS = {
"stars": (init_stars, draw_stars),
"fireflies": (init_fireflies, draw_fireflies),
"leaves": (init_leaves, draw_leaves),
"dust_motes": (init_dust_motes, draw_dust_motes),
"sparkles": (init_sparkles, draw_sparkles),
"rain": (init_rain, draw_rain),
"lightning": (init_lightning, draw_lightning),
"bubbles": (init_bubbles, draw_bubbles),
"embers": (init_embers, draw_embers),
"snowflakes": (init_snowflakes, draw_snowflakes),
"neon_pulse": (init_neon_pulse, draw_neon_pulse),
"heat_shimmer": (init_heat_shimmer, draw_heat_shimmer),
}
def _ensure_ffmpeg():
if shutil.which("ffmpeg") is None:
raise RuntimeError(
"ffmpeg not found on PATH. Install via your package manager or "
"download from https://ffmpeg.org/"
)
def pixel_art_video(
base_image,
output_path,
scene="night",
duration=6,
fps=15,
seed=None,
export_gif=False,
):
"""Overlay pixel animations onto a base image and encode to MP4.
Args:
base_image: path to source image (ideally already pixel-art styled)
output_path: path to MP4 output (GIF sibling written if export_gif=True)
scene: key from SCENES (night, urban, storm, snow, fire, ...)
duration: seconds of animation
fps: frames per second (default 15 for retro feel)
seed: optional int for reproducible animation placement
export_gif: also write a GIF alongside the MP4
Returns:
(mp4_path, gif_path_or_None)
"""
if scene not in SCENES:
raise ValueError(
f"Unknown scene {scene!r}. Choose from: {sorted(SCENES)}"
)
_ensure_ffmpeg()
base = Image.open(base_image).convert("RGB")
W, H = base.size
rng = random.Random(seed if seed is not None else 42)
layers = []
for name in SCENES[scene]:
init_fn, draw_fn = _LAYERS[name]
layers.append((draw_fn, init_fn(rng, W, H)))
n_frames = fps * duration
os.makedirs(os.path.dirname(os.path.abspath(output_path)) or ".", exist_ok=True)
with tempfile.TemporaryDirectory(prefix="pixelart_frames_") as frames_dir:
for frame_idx in range(n_frames):
canvas = base.copy()
draw = ImageDraw.Draw(canvas)
t = frame_idx / fps
for draw_fn, state in layers:
draw_fn(draw, state, t, W, H)
canvas.save(os.path.join(frames_dir, f"frame_{frame_idx:04d}.png"))
subprocess.run(
["ffmpeg", "-y", "-loglevel", "error",
"-framerate", str(fps),
"-i", os.path.join(frames_dir, "frame_%04d.png"),
"-c:v", "libx264", "-pix_fmt", "yuv420p", "-crf", "18",
output_path],
check=True,
)
gif_path = None
if export_gif:
gif_path = output_path.rsplit(".", 1)[0] + ".gif"
subprocess.run(
["ffmpeg", "-y", "-loglevel", "error",
"-framerate", str(fps),
"-i", os.path.join(frames_dir, "frame_%04d.png"),
"-vf",
"scale=320:-1:flags=neighbor,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse",
"-loop", "0",
gif_path],
check=True,
)
return output_path, gif_path
def main():
import argparse
p = argparse.ArgumentParser(description="Overlay pixel animations onto an image → MP4.")
p.add_argument("base_image")
p.add_argument("output")
p.add_argument("--scene", default="night", choices=sorted(SCENES))
p.add_argument("--duration", type=int, default=6)
p.add_argument("--fps", type=int, default=15)
p.add_argument("--seed", type=int, default=None)
p.add_argument("--gif", action="store_true")
args = p.parse_args()
mp4, gif = pixel_art_video(
args.base_image, args.output,
scene=args.scene, duration=args.duration,
fps=args.fps, seed=args.seed, export_gif=args.gif,
)
print(f"Wrote {mp4}")
if gif:
print(f"Wrote {gif}")
if __name__ == "__main__":
main()
@@ -0,0 +1,83 @@
#!/usr/bin/env python3
"""
Pipecat Bot Template — OpenAILLMService avec API Hermes
========================================================
Template minimal pour un bot texte via WebSocket.
Adapté de ~/workspace/jarvis-pipecat/bot.py
Prérequis: pipecat-ai[websocket]>=1.4.0, openai, uvicorn, websockets
"""
import json, os, subprocess, sys
from fastapi import FastAPI, WebSocket
from fastapi.responses import HTMLResponse
import uvicorn
# ⚠️ Pipecat 1.4.0 : utiliser `import module as m` (from-import cassé)
import pipecat.frames.frames as pframes
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.runner import PipelineRunner
from pipecat.pipeline.task import PipelineTask
from pipecat.processors.frame_processor import FrameProcessor, FrameDirection
from pipecat.services.openai.llm import OpenAILLMService
from pipecat.transports.websocket.fastapi import FastAPIWebsocketParams, FastAPIWebsocketTransport
# ─── Config ──────────────────────────────────────────────────
PORT = int(os.environ.get("BOT_PORT", "7081"))
HERMES_URL = os.environ.get("HERMES_API_URL", "http://localhost:8642/v1")
HERMES_KEY = os.environ.get("HERMES_API_KEY", "")
SYSTEM_PROMPT = "Tu es un assistant vocal concis. Parle en français, 3 phrases max."
# ─── Processeur custom : texte → LLMMessagesFrame ────────────
class TextToLLMProcessor(FrameProcessor):
"""Remplace le context aggregator (buggé en 1.4.0)."""
def __init__(self, system_prompt: str):
super().__init__()
self._messages = [{"role": "system", "content": system_prompt}]
async def process_frame(self, frame, direction):
await self.push_frame(frame, direction)
if isinstance(frame, pframes.InputTransportMessageFrame):
text = frame.message
if isinstance(text, str) and text.strip():
self._messages.append({"role": "user", "content": text})
if len(self._messages) > 21:
self._messages = [self._messages[0]] + self._messages[-20:]
await self.push_frame(pframes.LLMMessagesFrame(messages=list(self._messages)))
elif isinstance(frame, pframes.LLMFullResponseStartFrame):
self._response_text = ""
elif isinstance(frame, pframes.TextFrame):
if hasattr(self, '_response_text'):
self._response_text += frame.text
elif isinstance(frame, pframes.LLMFullResponseEndFrame):
if hasattr(self, '_response_text') and self._response_text.strip():
self._messages.append({"role": "assistant", "content": self._response_text.strip()})
del self._response_text
# ─── App ─────────────────────────────────────────────────────
app = FastAPI()
@app.websocket("/ws")
async def ws_endpoint(websocket: WebSocket):
transport = FastAPIWebsocketTransport(
websocket=websocket,
params=FastAPIWebsocketParams(add_wav_header=False, session_timeout=None),
)
llm = OpenAILLMService(
api_key=HERMES_KEY,
base_url=HERMES_URL,
settings=OpenAILLMService.Settings(model="Hermes Agent", temperature=0.8, max_tokens=300, system_instruction=SYSTEM_PROMPT),
)
proc = TextToLLMProcessor(SYSTEM_PROMPT)
pipeline = Pipeline(processors=[transport.input(), proc, llm, transport.output()])
task = PipelineTask(pipeline)
runner = PipelineRunner()
await runner.run(task)
if __name__ == "__main__":
uvicorn.run(app, host="0.0.0.0", port=PORT)
@@ -0,0 +1,88 @@
#!/usr/bin/env python3
"""Scan ~/.hermes/skills/ for corrupted .py scripts.
Usage: python corruption-scanner.py [--fix-dir /path/to/output]
Output: JSON summary of clean vs corrupted files.
"""
import json, os, sys
from pathlib import Path
def scan_skills(base_dir):
results = {"clean": [], "corrupted": [], "skipped": []}
for root, dirs, files in os.walk(base_dir):
if '__pycache__' in root or 'venv' in root:
continue
for fname in files:
if not fname.endswith('.py'):
continue
fpath = os.path.join(root, fname)
size = os.path.getsize(fpath)
# Skip empty __init__.py (normal)
if fname == '__init__.py' and size <= 50:
results["skipped"].append({"path": fpath, "reason": "empty __init__.py"})
continue
try:
with open(fpath, 'r') as f:
content = f.read(1000)
first_line = content.strip().split('\n')[0].lower()
if '% partial rewrite' in first_line[:200] or 'latex' in first_line[:200]:
results["corrupted"].append({
"path": fpath,
"size": size,
"type": "latex_contamination"
})
elif first_line.startswith('%pdf') or '%PDF' in first_line[:20]:
results["corrupted"].append({
"path": fpath,
"size": size,
"type": "pdf_binary"
})
elif len(content) > 200000:
results["corrupted"].append({
"path": fpath,
"size": size,
"type": "oversized"
})
else:
results["clean"].append({"path": fpath, "size": size})
except UnicodeDecodeError:
results["skipped"].append({"path": fpath, "reason": "binary file"})
except Exception as e:
results["skipped"].append({"path": fpath, "reason": str(e)})
return results
if __name__ == '__main__':
base = os.environ.get('HERMES_HOME', str(Path.home() / '.hermes'))
skills_dir = os.path.join(base, 'skills')
fix_dir = None
args = sys.argv[1:]
if '--fix-dir' in args:
idx = args.index('--fix-dir')
if idx + 1 < len(args):
fix_dir = args[idx + 1]
results = scan_skills(skills_dir)
print(json.dumps(results, indent=2, ensure_ascii=False))
# Summary
print(f"\n{'='*60}")
print(f"Scan complete:")
print(f" Clean: {len(results['clean'])}")
print(f" Corrupted: {len(results['corrupted'])}")
print(f" Skipped: {len(results['skipped'])}")
if results['corrupted']:
print(f"\nCorrupted files:")
for f in results['corrupted']:
print(f" ❌ {f['path']} ({f['size']:,} bytes) - {f['type']}")
@@ -0,0 +1,124 @@
#!/usr/bin/env python3
"""
Fetch a YouTube video transcript and output it as structured JSON.
Usage:
uv run python3 fetch_transcript.py <url_or_video_id> [--language en,tr] [--timestamps]
Output (JSON):
{
"video_id": "...",
"language": "en",
"segments": [{"text": "...", "start": 0.0, "duration": 2.5}, ...],
"full_text": "complete transcript as plain text",
"timestamped_text": "00:00 first line\n00:05 second line\n..."
}
Install dependency: uv pip install youtube-transcript-api
"""
import argparse
import json
import re
import sys
def extract_video_id(url_or_id: str) -> str:
"""Extract the 11-character video ID from various YouTube URL formats."""
url_or_id = url_or_id.strip()
patterns = [
r'(?:v=|youtu\.be/|shorts/|embed/|live/)([a-zA-Z0-9_-]{11})',
r'^([a-zA-Z0-9_-]{11})$',
]
for pattern in patterns:
match = re.search(pattern, url_or_id)
if match:
return match.group(1)
return url_or_id
def format_timestamp(seconds: float) -> str:
"""Convert seconds to HH:MM:SS or MM:SS format."""
total = int(seconds)
h, remainder = divmod(total, 3600)
m, s = divmod(remainder, 60)
if h > 0:
return f"{h}:{m:02d}:{s:02d}"
return f"{m}:{s:02d}"
def fetch_transcript(video_id: str, languages: list = None):
"""Fetch transcript segments from YouTube.
Returns a list of dicts with 'text', 'start', and 'duration' keys.
Compatible with youtube-transcript-api v1.x.
"""
try:
from youtube_transcript_api import YouTubeTranscriptApi
except ImportError:
print("Error: youtube-transcript-api not installed. Run: uv pip install youtube-transcript-api",
file=sys.stderr)
sys.exit(1)
api = YouTubeTranscriptApi()
if languages:
result = api.fetch(video_id, languages=languages)
else:
result = api.fetch(video_id)
# v1.x returns FetchedTranscriptSnippet objects; normalize to dicts
return [
{"text": seg.text, "start": seg.start, "duration": seg.duration}
for seg in result
]
def main():
parser = argparse.ArgumentParser(description="Fetch YouTube transcript as JSON")
parser.add_argument("url", help="YouTube URL or video ID")
parser.add_argument("--language", "-l", default=None,
help="Comma-separated language codes (e.g. en,tr). Default: auto")
parser.add_argument("--timestamps", "-t", action="store_true",
help="Include timestamped text in output")
parser.add_argument("--text-only", action="store_true",
help="Output plain text instead of JSON")
args = parser.parse_args()
video_id = extract_video_id(args.url)
languages = [l.strip() for l in args.language.split(",")] if args.language else None
try:
segments = fetch_transcript(video_id, languages)
except Exception as e:
error_msg = str(e)
if "disabled" in error_msg.lower():
print(json.dumps({"error": "Transcripts are disabled for this video."}))
elif "no transcript" in error_msg.lower():
print(json.dumps({"error": f"No transcript found. Try specifying a language with --language."}))
else:
print(json.dumps({"error": error_msg}))
sys.exit(1)
full_text = " ".join(seg["text"] for seg in segments)
timestamped = "\n".join(
f"{format_timestamp(seg['start'])} {seg['text']}" for seg in segments
)
if args.text_only:
print(timestamped if args.timestamps else full_text)
return
result = {
"video_id": video_id,
"segment_count": len(segments),
"duration": format_timestamp(segments[-1]["start"] + segments[-1]["duration"]) if segments else "0:00",
"full_text": full_text,
}
if args.timestamps:
result["timestamped_text"] = timestamped
print(json.dumps(result, ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()
@@ -0,0 +1,124 @@
#!/usr/bin/env python3
"""
Fetch a YouTube video transcript and output it as structured JSON.
Usage:
uv run python3 fetch_transcript.py <url_or_video_id> [--language en,tr] [--timestamps]
Output (JSON):
{
"video_id": "...",
"language": "en",
"segments": [{"text": "...", "start": 0.0, "duration": 2.5}, ...],
"full_text": "complete transcript as plain text",
"timestamped_text": "00:00 first line\n00:05 second line\n..."
}
Install dependency: uv pip install youtube-transcript-api
"""
import argparse
import json
import re
import sys
def extract_video_id(url_or_id: str) -> str:
"""Extract the 11-character video ID from various YouTube URL formats."""
url_or_id = url_or_id.strip()
patterns = [
r'(?:v=|youtu\.be/|shorts/|embed/|live/)([a-zA-Z0-9_-]{11})',
r'^([a-zA-Z0-9_-]{11})$',
]
for pattern in patterns:
match = re.search(pattern, url_or_id)
if match:
return match.group(1)
return url_or_id
def format_timestamp(seconds: float) -> str:
"""Convert seconds to HH:MM:SS or MM:SS format."""
total = int(seconds)
h, remainder = divmod(total, 3600)
m, s = divmod(remainder, 60)
if h > 0:
return f"{h}:{m:02d}:{s:02d}"
return f"{m}:{s:02d}"
def fetch_transcript(video_id: str, languages: list = None):
"""Fetch transcript segments from YouTube.
Returns a list of dicts with 'text', 'start', and 'duration' keys.
Compatible with youtube-transcript-api v1.x.
"""
try:
from youtube_transcript_api import YouTubeTranscriptApi
except ImportError:
print("Error: youtube-transcript-api not installed. Run: uv pip install youtube-transcript-api",
file=sys.stderr)
sys.exit(1)
api = YouTubeTranscriptApi()
if languages:
result = api.fetch(video_id, languages=languages)
else:
result = api.fetch(video_id)
# v1.x returns FetchedTranscriptSnippet objects; normalize to dicts
return [
{"text": seg.text, "start": seg.start, "duration": seg.duration}
for seg in result
]
def main():
parser = argparse.ArgumentParser(description="Fetch YouTube transcript as JSON")
parser.add_argument("url", help="YouTube URL or video ID")
parser.add_argument("--language", "-l", default=None,
help="Comma-separated language codes (e.g. en,tr). Default: auto")
parser.add_argument("--timestamps", "-t", action="store_true",
help="Include timestamped text in output")
parser.add_argument("--text-only", action="store_true",
help="Output plain text instead of JSON")
args = parser.parse_args()
video_id = extract_video_id(args.url)
languages = [l.strip() for l in args.language.split(",")] if args.language else None
try:
segments = fetch_transcript(video_id, languages)
except Exception as e:
error_msg = str(e)
if "disabled" in error_msg.lower():
print(json.dumps({"error": "Transcripts are disabled for this video."}))
elif "no transcript" in error_msg.lower():
print(json.dumps({"error": f"No transcript found. Try specifying a language with --language."}))
else:
print(json.dumps({"error": error_msg}))
sys.exit(1)
full_text = " ".join(seg["text"] for seg in segments)
timestamped = "\n".join(
f"{format_timestamp(seg['start'])} {seg['text']}" for seg in segments
)
if args.text_only:
print(timestamped if args.timestamps else full_text)
return
result = {
"video_id": video_id,
"segment_count": len(segments),
"duration": format_timestamp(segments[-1]["start"] + segments[-1]["duration"]) if segments else "0:00",
"full_text": full_text,
}
if args.timestamps:
result["timestamped_text"] = timestamped
print(json.dumps(result, ensure_ascii=False, indent=2))
if __name__ == "__main__":
main()
@@ -0,0 +1,54 @@
#!/usr/bin/env python3
"""
Analyse une image avec l'API Gemini et écrit le résultat dans un fichier.
Usage: python3 gemini_analyze_frame.py <image_path> <output_path>
La clé API Google doit être dans /tmp/gkey.txt (écrite par l'utilisateur
car Hermes masque les clés API dans tous les outils).
"""
import subprocess, base64, json, sys, os
IMAGE_PATH = sys.argv[1]
OUTPUT_PATH = sys.argv[2]
# Lire la clé depuis le fichier (contourne le masking Hermes)
with open("/tmp/gkey.txt") as f:
API_KEY = f.read().strip()
# Encoder l'image
with open(IMAGE_PATH, "rb") as f:
img_b64 = base64.b64encode(f.read()).decode()
# Construire la requête Gemini
body = {
"contents": [{
"parts": [
{"text": "Décris précisément cette image. Interface, texte, boutons, cases cochées, rôles, menus — tout ce qui est visible. Réponds en français."},
{"inline_data": {"mime_type": "image/png", "data": img_b64}}
]
}]
}
# Appeler Gemini (curl pour éviter les timeouts Python HTTP)
result = subprocess.run([
"curl", "-s",
"--max-time", "30",
"-H", "Content-Type: application/json",
"-d", json.dumps(body),
f"https://generativelanguage.googleapis.com/v1beta/models/gemini-2.0-flash:generateContent?key={API_KEY}"
], capture_output=True, text=True, timeout=35)
data = json.loads(result.stdout)
if "error" in data:
print(f"API ERROR: {data['error']}", file=sys.stderr)
sys.exit(1)
text = data["candidates"][0]["content"]["parts"][0]["text"]
# Écrire dans le fichier de sortie (contourne CCR)
with open(OUTPUT_PATH, "w") as f:
f.write(text)
print(f"✅ {len(text)} chars → {OUTPUT_PATH}")
print(text[:300])
@@ -0,0 +1,64 @@
#!/usr/bin/env python3
"""
Analyse toutes les frames d'un dossier via Gemini et écrit chaque résultat
dans un fichier séparé de < 100 caractères (contourne la compression CCR).
Usage: python3 gemini_analyze_frames.py <frames_dir> <keyfile>
Output: /tmp/fr0.txt, /tmp/fr1.txt, ...
"""
import subprocess, base64, json, sys, glob, os
FRAMES_DIR = sys.ar.../tmp/video_frames"
KEY_FILE = sys.ar...iven"
MODEL = "gemini-2.5-flash"
API = "https://generativelanguage.googleapis.com/v1beta/models"
with open(KEY_FILE) as f:
API_KEY = f.read().strip()
frames = sorted(glob.glob(f"{FRAMES_DIR}/frame*.png"))
if not frames:
frames = sorted(glob.glob(f"{FRAMES_DIR}/*.png"))
for i, frame in enumerate(frames):
try:
with open(frame, "rb") as f:
img_b64 = base64.b64encode(f.read()).decode()
body = {
"contents": [{
"parts": [
{"text": "Decris cette image en francais. Sois precis."},
{"inline_data": {"mime_type": "image/png", "data": img_b64}}
]
}]
}
with open("/tmp/gemini_req.json", "w") as f:
json.dump(body, f)
r = subprocess.run([
"curl", "-s", "-H", "Content-Type: application/json",
"-d", "@/tmp/gemini_req.json",
f"{API}/{MODEL}:generateContent?key={API_KEY}"
], capture_output=True, text=True, timeout=30)
data = json.loads(r.stdout)
if "candidates" in data and data["candidates"]:
text = data["candidates"][0]["content"]["parts"][0]["text"]
else:
text = f"ERR:{r.stdout[:150]}"
# Write to tiny file (CCR bypass)
with open(f"/tmp/fr{i}.txt", "w") as f:
f.write(f"FRAME {i}: {text[:100]}")
print(f"OK frame {i} → /tmp/fr{i}.txt")
except Exception as e:
with open(f"/tmp/fr{i}.txt", "w") as f:
f.write(f"FRAME {i} EXC: {e}")
print(f"ERR frame {i}: {e}")
print(f"\nDONE: {len(frames)} frames → /tmp/fr0.txt .. /tmp/fr{len(frames)-1}.txt")
print("Lire avec: read_file /tmp/fr0.txt")
@@ -0,0 +1,59 @@
# Copyright (c) 2020-202x The virtualenv developers
#
# Permission is hereby granted, free of charge, to any person obtaining
# a copy of this software and associated documentation files (the
# "Software"), to deal in the Software without restriction, including
# without limitation the rights to use, copy, modify, merge, publish,
# distribute, sublicense, and/or sell copies of the Software, and to
# permit persons to whom the Software is furnished to do so, subject to
# the following conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
# MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
# LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
# WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
"""
Activate virtualenv for current interpreter:
import runpy
runpy.run_path(this_file)
This can be used when you must use an existing Python interpreter, not the virtualenv bin/python.
""" # noqa: D415
from __future__ import annotations
import os
import site
import sys
try:
abs_file = os.path.abspath(__file__)
except NameError as exc:
msg = "You must use import runpy; runpy.run_path(this_file)"
raise AssertionError(msg) from exc
bin_dir = os.path.dirname(abs_file)
base = bin_dir[: -len("bin") - 1] # strip away the bin part from the __file__, plus the path separator
# prepend bin to PATH (this file is inside the bin directory)
os.environ["PATH"] = os.pathsep.join([bin_dir, *os.environ.get("PATH", "").split(os.pathsep)])
os.environ["VIRTUAL_ENV"] = base # virtual env is right above bin directory
os.environ["VIRTUAL_ENV_PROMPT"] = "" or os.path.basename(base) # noqa: SIM222
# add the virtual environments libraries to the host python import mechanism
prev_length = len(sys.path)
for lib in "../lib/python3.11/site-packages".split(os.pathsep):
path = os.path.realpath(os.path.join(bin_dir, lib))
site.addsitedir(path)
sys.path[:] = sys.path[prev_length:] + sys.path[0:prev_length]
sys.real_prefix = sys.prefix
sys.prefix = base
@@ -0,0 +1,101 @@
"""Patches that are applied at runtime to the virtual environment."""
import os
import sys
VIRTUALENV_PATCH_FILE = os.path.join(__file__)
def patch_dist(dist):
"""
Distutils allows user to configure some arguments via a configuration file:
https://docs.python.org/3.11/install/index.html#distutils-configuration-files.
Some of this arguments though don't make sense in context of the virtual environment files, let's fix them up.
""" # noqa: D205
# we cannot allow some install config as that would get packages installed outside of the virtual environment
old_parse_config_files = dist.Distribution.parse_config_files
def parse_config_files(self, *args, **kwargs):
result = old_parse_config_files(self, *args, **kwargs)
install = self.get_option_dict("install")
if "prefix" in install: # the prefix governs where to install the libraries
install["prefix"] = VIRTUALENV_PATCH_FILE, os.path.abspath(sys.prefix)
for base in ("purelib", "platlib", "headers", "scripts", "data"):
key = f"install_{base}"
if key in install: # do not allow global configs to hijack venv paths
install.pop(key, None)
return result
dist.Distribution.parse_config_files = parse_config_files
# Import hook that patches some modules to ignore configuration values that break package installation in case
# of virtual environments.
_DISTUTILS_PATCH = "distutils.dist", "setuptools.dist"
# https://docs.python.org/3/library/importlib.html#setting-up-an-importer
class _Finder:
"""A meta path finder that allows patching the imported distutils modules."""
fullname = None
# lock[0] is threading.Lock(), but initialized lazily to avoid importing threading very early at startup,
# because there are gevent-based applications that need to be first to import threading by themselves.
# See https://github.com/pypa/virtualenv/issues/1895 for details.
lock = [] # noqa: RUF012
def find_spec(self, fullname, path, target=None): # noqa: ARG002
if fullname in _DISTUTILS_PATCH and self.fullname is None:
# initialize lock[0] lazily
if len(self.lock) == 0:
import threading
lock = threading.Lock()
# there is possibility that two threads T1 and T2 are simultaneously running into find_spec,
# observing .lock as empty, and further going into hereby initialization. However due to the GIL,
# list.append() operation is atomic and this way only one of the threads will "win" to put the lock
# - that every thread will use - into .lock[0].
# https://docs.python.org/3/faq/library.html#what-kinds-of-global-value-mutation-are-thread-safe
self.lock.append(lock)
from functools import partial
from importlib.util import find_spec
with self.lock[0]:
self.fullname = fullname
try:
spec = find_spec(fullname, path)
if spec is not None:
# https://www.python.org/dev/peps/pep-0451/#how-loading-will-work
is_new_api = hasattr(spec.loader, "exec_module")
func_name = "exec_module" if is_new_api else "load_module"
old = getattr(spec.loader, func_name)
func = self.exec_module if is_new_api else self.load_module
if old is not func:
try: # noqa: SIM105
setattr(spec.loader, func_name, partial(func, old))
except AttributeError:
pass # C-Extension loaders are r/o such as zipimporter with <3.7
return spec
finally:
self.fullname = None
return None
@staticmethod
def exec_module(old, module):
old(module)
if module.__name__ in _DISTUTILS_PATCH:
patch_dist(module)
@staticmethod
def load_module(old, name):
module = old(name)
if module.__name__ in _DISTUTILS_PATCH:
patch_dist(module)
return module
sys.meta_path.insert(0, _Finder())
@@ -0,0 +1,4 @@
from .core import contents, where
__all__ = ["contents", "where"]
__version__ = "2026.07.22"
@@ -0,0 +1,12 @@
import argparse
from certifi import contents, where
parser = argparse.ArgumentParser()
parser.add_argument("-c", "--contents", action="store_true")
args = parser.parse_args()
if args.contents:
print(contents())
else:
print(where())
@@ -0,0 +1,83 @@
"""
certifi.py
~~~~~~~~~~
This module returns the installation location of cacert.pem or its contents.
"""
import sys
import atexit
def exit_cacert_ctx() -> None:
_CACERT_CTX.__exit__(None, None, None) # type: ignore[union-attr]
if sys.version_info >= (3, 11):
from importlib.resources import as_file, files
_CACERT_CTX = None
_CACERT_PATH = None
def where() -> str:
# This is slightly terrible, but we want to delay extracting the file
# in cases where we're inside of a zipimport situation until someone
# actually calls where(), but we don't want to re-extract the file
# on every call of where(), so we'll do it once then store it in a
# global variable.
global _CACERT_CTX
global _CACERT_PATH
if _CACERT_PATH is None:
# This is slightly janky, the importlib.resources API wants you to
# manage the cleanup of this file, so it doesn't actually return a
# path, it returns a context manager that will give you the path
# when you enter it and will do any cleanup when you leave it. In
# the common case of not needing a temporary file, it will just
# return the file system location and the __exit__() is a no-op.
#
# We also have to hold onto the actual context manager, because
# it will do the cleanup whenever it gets garbage collected, so
# we will also store that at the global level as well.
_CACERT_CTX = as_file(files("certifi").joinpath("cacert.pem"))
_CACERT_PATH = str(_CACERT_CTX.__enter__())
atexit.register(exit_cacert_ctx)
return _CACERT_PATH
def contents() -> str:
return files("certifi").joinpath("cacert.pem").read_text(encoding="ascii")
else:
from importlib.resources import path as get_path, read_text
_CACERT_CTX = None
_CACERT_PATH = None
def where() -> str:
# This is slightly terrible, but we want to delay extracting the
# file in cases where we're inside of a zipimport situation until
# someone actually calls where(), but we don't want to re-extract
# the file on every call of where(), so we'll do it once then store
# it in a global variable.
global _CACERT_CTX
global _CACERT_PATH
if _CACERT_PATH is None:
# This is slightly janky, the importlib.resources API wants you
# to manage the cleanup of this file, so it doesn't actually
# return a path, it returns a context manager that will give
# you the path when you enter it and will do any cleanup when
# you leave it. In the common case of not needing a temporary
# file, it will just return the file system location and the
# __exit__() is a no-op.
#
# We also have to hold onto the actual context manager, because
# it will do the cleanup whenever it gets garbage collected, so
# we will also store that at the global level as well.
_CACERT_CTX = get_path("certifi", "cacert.pem")
_CACERT_PATH = str(_CACERT_CTX.__enter__())
atexit.register(exit_cacert_ctx)
return _CACERT_PATH
def contents() -> str:
return read_text("certifi", "cacert.pem", encoding="ascii")
@@ -0,0 +1,18 @@
import os
import unittest
import certifi
class TestCertifi(unittest.TestCase):
def test_cabundle_exists(self) -> None:
assert os.path.exists(certifi.where())
def test_read_contents(self) -> None:
content = certifi.contents()
assert "-----BEGIN CERTIFICATE-----" in content
def test_py_typed_exists(self) -> None:
assert os.path.exists(
os.path.join(os.path.dirname(certifi.__file__), 'py.typed')
)
@@ -0,0 +1,48 @@
"""
Charset-Normalizer
~~~~~~~~~~~~~~
The Real First Universal Charset Detector.
A library that helps you read text from an unknown charset encoding.
Motivated by chardet, This package is trying to resolve the issue by taking a new approach.
All IANA character set names for which the Python core library provides codecs are supported.
Basic usage:
>>> from charset_normalizer import from_bytes
>>> results = from_bytes('Bсеки човек има право на образование. Oбразованието!'.encode('utf_8'))
>>> best_guess = results.best()
>>> str(best_guess)
'Bсеки човек има право на образование. Oбразованието!'
Others methods and usages are available - see the full documentation
at <https://github.com/Ousret/charset_normalizer>.
:copyright: (c) Ahmed TAHRI
:license: MIT, see LICENSE for more details.
"""
from __future__ import annotations
import logging
from .api import from_bytes, from_fp, from_path, is_binary
from .legacy import detect
from .models import CharsetMatch, CharsetMatches
from .utils import set_logging_handler
from .version import VERSION, __version__
__all__ = (
"from_fp",
"from_path",
"from_bytes",
"is_binary",
"detect",
"CharsetMatch",
"CharsetMatches",
"__version__",
"VERSION",
"set_logging_handler",
)
# Attach a NullHandler to the top level logger by default
# https://docs.python.org/3.3/howto/logging.html#configuring-logging-for-a-library
logging.getLogger("charset_normalizer").addHandler(logging.NullHandler())
@@ -0,0 +1,6 @@
from __future__ import annotations
from .cli import cli_detect
if __name__ == "__main__":
cli_detect()
@@ -0,0 +1,934 @@
from __future__ import annotations
import logging
from functools import lru_cache
from os import PathLike
from typing import BinaryIO
from .cd import (
coherence_ratio,
encoding_languages,
mb_encoding_languages,
merge_coherence_ratios,
)
from .constant import (
IANA_SUPPORTED,
IANA_SUPPORTED_SIMILAR,
TOO_BIG_SEQUENCE,
TOO_SMALL_SEQUENCE,
TRACE,
)
from .md import mess_ratio
from .models import CharsetMatch, CharsetMatches
from .utils import (
any_specified_encoding,
cut_sequence_chunks,
iana_name,
identify_sig_or_bom,
is_multi_byte_encoding,
should_strip_sig_or_bom,
)
logger = logging.getLogger("charset_normalizer")
explain_handler = logging.StreamHandler()
explain_handler.setFormatter(
logging.Formatter("%(asctime)s | %(levelname)s | %(message)s")
)
# Pre-compute a reordered encoding list: multibyte first, then single-byte.
# This allows the mb_definitive_match optimization to fire earlier, skipping
# all single-byte encodings for genuine CJK content. Multibyte codecs
# hard-fail (UnicodeDecodeError) on single-byte data almost instantly, so
# testing them first costs negligible time for non-CJK files.
# Stable sort on a boolean key: multibyte (False) first, IANA order kept
# within each group.
IANA_SUPPORTED_MB_FIRST: list[str] = sorted(
IANA_SUPPORTED, key=lambda encoding: not is_multi_byte_encoding(encoding)
)
def from_bytes(
sequences: bytes | bytearray,
steps: int = 5,
chunk_size: int = 512,
threshold: float = 0.2,
cp_isolation: list[str] | None = None,
cp_exclusion: list[str] | None = None,
preemptive_behaviour: bool = True,
explain: bool = False,
language_threshold: float = 0.1,
enable_fallback: bool = True,
) -> CharsetMatches:
"""
Given a raw bytes sequence, return the best possibles charset usable to render str objects.
If there is no results, it is a strong indicator that the source is binary/not text.
By default, the process will extract 5 blocks of 512o each to assess the mess and coherence of a given sequence.
And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will.
The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page
but never take it for granted. Can improve the performance.
You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that
purpose.
This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32.
By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain'
toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging.
Custom logging format and handler can be set manually.
"""
if not isinstance(sequences, (bytearray, bytes)):
raise TypeError(
"Expected object of type bytes or bytearray, got: {}".format(
type(sequences)
)
)
if explain:
previous_logger_level: int = logger.level
logger.addHandler(explain_handler)
logger.setLevel(TRACE)
length: int = len(sequences)
if length == 0:
logger.debug("Encoding detection on empty bytes, assuming utf_8 intention.")
if explain: # Defensive: ensure exit path clean handler
logger.removeHandler(explain_handler)
logger.setLevel(previous_logger_level)
return CharsetMatches([CharsetMatch(sequences, "utf_8", 0.0, False, [], "")])
if cp_isolation is not None:
logger.log(
TRACE,
"cp_isolation is set. use this flag for debugging purpose. "
"limited list of encoding allowed : %s.",
", ".join(cp_isolation),
)
cp_isolation = [iana_name(cp, False) for cp in cp_isolation]
else:
cp_isolation = []
if cp_exclusion is not None:
logger.log(
TRACE,
"cp_exclusion is set. use this flag for debugging purpose. "
"limited list of encoding excluded : %s.",
", ".join(cp_exclusion),
)
cp_exclusion = [iana_name(cp, False) for cp in cp_exclusion]
else:
cp_exclusion = []
if length <= (chunk_size * steps):
logger.log(
TRACE,
"override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.",
steps,
chunk_size,
length,
)
steps = 1
chunk_size = length
if steps > 1 and length / steps < chunk_size:
chunk_size = int(length / steps)
is_too_small_sequence: bool = len(sequences) < TOO_SMALL_SEQUENCE
is_too_large_sequence: bool = len(sequences) >= TOO_BIG_SEQUENCE
if is_too_small_sequence:
logger.log(
TRACE,
"Trying to detect encoding from a tiny portion of (%i) byte(s).",
length,
)
elif is_too_large_sequence:
logger.log(
TRACE,
"Using lazy str decoding because the payload is quite large, (%i) byte(s).",
length,
)
prioritized_encodings: list[str] = []
specified_encoding: str | None = (
any_specified_encoding(sequences) if preemptive_behaviour else None
)
if specified_encoding is not None:
prioritized_encodings.append(specified_encoding)
logger.log(
TRACE,
"Detected declarative mark in sequence. Priority +1 given for %s.",
specified_encoding,
)
tested: set[str] = set()
tested_but_hard_failure: list[str] = []
tested_but_soft_failure: list[str] = []
soft_failure_skip: set[str] = set()
# Avoid unoptimized RSS usage.
# this cache is mostly interesting for
# local usage. Garbage collected at the
# end. Like it should.
cached_mess_ratio = lru_cache(maxsize=None)(mess_ratio)
cached_coherence_ratio = lru_cache(maxsize=None)(coherence_ratio)
# When a definitive result (chaos=0.0 and good coherence) is found after testing
# the prioritized encodings (ascii, utf_8), we can significantly reduce the remaining
# work. Encodings that target completely different language families (e.g., Cyrillic
# when the definitive match is Latin) are skipped entirely.
# Additionally, for same-family encodings that pass chaos probing, we reuse the
# definitive match's coherence ratios instead of recomputing them — a major savings
# since coherence_ratio accounts for ~30% of total time on slow Latin files.
definitive_match_found: bool = False
definitive_target_languages: set[str] = set()
# After the definitive match fires, we cap the number of additional same-family
# single-byte encodings that pass chaos probing. Once we've accumulated enough
# good candidates (N), further same-family SB encodings are unlikely to produce
# a better best() result and just waste mess_ratio + coherence_ratio time.
# The first encoding to trigger the definitive match is NOT counted (it's already in).
post_definitive_sb_success_count: int = 0
POST_DEFINITIVE_SB_CAP: int = 7
# When a non-UTF multibyte encoding passes chaos probing with significant multibyte
# content (decoded length < 98% of raw length), skip all remaining single-byte encodings.
# Rationale: multi-byte decoders (CJK) have strict byte-sequence validation — if they
# decode without error AND pass chaos probing with substantial multibyte content, the
# data is genuinely multibyte encoded. Single-byte encodings will always decode (every
# byte maps to something) but waste time on mess_ratio before failing.
# The 98% threshold prevents false triggers on files that happen to have a few valid
# multibyte pairs (e.g., cp424/_ude_1.txt where big5 decodes with 99% ratio).
mb_definitive_match_found: bool = False
fallback_ascii: CharsetMatch | None = None
fallback_u8: CharsetMatch | None = None
fallback_specified: CharsetMatch | None = None
results: CharsetMatches = CharsetMatches()
early_stop_results: CharsetMatches = CharsetMatches()
sig_encoding, sig_payload = identify_sig_or_bom(sequences)
if sig_encoding is not None:
prioritized_encodings.insert(0, sig_encoding)
logger.log(
TRACE,
"Detected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.",
len(sig_payload),
sig_encoding,
)
prioritized_encodings.append("ascii")
if "utf_8" not in prioritized_encodings:
prioritized_encodings.append("utf_8")
for encoding_iana in prioritized_encodings + IANA_SUPPORTED_MB_FIRST:
if cp_isolation and encoding_iana not in cp_isolation:
continue
if cp_exclusion and encoding_iana in cp_exclusion:
continue
if encoding_iana in tested:
continue
tested.add(encoding_iana)
decoded_payload: str | None = None
bom_or_sig_available: bool = sig_encoding == encoding_iana
strip_sig_or_bom: bool = bom_or_sig_available and should_strip_sig_or_bom(
encoding_iana
)
if encoding_iana in {"utf_16", "utf_32"} and not bom_or_sig_available:
logger.log(
TRACE,
"Encoding %s won't be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.",
encoding_iana,
)
continue
if encoding_iana in {"utf_7"} and not bom_or_sig_available:
logger.log(
TRACE,
"Encoding %s won't be tested as-is because detection is unreliable without BOM/SIG.",
encoding_iana,
)
continue
# Skip encodings similar to ones that already soft-failed (high mess ratio).
# Checked BEFORE the expensive decode attempt.
if encoding_iana in soft_failure_skip:
logger.log(
TRACE,
"%s is deemed too similar to a code page that was already considered unsuited. Continuing!",
encoding_iana,
)
continue
try:
is_multi_byte_decoder: bool = is_multi_byte_encoding(encoding_iana)
except (ModuleNotFoundError, ImportError): # Defensive:
logger.log(
TRACE,
"Encoding %s does not provide an IncrementalDecoder",
encoding_iana,
)
continue
# When we've already found a definitive match (chaos=0.0 with good coherence)
# after testing the prioritized encodings, skip encodings that target
# completely different language families. This avoids running expensive
# mess_ratio + coherence_ratio on clearly unrelated candidates (e.g., Cyrillic
# when the definitive match is Latin-based).
if definitive_match_found:
if not is_multi_byte_decoder:
enc_languages = set(encoding_languages(encoding_iana))
else:
enc_languages = set(mb_encoding_languages(encoding_iana))
if not enc_languages.intersection(definitive_target_languages):
logger.log(
TRACE,
"Skipping %s: definitive match already found, this encoding targets different languages (%s vs %s).",
encoding_iana,
enc_languages,
definitive_target_languages,
)
continue
# After the definitive match, cap the number of additional same-family
# single-byte encodings that pass chaos probing. This avoids testing the
# tail of rare, low-value same-family encodings (mac_iceland, cp860, etc.)
# that almost never change best() but each cost ~1-2ms of mess_ratio + coherence.
if (
definitive_match_found
and not is_multi_byte_decoder
and post_definitive_sb_success_count >= POST_DEFINITIVE_SB_CAP
):
logger.log(
TRACE,
"Skipping %s: already accumulated %d same-family results after definitive match (cap=%d).",
encoding_iana,
post_definitive_sb_success_count,
POST_DEFINITIVE_SB_CAP,
)
continue
# When a multibyte encoding with significant multibyte content has already
# passed chaos probing, skip all single-byte encodings. They will either fail
# chaos probing (wasting mess_ratio time) or produce inferior results.
if mb_definitive_match_found and not is_multi_byte_decoder:
logger.log(
TRACE,
"Skipping single-byte %s: multi-byte definitive match already found.",
encoding_iana,
)
continue
# Single-byte candidates of regular size defer the expensive whole
# payload decode until after chunk probing: single-byte codecs are
# stateless (1 byte == 1 char) so decoding chunk slices is provably
# identical to slicing the decoded payload, and candidates rejected
# by chaos probing (the common case) never pay the full decode nor
# the payload hash.
deferred_decoding: bool = (
not is_multi_byte_decoder and not is_too_large_sequence
)
try:
if is_too_large_sequence and not is_multi_byte_decoder:
str(
(
sequences[: int(50e4)]
if not strip_sig_or_bom
else sequences[len(sig_payload) : int(50e4)]
),
encoding=encoding_iana,
)
elif not deferred_decoding:
# UTF-7 BOM is encoded in modified Base64 whose byte boundary
# can overlap with the next character. Stripping raw SIG bytes
# before decoding may leave stray bytes that decode as garbage.
# Decode the full sequence and remove the leading BOM char instead.
# see https://github.com/jawah/charset_normalizer/issues/718
# and https://github.com/jawah/charset_normalizer/issues/716
if encoding_iana == "utf_7" and bom_or_sig_available:
decoded_payload = str(
sequences,
encoding=encoding_iana,
)
if decoded_payload and decoded_payload[0] == "\ufeff":
decoded_payload = decoded_payload[1:]
else:
decoded_payload = str(
(
sequences
if not strip_sig_or_bom
else sequences[len(sig_payload) :]
),
encoding=encoding_iana,
)
except (UnicodeDecodeError, LookupError) as e:
if not isinstance(e, LookupError):
logger.log(
TRACE,
"Code page %s does not fit given bytes sequence at ALL. %s",
encoding_iana,
str(e),
)
tested_but_hard_failure.append(encoding_iana)
continue
r_ = range(
0 if not bom_or_sig_available else len(sig_payload),
length,
int(length / steps),
)
multi_byte_bonus: bool = (
is_multi_byte_decoder
and decoded_payload is not None
and len(decoded_payload) < length
)
if multi_byte_bonus:
logger.log(
TRACE,
"Code page %s is a multi byte encoding table and it appear that at least one character "
"was encoded using n-bytes.",
encoding_iana,
)
max_chunk_gave_up: int = int(len(r_) / 4)
max_chunk_gave_up = max(max_chunk_gave_up, 2)
early_stop_count: int = 0
lazy_str_hard_failure = False
md_chunks: list[str] = []
md_ratios = []
try:
for chunk in cut_sequence_chunks(
sequences,
encoding_iana,
r_,
chunk_size,
bom_or_sig_available,
strip_sig_or_bom,
sig_payload,
is_multi_byte_decoder,
decoded_payload,
deferred_decoding,
):
md_chunks.append(chunk)
md_ratios.append(
cached_mess_ratio(
chunk,
threshold,
explain and 1 <= len(cp_isolation) <= 2,
)
)
if md_ratios[-1] >= threshold:
early_stop_count += 1
if (early_stop_count >= max_chunk_gave_up) or (
bom_or_sig_available and not strip_sig_or_bom
):
break
except (
UnicodeDecodeError,
LookupError,
) as e: # Lazy str loading may have missed something there
if deferred_decoding:
# Deferred single-byte validation failed on a chunk (or the
# codec is unavailable on this interpreter build): identical
# outcome and bookkeeping to the eager full-decode failure.
logger.log(
TRACE,
"Code page %s does not fit given bytes sequence at ALL. %s",
encoding_iana,
str(e),
)
tested_but_hard_failure.append(encoding_iana)
continue
logger.log(
TRACE,
"LazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %s",
encoding_iana,
str(e),
)
early_stop_count = max_chunk_gave_up
lazy_str_hard_failure = True
mean_mess_ratio: float = sum(md_ratios) / len(md_ratios) if md_ratios else 0.0
# We might want to check the sequence again with the whole content,
# but only if initial MD tests passed.
if (
not lazy_str_hard_failure
and is_too_large_sequence
and not is_multi_byte_decoder
and mean_mess_ratio < threshold
and early_stop_count < max_chunk_gave_up
):
try:
sequences[int(50e3) :].decode(encoding_iana, errors="strict")
except UnicodeDecodeError as e:
logger.log(
TRACE,
"LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %s",
encoding_iana,
str(e),
)
tested_but_hard_failure.append(encoding_iana)
continue
if mean_mess_ratio >= threshold or early_stop_count >= max_chunk_gave_up:
tested_but_soft_failure.append(encoding_iana)
if encoding_iana in IANA_SUPPORTED_SIMILAR:
soft_failure_skip.update(IANA_SUPPORTED_SIMILAR[encoding_iana])
logger.log(
TRACE,
"%s was excluded because of initial chaos probing. Gave up %i time(s). "
"Computed mean chaos is %f %%.",
encoding_iana,
early_stop_count,
round(mean_mess_ratio * 100, ndigits=3),
)
# Preparing those fallbacks in case we got nothing.
if (
enable_fallback
and encoding_iana
in ["ascii", "utf_8", specified_encoding, "utf_16", "utf_32"]
and not lazy_str_hard_failure
):
# Always fully decode payload before.
# We've missed a UnicodeDecodeError proof
# while issuing release 3.4.8
# see https://github.com/jawah/charset_normalizer/issues/771
if decoded_payload is None:
try:
decoded_payload = str(
(
sequences
if not strip_sig_or_bom
else sequences[len(sig_payload) :]
),
encoding=encoding_iana,
)
except (UnicodeDecodeError, LookupError):
logger.log(
TRACE,
"%s does not decode the whole payload: fallback entry withheld.",
encoding_iana,
)
continue
if is_too_large_sequence:
# Don't retain huge payload in RAM.
decoded_payload = None
fallback_entry = CharsetMatch(
sequences,
encoding_iana,
threshold,
bom_or_sig_available,
[],
decoded_payload,
preemptive_declaration=specified_encoding,
)
if encoding_iana == specified_encoding:
fallback_specified = fallback_entry
elif encoding_iana == "ascii":
fallback_ascii = fallback_entry
else:
fallback_u8 = fallback_entry
continue
if deferred_decoding:
# The candidate passed chaos probing: perform the whole payload
# decode (validation + payload reuse) that was deferred earlier.
try:
decoded_payload = str(
(
sequences
if not strip_sig_or_bom
else sequences[len(sig_payload) :]
),
encoding=encoding_iana,
)
except (UnicodeDecodeError, LookupError) as e:
logger.log(
TRACE,
"Code page %s does not fit given bytes sequence at ALL. %s",
encoding_iana,
str(e),
)
tested_but_hard_failure.append(encoding_iana)
continue
logger.log(
TRACE,
"%s passed initial chaos probing. Mean measured chaos is %f %%",
encoding_iana,
round(mean_mess_ratio * 100, ndigits=3),
)
if not is_multi_byte_decoder:
target_languages: list[str] = encoding_languages(encoding_iana)
else:
target_languages = mb_encoding_languages(encoding_iana)
if target_languages:
logger.log(
TRACE,
"%s should target any language(s) of %s",
encoding_iana,
target_languages,
)
cd_ratios = []
# Run coherence detection on all chunks. We previously tried limiting to
# 1-2 chunks for post-definitive encodings to save time, but this caused
# coverage regressions by producing unrepresentative coherence scores.
# The SB cap and language-family skip optimizations provide sufficient
# speedup without sacrificing coherence accuracy.
if encoding_iana != "ascii":
# We shall skip the CD when its about ASCII
# Most of the time its not relevant to run "language-detection" on it.
lg_inclusion: str | None = (
",".join(target_languages) if target_languages else None
)
for chunk in md_chunks:
chunk_languages = cached_coherence_ratio(
chunk,
language_threshold,
lg_inclusion,
)
cd_ratios.append(chunk_languages)
cd_ratios_merged = merge_coherence_ratios(cd_ratios)
if cd_ratios_merged:
logger.log(
TRACE,
"We detected language %s using %s",
cd_ratios_merged,
encoding_iana,
)
current_match = CharsetMatch(
sequences,
encoding_iana,
mean_mess_ratio,
bom_or_sig_available,
cd_ratios_merged,
(
decoded_payload
if (
not is_too_large_sequence
or encoding_iana in [specified_encoding, "ascii", "utf_8"]
)
else None
),
preemptive_declaration=specified_encoding,
)
results.append(current_match)
# Count post-definitive same-family SB successes for the early termination cap.
# Only count low-mess encodings (< 2%) toward the cap. High-mess encodings are
# marginal results that shouldn't prevent better-quality candidates from being
# tested. For example, iso8859_4 (mess=0%) should not be skipped just because
# 7 high-mess Latin encodings (cp1252 at 8%, etc.) were tried first.
if (
definitive_match_found
and not is_multi_byte_decoder
and mean_mess_ratio < 0.02
):
post_definitive_sb_success_count += 1
if (
encoding_iana in [specified_encoding, "ascii", "utf_8"]
and mean_mess_ratio < 0.1
):
# If md says nothing to worry about, then... stop immediately!
if mean_mess_ratio == 0.0:
logger.debug(
"Encoding detection: %s is most likely the one.",
current_match.encoding,
)
if explain: # Defensive: ensure exit path clean handler
logger.removeHandler(explain_handler)
logger.setLevel(previous_logger_level)
return CharsetMatches([current_match])
early_stop_results.append(current_match)
if (
len(early_stop_results)
and (specified_encoding is None or specified_encoding in tested)
and "ascii" in tested
and "utf_8" in tested
):
probable_result = early_stop_results.best()
assert probable_result is not None
logger.debug(
"Encoding detection: %s is most likely the one.",
probable_result.encoding,
)
if explain: # Defensive: ensure exit path clean handler
logger.removeHandler(explain_handler)
logger.setLevel(previous_logger_level)
return CharsetMatches([probable_result])
# Once we find a result with good coherence (>= 0.5) after testing the
# prioritized encodings (ascii, utf_8), activate "definitive mode": skip
# encodings that target completely different language families. This avoids
# running expensive mess_ratio + coherence_ratio on clearly unrelated
# candidates (e.g., Cyrillic encodings when the match is Latin-based).
# We require coherence >= 0.5 to avoid false positives (e.g., cp1251 decoding
# Hebrew text with 0.0 chaos but wrong language detection at coherence 0.33).
if not definitive_match_found and not is_multi_byte_decoder:
best_coherence = (
max((v for _, v in cd_ratios_merged), default=0.0)
if cd_ratios_merged
else 0.0
)
if best_coherence >= 0.5 and "ascii" in tested and "utf_8" in tested:
definitive_match_found = True
definitive_target_languages.update(target_languages)
logger.log(
TRACE,
"Definitive match found: %s (chaos=%.3f, coherence=%.2f). Encodings targeting different language families will be skipped.",
encoding_iana,
mean_mess_ratio,
best_coherence,
)
# When a non-UTF multibyte encoding passes chaos probing with significant
# multibyte content (decoded < 98% of raw), activate mb_definitive_match.
# This skips all remaining single-byte encodings which would either soft-fail
# (running expensive mess_ratio for nothing) or produce inferior results.
if (
not mb_definitive_match_found
and is_multi_byte_decoder
and multi_byte_bonus
and decoded_payload is not None
and len(decoded_payload) < length * 0.98
and encoding_iana
not in {
"utf_8",
"utf_8_sig",
"utf_16",
"utf_16_be",
"utf_16_le",
"utf_32",
"utf_32_be",
"utf_32_le",
"utf_7",
}
and "ascii" in tested
and "utf_8" in tested
):
mb_definitive_match_found = True
logger.log(
TRACE,
"Multi-byte definitive match: %s (chaos=%.3f, decoded=%d/%d=%.1f%%). Single-byte encodings will be skipped.",
encoding_iana,
mean_mess_ratio,
len(decoded_payload),
length,
len(decoded_payload) / length * 100,
)
if encoding_iana == sig_encoding:
logger.debug(
"Encoding detection: %s is most likely the one as we detected a BOM or SIG within "
"the beginning of the sequence.",
encoding_iana,
)
if explain: # Defensive: ensure exit path clean handler
logger.removeHandler(explain_handler)
logger.setLevel(previous_logger_level)
return CharsetMatches([results[encoding_iana]])
if len(results) == 0:
if fallback_u8 or fallback_ascii or fallback_specified:
logger.log(
TRACE,
"Nothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.",
)
if fallback_specified:
logger.debug(
"Encoding detection: %s will be used as a fallback match",
fallback_specified.encoding,
)
results.append(fallback_specified)
elif (
(fallback_u8 and fallback_ascii is None)
or (
fallback_u8
and fallback_ascii
and fallback_u8.fingerprint != fallback_ascii.fingerprint
)
or (fallback_u8 is not None)
):
logger.debug("Encoding detection: utf_8 will be used as a fallback match")
results.append(fallback_u8)
elif fallback_ascii:
logger.debug("Encoding detection: ascii will be used as a fallback match")
results.append(fallback_ascii)
if results:
logger.debug(
"Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.",
results.best().encoding, # type: ignore
len(results) - 1,
)
else:
logger.debug("Encoding detection: Unable to determine any suitable charset.")
if explain:
logger.removeHandler(explain_handler)
logger.setLevel(previous_logger_level)
return results
def from_fp(
fp: BinaryIO,
steps: int = 5,
chunk_size: int = 512,
threshold: float = 0.20,
cp_isolation: list[str] | None = None,
cp_exclusion: list[str] | None = None,
preemptive_behaviour: bool = True,
explain: bool = False,
language_threshold: float = 0.1,
enable_fallback: bool = True,
) -> CharsetMatches:
"""
Same thing than the function from_bytes but using a file pointer that is already ready.
Will not close the file pointer.
"""
return from_bytes(
fp.read(),
steps,
chunk_size,
threshold,
cp_isolation,
cp_exclusion,
preemptive_behaviour,
explain,
language_threshold,
enable_fallback,
)
def from_path(
path: str | bytes | PathLike, # type: ignore[type-arg]
steps: int = 5,
chunk_size: int = 512,
threshold: float = 0.20,
cp_isolation: list[str] | None = None,
cp_exclusion: list[str] | None = None,
preemptive_behaviour: bool = True,
explain: bool = False,
language_threshold: float = 0.1,
enable_fallback: bool = True,
) -> CharsetMatches:
"""
Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode.
Can raise IOError.
"""
with open(path, "rb") as fp:
return from_fp(
fp,
steps,
chunk_size,
threshold,
cp_isolation,
cp_exclusion,
preemptive_behaviour,
explain,
language_threshold,
enable_fallback,
)
def is_binary(
fp_or_path_or_payload: PathLike | str | BinaryIO | bytes, # type: ignore[type-arg]
steps: int = 5,
chunk_size: int = 512,
threshold: float = 0.20,
cp_isolation: list[str] | None = None,
cp_exclusion: list[str] | None = None,
preemptive_behaviour: bool = True,
explain: bool = False,
language_threshold: float = 0.1,
enable_fallback: bool = False,
) -> bool:
"""
Detect if the given input (file, bytes, or path) points to a binary file. aka. not a string.
Based on the same main heuristic algorithms and default kwargs at the sole exception that fallbacks match
are disabled to be stricter around ASCII-compatible but unlikely to be a string.
"""
if isinstance(fp_or_path_or_payload, (str, PathLike)):
guesses = from_path(
fp_or_path_or_payload,
steps=steps,
chunk_size=chunk_size,
threshold=threshold,
cp_isolation=cp_isolation,
cp_exclusion=cp_exclusion,
preemptive_behaviour=preemptive_behaviour,
explain=explain,
language_threshold=language_threshold,
enable_fallback=enable_fallback,
)
elif isinstance(
fp_or_path_or_payload,
(
bytes,
bytearray,
),
):
guesses = from_bytes(
fp_or_path_or_payload,
steps=steps,
chunk_size=chunk_size,
threshold=threshold,
cp_isolation=cp_isolation,
cp_exclusion=cp_exclusion,
preemptive_behaviour=preemptive_behaviour,
explain=explain,
language_threshold=language_threshold,
enable_fallback=enable_fallback,
)
else:
guesses = from_fp(
fp_or_path_or_payload,
steps=steps,
chunk_size=chunk_size,
threshold=threshold,
cp_isolation=cp_isolation,
cp_exclusion=cp_exclusion,
preemptive_behaviour=preemptive_behaviour,
explain=explain,
language_threshold=language_threshold,
enable_fallback=enable_fallback,
)
return not guesses
@@ -0,0 +1,467 @@
from __future__ import annotations
import importlib
from codecs import IncrementalDecoder
from functools import lru_cache
from .constant import (
FREQUENCIES,
KO_NAMES,
LANGUAGE_SUPPORTED_COUNT,
TOO_SMALL_SEQUENCE,
ZH_NAMES,
_FREQUENCIES_SET,
_FREQUENCIES_RANK,
)
from .md import _ASCII_CHAR_INFO, _char_info, is_suspiciously_successive_range
from .models import CoherenceMatches
from .utils import (
is_multi_byte_encoding,
is_unicode_range_secondary,
)
def encoding_unicode_range(iana_name: str) -> list[str]:
"""
Return associated unicode ranges in a single byte code page.
"""
if is_multi_byte_encoding(iana_name):
raise OSError( # Defensive:
"Function not supported on multi-byte code page"
)
decoder = importlib.import_module(f"encodings.{iana_name}").IncrementalDecoder
p: IncrementalDecoder = decoder(errors="ignore")
seen_ranges: dict[str, int] = {}
character_count: int = 0
for i in range(0x40, 0xFF):
chunk: str = p.decode(bytes([i]))
if chunk:
chunk_codepoint = ord(chunk)
character_range: str | None = (
_ASCII_CHAR_INFO[chunk_codepoint].range
if chunk_codepoint < 128
else _char_info(chunk).range
)
if character_range is None:
continue
if not is_unicode_range_secondary(character_range):
if character_range not in seen_ranges:
seen_ranges[character_range] = 0
seen_ranges[character_range] += 1
character_count += 1
return sorted(
[
character_range
for character_range in seen_ranges
if seen_ranges[character_range] / character_count >= 0.15
]
)
def unicode_range_languages(primary_range: str) -> list[str]:
"""
Return inferred languages used with a unicode range.
"""
languages: list[str] = []
for language, characters in FREQUENCIES.items():
for character in characters:
codepoint = ord(character)
info = (
_ASCII_CHAR_INFO[codepoint]
if codepoint < 128
else _char_info(character)
)
if info.range == primary_range:
languages.append(language)
break
return languages
@lru_cache()
def encoding_languages(iana_name: str) -> list[str]:
"""
Single-byte encoding language association. Some code page are heavily linked to particular language(s).
This function does the correspondence.
"""
try:
unicode_ranges: list[str] = encoding_unicode_range(iana_name)
except ImportError: # Defensive: encoding unavailable on this build.
return []
primary_range: str | None = None
for specified_range in unicode_ranges:
if "Latin" not in specified_range:
primary_range = specified_range
break
if primary_range is None:
return ["Latin Based"]
return unicode_range_languages(primary_range)
@lru_cache()
def mb_encoding_languages(iana_name: str) -> list[str]:
"""
Multi-byte encoding language association. Some code page are heavily linked to particular language(s).
This function does the correspondence.
"""
if (
iana_name.startswith("shift_")
or iana_name.startswith("iso2022_jp")
or iana_name.startswith("euc_j")
or iana_name == "cp932"
):
return ["Japanese"]
if iana_name.startswith("gb") or iana_name in ZH_NAMES:
return ["Chinese"]
if iana_name.startswith("iso2022_kr") or iana_name in KO_NAMES:
return ["Korean"]
return []
@lru_cache(maxsize=LANGUAGE_SUPPORTED_COUNT)
def get_target_features(language: str) -> tuple[bool, bool]:
"""
Determine main aspects from a supported language if it contains accents and if is pure Latin.
"""
target_have_accents: bool = False
target_pure_latin: bool = True
for character in FREQUENCIES[language]:
codepoint = ord(character)
info = _ASCII_CHAR_INFO[codepoint] if codepoint < 128 else _char_info(character)
if not target_have_accents and info.accentuated:
target_have_accents = True
if target_pure_latin and not info.latin:
target_pure_latin = False
return target_have_accents, target_pure_latin
def alphabet_languages(
characters: list[str], ignore_non_latin: bool = False
) -> list[str]:
"""
Return associated languages associated to given characters.
"""
languages: list[tuple[str, float]] = []
characters_set: frozenset[str] = frozenset(characters)
source_have_accents = False
for character in characters:
codepoint = ord(character)
info = _ASCII_CHAR_INFO[codepoint] if codepoint < 128 else _char_info(character)
if info.accentuated:
source_have_accents = True
break
for language, language_characters in FREQUENCIES.items():
target_have_accents, target_pure_latin = get_target_features(language)
if ignore_non_latin and not target_pure_latin:
continue
if not target_have_accents and source_have_accents:
continue
character_count: int = len(language_characters)
character_match_count: int = len(_FREQUENCIES_SET[language] & characters_set)
ratio: float = character_match_count / character_count
if ratio >= 0.2:
languages.append((language, ratio))
languages = sorted(languages, key=lambda x: x[1], reverse=True)
return [compatible_language[0] for compatible_language in languages]
def characters_popularity_compare(
language: str, ordered_characters: list[str]
) -> float:
"""
Determine if a ordered characters list (by occurrence from most appearance to rarest) match a particular language.
The result is a ratio between 0. (absolutely no correspondence) and 1. (near perfect fit).
Beware that is function is not strict on the match in order to ease the detection. (Meaning close match is 1.)
"""
if language not in FREQUENCIES:
raise ValueError(f"{language} not available") # Defensive:
character_approved_count: int = 0
lang_rank: dict[str, int] = _FREQUENCIES_RANK[language]
ordered_characters_count: int = len(ordered_characters)
target_language_characters_count: int = len(FREQUENCIES[language])
large_alphabet: bool = target_language_characters_count > 26
large_alphabet_threshold: float = target_language_characters_count / 3
expected_projection_ratio: float = (
target_language_characters_count / ordered_characters_count
)
# Single pass: characters present in the language vocabulary, as
# (language rank, popularity rank) pairs. The scoring below only ever
# needs ranks, never the characters themselves.
common_lr: list[int] = []
common_orr: list[int] = []
for popularity_rank, character in enumerate(ordered_characters):
language_rank = lang_rank.get(character)
if language_rank is not None:
common_lr.append(language_rank)
common_orr.append(popularity_rank)
for character_rank_in_language, character_rank in zip(common_lr, common_orr):
character_rank_projection: int = int(character_rank * expected_projection_ratio)
if (
not large_alphabet
and abs(character_rank_projection - character_rank_in_language) > 4
):
continue
if (
large_alphabet
and abs(character_rank_projection - character_rank_in_language)
< large_alphabet_threshold
):
character_approved_count += 1
continue
if character_rank_in_language == 0:
# before_match_count is structurally 0 here (no pair can have a
# smaller language rank): the historic "before <= 4" acceptance
# always holds. (The symmetric "after_len == 0" case is
# impossible: language ranks are strictly below the language
# character count, hence after_len >= 1.)
character_approved_count += 1
continue
after_len: int = target_language_characters_count - character_rank_in_language
# Count how many characters appear "before" in both orderings, and
# how many appear "at or after" in both orderings. Both counts grow
# monotonically and the approval thresholds
# (before / rank >= 0.4 or after / after_len >= 0.4) are known
# upfront, expressed below as exact integer comparisons: exit as
# soon as one is crossed.
before_match_count: int = 0
after_match_count: int = 0
for lr_i, orr_i in zip(common_lr, common_orr):
if lr_i < character_rank_in_language:
if orr_i < character_rank:
before_match_count += 1
if 5 * before_match_count >= 2 * character_rank_in_language:
character_approved_count += 1
break
else:
if orr_i >= character_rank:
after_match_count += 1
if 5 * after_match_count >= 2 * after_len:
character_approved_count += 1
break
return character_approved_count / len(ordered_characters)
def alpha_unicode_split(decoded_sequence: str) -> list[str]:
"""
Given a decoded text sequence, return a list of str. Unicode range / alphabet separation.
Ex. a text containing English/Latin with a bit a Hebrew will return two items in the resulting list;
One containing the latin letters and the other hebrew.
"""
layers: dict[str, list[str]] = {}
# Fast path: track single-layer key to skip dict iteration for single-script text.
single_layer_key: str | None = None
multi_layer: bool = False
# Cache the last character_range and its resolved layer to avoid repeated
# is_suspiciously_successive_range calls for consecutive same-range chars.
prev_character_range: str | None = None
prev_layer_target: str | None = None
for character in decoded_sequence:
# Reuse the per-codepoint CharInfo cache: info.alpha and info.range
# are computed with the very same str.isalpha() / unicode_range()
# calls this loop historically made per character occurrence.
codepoint: int = ord(character)
if codepoint < 128:
info = _ASCII_CHAR_INFO[codepoint]
else:
info = _char_info(character)
if not info.alpha:
continue
character_range: str | None = info.range
if character_range is None:
continue
# Fast path: same range as previous character → reuse cached layer target.
if character_range == prev_character_range:
if prev_layer_target is not None:
layers[prev_layer_target].append(character)
continue
layer_target_range: str | None = None
if multi_layer:
for discovered_range in layers:
if not is_suspiciously_successive_range(
discovered_range, character_range
):
layer_target_range = discovered_range
break
elif single_layer_key is not None:
if not is_suspiciously_successive_range(single_layer_key, character_range):
layer_target_range = single_layer_key
if layer_target_range is None:
layer_target_range = character_range
if layer_target_range not in layers:
layers[layer_target_range] = []
if single_layer_key is None:
single_layer_key = layer_target_range
else:
multi_layer = True
layers[layer_target_range].append(character)
# Cache for next iteration
prev_character_range = character_range
prev_layer_target = layer_target_range
return ["".join(chars).lower() for chars in layers.values()]
def merge_coherence_ratios(results: list[CoherenceMatches]) -> CoherenceMatches:
"""
This function merge results previously given by the function coherence_ratio.
The return type is the same as coherence_ratio.
"""
per_language_ratios: dict[str, list[float]] = {}
for result in results:
for sub_result in result:
language, ratio = sub_result
if language not in per_language_ratios:
per_language_ratios[language] = [ratio]
continue
per_language_ratios[language].append(ratio)
merge = [
(
language,
round(
sum(per_language_ratios[language]) / len(per_language_ratios[language]),
4,
),
)
for language in per_language_ratios
]
return sorted(merge, key=lambda x: x[1], reverse=True)
def filter_alt_coherence_matches(results: CoherenceMatches) -> CoherenceMatches:
"""
We shall NOT return "English—" in CoherenceMatches because it is an alternative
of "English". This function only keeps the best match and remove the em-dash in it.
"""
index_results: dict[str, list[float]] = dict()
for result in results:
language, ratio = result
no_em_name: str = language.replace("—", "")
if no_em_name not in index_results:
index_results[no_em_name] = []
index_results[no_em_name].append(ratio)
if any(len(index_results[e]) > 1 for e in index_results):
filtered_results: CoherenceMatches = []
for language in index_results:
filtered_results.append((language, max(index_results[language])))
return filtered_results
return results
def coherence_ratio(
decoded_sequence: str, threshold: float = 0.1, lg_inclusion: str | None = None
) -> CoherenceMatches:
"""
Detect ANY language that can be identified in given sequence. The sequence will be analysed by layers.
A layer = Character extraction by alphabets/ranges.
"""
results: list[tuple[str, float]] = []
ignore_non_latin: bool = False
sufficient_match_count: int = 0
lg_inclusion_list = lg_inclusion.split(",") if lg_inclusion is not None else []
if "Latin Based" in lg_inclusion_list:
ignore_non_latin = True
lg_inclusion_list.remove("Latin Based")
for layer in alpha_unicode_split(decoded_sequence):
# Native counting + stable sort reproduce Counter.most_common()
# ordering exactly (ties keep first-appearance order) without the
# interpreted Counter machinery in the compiled hot path.
char_counts: dict[str, int] = {}
for layer_character in layer:
char_counts[layer_character] = char_counts.get(layer_character, 0) + 1
character_count: int = len(layer)
if character_count <= TOO_SMALL_SEQUENCE:
continue
popular_character_ordered: list[str] = [
item[0]
for item in sorted(
char_counts.items(), key=lambda item: item[1], reverse=True
)
]
for language in lg_inclusion_list or alphabet_languages(
popular_character_ordered, ignore_non_latin
):
ratio: float = characters_popularity_compare(
language, popular_character_ordered
)
if ratio < threshold:
continue
elif ratio >= 0.8:
sufficient_match_count += 1
results.append((language, round(ratio, 4)))
if sufficient_match_count >= 3:
break
return sorted(
filter_alt_coherence_matches(results), key=lambda x: x[1], reverse=True
)
@@ -0,0 +1,8 @@
from __future__ import annotations
from .__main__ import cli_detect, query_yes_no
__all__ = (
"cli_detect",
"query_yes_no",
)
@@ -0,0 +1,363 @@
from __future__ import annotations
import argparse
import sys
import typing
from os.path import abspath, basename, dirname, join, realpath
from platform import python_version
from unicodedata import unidata_version
import charset_normalizer.md as md_module
from charset_normalizer import from_fp
from charset_normalizer.models import CliDetectionResult
from charset_normalizer.version import __version__
def query_yes_no(question: str, default: str = "yes") -> bool: # Defensive:
"""Ask a yes/no question via input() and return the answer as a bool."""
prompt = " [Y/n] " if default == "yes" else " [y/N] "
while True:
choice = input(question + prompt).strip().lower()
if not choice:
return default == "yes"
if choice in ("y", "yes"):
return True
if choice in ("n", "no"):
return False
print("Please respond with 'y' or 'n'.")
class FileType:
"""Factory for creating file object types
Instances of FileType are typically passed as type= arguments to the
ArgumentParser add_argument() method.
Keyword Arguments:
- mode -- A string indicating how the file is to be opened. Accepts the
same values as the builtin open() function.
- bufsize -- The file's desired buffer size. Accepts the same values as
the builtin open() function.
- encoding -- The file's encoding. Accepts the same values as the
builtin open() function.
- errors -- A string indicating how encoding and decoding errors are to
be handled. Accepts the same value as the builtin open() function.
Backported from CPython 3.12
"""
def __init__(
self,
mode: str = "r",
bufsize: int = -1,
encoding: str | None = None,
errors: str | None = None,
):
self._mode = mode
self._bufsize = bufsize
self._encoding = encoding
self._errors = errors
def __call__(self, string: str) -> typing.IO: # type: ignore[type-arg]
# the special argument "-" means sys.std{in,out}
if string == "-":
if "r" in self._mode:
return sys.stdin.buffer if "b" in self._mode else sys.stdin
elif any(c in self._mode for c in "wax"):
return sys.stdout.buffer if "b" in self._mode else sys.stdout
else:
msg = f'argument "-" with mode {self._mode}'
raise ValueError(msg)
# all other arguments are used as file names
try:
return open(string, self._mode, self._bufsize, self._encoding, self._errors)
except OSError as e:
message = f"can't open '{string}': {e}"
raise argparse.ArgumentTypeError(message)
def __repr__(self) -> str:
args = self._mode, self._bufsize
kwargs = [("encoding", self._encoding), ("errors", self._errors)]
args_str = ", ".join(
[repr(arg) for arg in args if arg != -1]
+ [f"{kw}={arg!r}" for kw, arg in kwargs if arg is not None]
)
return f"{type(self).__name__}({args_str})"
def cli_detect(argv: list[str] | None = None) -> int:
"""
CLI assistant using ARGV and ArgumentParser
:param argv:
:return: 0 if everything is fine, anything else equal trouble
"""
parser = argparse.ArgumentParser(
description="The Real First Universal Charset Detector. "
"Discover originating encoding used on text file. "
"Normalize text to unicode."
)
parser.add_argument(
"files", type=FileType("rb"), nargs="+", help="File(s) to be analysed"
)
parser.add_argument(
"-v",
"--verbose",
action="store_true",
default=False,
dest="verbose",
help="Display complementary information about file if any. "
"Stdout will contain logs about the detection process.",
)
parser.add_argument(
"-a",
"--with-alternative",
action="store_true",
default=False,
dest="alternatives",
help="Output complementary possibilities if any. Top-level JSON WILL be a list.",
)
parser.add_argument(
"-n",
"--normalize",
action="store_true",
default=False,
dest="normalize",
help="Permit to normalize input file. If not set, program does not write anything.",
)
parser.add_argument(
"-m",
"--minimal",
action="store_true",
default=False,
dest="minimal",
help="Only output the charset detected to STDOUT. Disabling JSON output.",
)
parser.add_argument(
"-r",
"--replace",
action="store_true",
default=False,
dest="replace",
help="Replace file when trying to normalize it instead of creating a new one.",
)
parser.add_argument(
"-f",
"--force",
action="store_true",
default=False,
dest="force",
help="Replace file without asking if you are sure, use this flag with caution.",
)
parser.add_argument(
"-i",
"--no-preemptive",
action="store_true",
default=False,
dest="no_preemptive",
help="Disable looking at a charset declaration to hint the detector.",
)
parser.add_argument(
"-t",
"--threshold",
action="store",
default=0.2,
type=float,
dest="threshold",
help="Define a custom maximum amount of noise allowed in decoded content. 0. <= noise <= 1.",
)
parser.add_argument(
"--version",
action="version",
version="Charset-Normalizer {} - Python {} - Unicode {} - SpeedUp {}".format(
__version__,
python_version(),
unidata_version,
"OFF" if md_module.__file__.lower().endswith(".py") else "ON",
),
help="Show version information and exit.",
)
args = parser.parse_args(argv)
if args.replace is True and args.normalize is False:
if args.files:
for my_file in args.files:
my_file.close()
print("Use --replace in addition of --normalize only.", file=sys.stderr)
return 1
if args.force is True and args.replace is False:
if args.files:
for my_file in args.files:
my_file.close()
print("Use --force in addition of --replace only.", file=sys.stderr)
return 1
if args.threshold < 0.0 or args.threshold > 1.0:
if args.files:
for my_file in args.files:
my_file.close()
print("--threshold VALUE should be between 0. AND 1.", file=sys.stderr)
return 1
x_ = []
for my_file in args.files:
matches = from_fp(
my_file,
threshold=args.threshold,
explain=args.verbose,
preemptive_behaviour=args.no_preemptive is False,
)
best_guess = matches.best()
if best_guess is None:
print(
'Unable to identify originating encoding for "{}". {}'.format(
my_file.name,
(
"Maybe try increasing maximum amount of chaos."
if args.threshold < 1.0
else ""
),
),
file=sys.stderr,
)
x_.append(
CliDetectionResult(
abspath(my_file.name),
None,
[],
[],
"Unknown",
[],
False,
1.0,
0.0,
None,
True,
)
)
else:
cli_result = CliDetectionResult(
abspath(my_file.name),
best_guess.encoding,
best_guess.encoding_aliases,
[
cp
for cp in best_guess.could_be_from_charset
if cp != best_guess.encoding
],
best_guess.language,
best_guess.alphabets,
best_guess.bom,
best_guess.percent_chaos,
best_guess.percent_coherence,
None,
True,
)
x_.append(cli_result)
if len(matches) > 1 and args.alternatives:
for el in matches:
if el != best_guess:
x_.append(
CliDetectionResult(
abspath(my_file.name),
el.encoding,
el.encoding_aliases,
[
cp
for cp in el.could_be_from_charset
if cp != el.encoding
],
el.language,
el.alphabets,
el.bom,
el.percent_chaos,
el.percent_coherence,
None,
False,
)
)
if args.normalize is True:
if best_guess.encoding.startswith("utf") is True:
print(
'"{}" file does not need to be normalized, as it already came from unicode.'.format(
my_file.name
),
file=sys.stderr,
)
if my_file.closed is False:
my_file.close()
continue
dir_path = dirname(realpath(my_file.name))
file_name = basename(realpath(my_file.name))
o_: list[str] = file_name.split(".")
if args.replace is False:
o_.insert(-1, best_guess.encoding)
if my_file.closed is False:
my_file.close()
elif (
args.force is False
and query_yes_no(
'Are you sure to normalize "{}" by replacing it ?'.format(
my_file.name
),
"no",
)
is False
):
if my_file.closed is False:
my_file.close()
continue
try:
cli_result.unicode_path = join(dir_path, ".".join(o_))
with open(cli_result.unicode_path, "wb") as fp:
fp.write(best_guess.output())
except OSError as e: # Defensive:
print(str(e), file=sys.stderr)
if my_file.closed is False:
my_file.close()
return 2
if my_file.closed is False:
my_file.close()
if args.minimal is False:
from json import dumps
print(
dumps(
[el.__dict__ for el in x_] if len(x_) > 1 else x_[0].__dict__,
ensure_ascii=True,
indent=4,
)
)
else:
for my_file in args.files:
print(
", ".join(
[
el.encoding or "undefined"
for el in x_
if el.path == abspath(my_file.name)
]
)
)
return 0
if __name__ == "__main__": # Defensive:
cli_detect()
@@ -0,0 +1,79 @@
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from warnings import warn
from .api import from_bytes
from .constant import CHARDET_CORRESPONDENCE, TOO_SMALL_SEQUENCE
if TYPE_CHECKING:
from typing import TypedDict
class ResultDict(TypedDict):
encoding: str | None
language: str
confidence: float | None
def detect(
byte_str: bytes, should_rename_legacy: bool = False, **kwargs: Any
) -> ResultDict:
"""
chardet legacy method
Detect the encoding of the given byte string. It should be mostly backward-compatible.
Encoding name will match Chardet own writing whenever possible. (Not on encoding name unsupported by it)
This function is deprecated and should be used to migrate your project easily, consult the documentation for
further information. Not planned for removal.
:param byte_str: The byte sequence to examine.
:param should_rename_legacy: Should we rename legacy encodings
to their more modern equivalents?
"""
if len(kwargs):
warn(
f"charset-normalizer disregard arguments '{','.join(list(kwargs.keys()))}' in legacy function detect()"
)
if not isinstance(byte_str, (bytearray, bytes)):
raise TypeError( # pragma: nocover
f"Expected object of type bytes or bytearray, got: {type(byte_str)}"
)
if isinstance(byte_str, bytearray):
byte_str = bytes(byte_str)
r = from_bytes(byte_str).best()
encoding = r.encoding if r is not None else None
language = r.language if r is not None and r.language != "Unknown" else ""
confidence = 1.0 - r.chaos if r is not None else None
# automatically lower confidence
# on small bytes samples.
# https://github.com/jawah/charset_normalizer/issues/391
if (
confidence is not None
and confidence >= 0.9
and encoding
not in {
"utf_8",
"ascii",
}
and not r.bom # type: ignore[union-attr]
and len(byte_str) < TOO_SMALL_SEQUENCE
):
confidence -= 0.2
# Note: CharsetNormalizer does not return 'UTF-8-SIG' as the sig get stripped in the detection/normalization process
# but chardet does return 'utf-8-sig' and it is a valid codec name.
if r is not None and encoding == "utf_8" and r.bom:
encoding += "_sig"
if not should_rename_legacy and encoding in CHARDET_CORRESPONDENCE:
encoding = CHARDET_CORRESPONDENCE[encoding]
return {
"encoding": encoding,
"language": language,
"confidence": confidence,
}
@@ -0,0 +1,387 @@
from __future__ import annotations
from encodings.aliases import aliases
from re import sub
from typing import Any, Iterator, List, Tuple
from .constant import RE_POSSIBLE_ENCODING_INDICATION, TOO_BIG_SEQUENCE
from .utils import iana_name, is_multi_byte_encoding, unicode_range
class CharsetMatch:
def __init__(
self,
payload: bytes | bytearray,
guessed_encoding: str,
mean_mess_ratio: float,
has_sig_or_bom: bool,
languages: CoherenceMatches,
decoded_payload: str | None = None,
preemptive_declaration: str | None = None,
):
self._payload: bytes | bytearray = payload
self._encoding: str = guessed_encoding
self._mean_mess_ratio: float = mean_mess_ratio
self._languages: CoherenceMatches = languages
self._has_sig_or_bom: bool = has_sig_or_bom
self._unicode_ranges: list[str] | None = None
self._leaves: list[CharsetMatch] = []
self._mean_coherence_ratio: float = 0.0
self._output_payload: bytes | None = None
self._output_encoding: str | None = None
self._string: str | None = decoded_payload
self._preemptive_declaration: str | None = preemptive_declaration
def __eq__(self, other: object) -> bool:
if not isinstance(other, CharsetMatch):
if isinstance(other, str):
# Use non-strict iana_name so an operand that is not a known
# encoding alias compares unequal instead of raising, keeping
# __eq__ total as the data model requires (mirrors the lookup
# in CharsetMatches.__getitem__).
return iana_name(other, False) == self.encoding
return False
return self.encoding == other.encoding and self.fingerprint == other.fingerprint
def __lt__(self, other: object) -> bool:
"""
Implemented to make sorted available upon CharsetMatches items.
"""
if not isinstance(other, CharsetMatch):
raise ValueError
chaos_difference: float = abs(self.chaos - other.chaos)
coherence_difference: float = abs(self.coherence - other.coherence)
# Below 0.5% difference --> Use Coherence
if chaos_difference < 0.005 and coherence_difference > 0.02:
return self.coherence > other.coherence
elif chaos_difference < 0.005 and coherence_difference <= 0.02:
# When having a difficult decision, use the result that decoded as many multi-byte as possible.
# preserve RAM usage!
if len(self._payload) >= TOO_BIG_SEQUENCE:
return self.chaos < other.chaos
return self.multi_byte_usage > other.multi_byte_usage
return self.chaos < other.chaos
@property
def multi_byte_usage(self) -> float:
# Empty payloads are valid (from_bytes(b"") returns a match); avoid /0.
raw_len = len(self.raw)
if raw_len == 0:
return 0.0
return 1.0 - (len(str(self)) / raw_len)
def __str__(self) -> str:
# Lazy Str Loading
if self._string is None:
self._string = str(self._payload, self._encoding, "strict")
# UTF-7 BOM is encoded in modified Base64 whose byte boundary
# can overlap with the next character, so raw-byte stripping
# is unreliable. Strip the decoded BOM character instead.
if (
self._has_sig_or_bom
and self._encoding == "utf_7"
and self._string
and self._string[0] == "\ufeff"
):
self._string = self._string[1:]
return self._string
def __repr__(self) -> str:
return f"<CharsetMatch '{self.encoding}' fp({self.fingerprint})>"
def add_submatch(self, other: CharsetMatch) -> None:
if not isinstance(other, CharsetMatch) or other == self:
raise ValueError(
"Unable to add instance <{}> as a submatch of a CharsetMatch".format(
other.__class__
)
)
other._string = None # Unload RAM usage; dirty trick.
self._leaves.append(other)
@property
def encoding(self) -> str:
return self._encoding
@property
def encoding_aliases(self) -> list[str]:
"""
Encoding name are known by many name, using this could help when searching for IBM855 when it's listed as CP855.
"""
also_known_as: list[str] = []
for u, p in aliases.items():
if self.encoding == u:
also_known_as.append(p)
elif self.encoding == p:
also_known_as.append(u)
return also_known_as
@property
def bom(self) -> bool:
return self._has_sig_or_bom
@property
def byte_order_mark(self) -> bool:
return self._has_sig_or_bom
@property
def languages(self) -> list[str]:
"""
Return the complete list of possible languages found in decoded sequence.
Usually not really useful. Returned list may be empty even if 'language' property return something != 'Unknown'.
"""
return [e[0] for e in self._languages]
@property
def language(self) -> str:
"""
Most probable language found in decoded sequence. If none were detected or inferred, the property will return
"Unknown".
"""
if not self._languages:
# Trying to infer the language based on the given encoding
# Its either English or we should not pronounce ourselves in certain cases.
if "ascii" in self.could_be_from_charset:
return "English"
# doing it there to avoid circular import
from charset_normalizer.cd import encoding_languages, mb_encoding_languages
languages = (
mb_encoding_languages(self.encoding)
if is_multi_byte_encoding(self.encoding)
else encoding_languages(self.encoding)
)
if len(languages) == 0 or "Latin Based" in languages:
return "Unknown"
return languages[0]
return self._languages[0][0]
@property
def chaos(self) -> float:
return self._mean_mess_ratio
@property
def coherence(self) -> float:
if not self._languages:
return 0.0
return self._languages[0][1]
@property
def percent_chaos(self) -> float:
return round(self.chaos * 100, ndigits=3)
@property
def percent_coherence(self) -> float:
return round(self.coherence * 100, ndigits=3)
@property
def raw(self) -> bytes | bytearray:
"""
Original untouched bytes.
"""
return self._payload
@property
def submatch(self) -> list[CharsetMatch]:
return self._leaves
@property
def has_submatch(self) -> bool:
return len(self._leaves) > 0
@property
def alphabets(self) -> list[str]:
if self._unicode_ranges is not None:
return self._unicode_ranges
# list detected ranges
detected_ranges: list[str | None] = [unicode_range(char) for char in str(self)]
# filter and sort
self._unicode_ranges = sorted(list({r for r in detected_ranges if r}))
return self._unicode_ranges
@property
def could_be_from_charset(self) -> list[str]:
"""
The complete list of encoding that output the exact SAME str result and therefore could be the originating
encoding.
This list does include the encoding available in property 'encoding'.
"""
return [self._encoding] + [m.encoding for m in self._leaves]
def output(self, encoding: str = "utf_8") -> bytes:
"""
Method to get re-encoded bytes payload using given target encoding. Default to UTF-8.
Any errors will be simply ignored by the encoder NOT replaced.
"""
if self._output_encoding is None or self._output_encoding != encoding:
self._output_encoding = encoding
decoded_string = str(self)
if (
self._preemptive_declaration is not None
and self._preemptive_declaration.lower()
not in ["utf-8", "utf8", "utf_8"]
):
patched_header = sub(
RE_POSSIBLE_ENCODING_INDICATION,
lambda m: m.string[m.span()[0] : m.span()[1]].replace(
m.groups()[0],
iana_name(self._output_encoding).replace("_", "-"), # type: ignore[arg-type]
),
decoded_string[:8192],
count=1,
)
decoded_string = patched_header + decoded_string[8192:]
self._output_payload = decoded_string.encode(encoding, "replace")
return self._output_payload # type: ignore
@property
def fingerprint(self) -> int:
"""
Retrieve a hash fingerprint of the decoded payload, used for deduplication.
"""
return hash(str(self))
class CharsetMatches:
"""
Container with every CharsetMatch items ordered by default from most probable to the less one.
Act like a list(iterable) but does not implements all related methods.
"""
def __init__(self, results: list[CharsetMatch] | None = None):
self._results: list[CharsetMatch] = sorted(results) if results else []
self._is_sorted: bool = True
def _ensure_sorted(self) -> None:
if not self._is_sorted:
self._results.sort()
self._is_sorted = True
def __iter__(self) -> Iterator[CharsetMatch]:
self._ensure_sorted()
yield from self._results
def __getitem__(self, item: int | str) -> CharsetMatch:
"""
Retrieve a single item either by its position or encoding name (alias may be used here).
Raise KeyError upon invalid index or encoding not present in results.
"""
if isinstance(item, int):
self._ensure_sorted()
return self._results[item]
if isinstance(item, str):
item = iana_name(item, False)
for result in self._results:
if item in result.could_be_from_charset:
return result
raise KeyError
def __len__(self) -> int:
return len(self._results)
def __bool__(self) -> bool:
return len(self._results) > 0
def append(self, item: CharsetMatch) -> None:
"""
Insert a single match. Will be inserted accordingly to preserve sort.
Can be inserted as a submatch.
"""
if not isinstance(item, CharsetMatch):
raise ValueError(
"Cannot append instance '{}' to CharsetMatches".format(
str(item.__class__)
)
)
# We should disable the submatch factoring when the input file is too heavy (conserve RAM usage)
if len(item.raw) < TOO_BIG_SEQUENCE:
for match in self._results:
if match.fingerprint == item.fingerprint and match.chaos == item.chaos:
match.add_submatch(item)
return
self._results.append(item)
self._is_sorted = False
def best(self) -> CharsetMatch | None:
"""
Simply return the first match. Strict equivalent to matches[0].
"""
if not self._results:
return None
self._ensure_sorted()
return self._results[0]
def first(self) -> CharsetMatch | None:
"""
Redundant method, call the method best(). Kept for BC reasons.
"""
return self.best()
CoherenceMatch = Tuple[str, float]
CoherenceMatches = List[CoherenceMatch]
class CliDetectionResult:
def __init__(
self,
path: str,
encoding: str | None,
encoding_aliases: list[str],
alternative_encodings: list[str],
language: str,
alphabets: list[str],
has_sig_or_bom: bool,
chaos: float,
coherence: float,
unicode_path: str | None,
is_preferred: bool,
):
self.path: str = path
self.unicode_path: str | None = unicode_path
self.encoding: str | None = encoding
self.encoding_aliases: list[str] = encoding_aliases
self.alternative_encodings: list[str] = alternative_encodings
self.language: str = language
self.alphabets: list[str] = alphabets
self.has_sig_or_bom: bool = has_sig_or_bom
self.chaos: float = chaos
self.coherence: float = coherence
self.is_preferred: bool = is_preferred
@property
def __dict__(self) -> dict[str, Any]: # type: ignore
return {
"path": self.path,
"encoding": self.encoding,
"encoding_aliases": self.encoding_aliases,
"alternative_encodings": self.alternative_encodings,
"language": self.language,
"alphabets": self.alphabets,
"has_sig_or_bom": self.has_sig_or_bom,
"chaos": self.chaos,
"coherence": self.coherence,
"unicode_path": self.unicode_path,
"is_preferred": self.is_preferred,
}
def to_json(self) -> str:
from json import dumps
return dumps(self.__dict__, ensure_ascii=True, indent=4)
@@ -0,0 +1,452 @@
from __future__ import annotations
import importlib
import logging
import unicodedata
from bisect import bisect_right
from codecs import IncrementalDecoder
from functools import lru_cache
from typing import Generator
from .constant import (
ENCODING_MARKS,
IANA_SUPPORTED_SIMILAR,
RE_POSSIBLE_ENCODING_INDICATION,
UNICODE_RANGES_COMBINED,
_SECONDARY_RANGE_NAMES,
COMMON_CJK_CHARACTERS,
_LATIN,
_CJK,
_HANGUL,
_KATAKANA,
_HIRAGANA,
_HALFWIDTH_KATAKANA,
_THAI,
_ARABIC,
_ARABIC_ISOLATED_FORM,
_LIGATURE,
_SUPERSCRIPT,
_SENTENCE_OPEN_PUNCTUATION,
_ACCENT_KEYWORDS,
_ACCENTUATED,
_KNOWN_MB_DECODERS,
_KNOWN_MB_CLASSES,
_IANA_NAMES,
_MULTIBYTE_SEARCH_RADIUS,
)
def _character_flags(character: str) -> int:
"""Compute all name-based classification flags with a single unicodedata.name() call."""
try:
desc: str = unicodedata.name(character)
except ValueError:
return 0
flags: int = 0
if "LATIN" in desc:
flags |= _LATIN
if "CJK" in desc:
flags |= _CJK
if "HANGUL" in desc:
flags |= _HANGUL
if "KATAKANA" in desc:
flags |= _KATAKANA
if "HALFWIDTH" in desc:
flags |= _HALFWIDTH_KATAKANA
if "HIRAGANA" in desc:
flags |= _HIRAGANA
if "THAI" in desc:
flags |= _THAI
if "ARABIC" in desc:
flags |= _ARABIC
if "ISOLATED FORM" in desc:
flags |= _ARABIC_ISOLATED_FORM
if "LIGATURE" in desc or desc.endswith("LETTER AE"):
flags |= _LIGATURE
if "SUPERSCRIPT" in desc:
flags |= _SUPERSCRIPT
if desc in {"INVERTED QUESTION MARK", "INVERTED EXCLAMATION MARK"}:
flags |= _SENTENCE_OPEN_PUNCTUATION
for kw in _ACCENT_KEYWORDS:
if kw in desc:
flags |= _ACCENTUATED
break
return flags
def is_accentuated(character: str) -> bool:
return bool(_character_flags(character) & _ACCENTUATED)
def remove_accent(character: str) -> str:
decomposed: str = unicodedata.decomposition(character)
if not decomposed:
return character
codes: list[str] = decomposed.split(" ")
return chr(int(codes[0], 16))
# Pre-built sorted lookup table for O(log n) binary search in unicode_range().
# Each entry is (range_start, range_end_exclusive, range_name).
_UNICODE_RANGES_SORTED: list[tuple[int, int, str]] = sorted(
(ord_range.start, ord_range.stop, name)
for name, ord_range in UNICODE_RANGES_COMBINED.items()
)
_UNICODE_RANGE_STARTS: list[int] = [e[0] for e in _UNICODE_RANGES_SORTED]
def unicode_range(character: str) -> str | None:
"""
Retrieve the Unicode range official name from a single character.
"""
character_ord: int = ord(character)
if character_ord < 32:
return "Control character"
if character_ord < 128:
return "Basic Latin"
# Binary search: find the rightmost range whose start <= character_ord
idx = bisect_right(_UNICODE_RANGE_STARTS, character_ord) - 1
if idx >= 0:
start, stop, name = _UNICODE_RANGES_SORTED[idx]
if character_ord < stop:
return name
return None
def is_latin(character: str) -> bool:
return bool(_character_flags(character) & _LATIN)
def is_punctuation(character: str) -> bool:
character_category: str = unicodedata.category(character)
if "P" in character_category:
return True
character_range: str | None = unicode_range(character)
if character_range is None:
return False
return "Punctuation" in character_range
def is_symbol(character: str) -> bool:
character_category: str = unicodedata.category(character)
if "S" in character_category or "N" in character_category:
return True
character_range: str | None = unicode_range(character)
if character_range is None:
return False
return "Forms" in character_range and character_category != "Lo"
def is_emoticon(character: str) -> bool:
character_range: str | None = unicode_range(character)
if character_range is None:
return False
return "Emoticons" in character_range or "Pictographs" in character_range
def is_separator(character: str) -> bool:
if character.isspace() or character in {"|", "+", "<", ">"}:
return True
character_category: str = unicodedata.category(character)
return "Z" in character_category or character_category in {"Po", "Pd", "Pc"}
def is_case_variable(character: str) -> bool:
return character.islower() != character.isupper()
def is_cjk(character: str) -> bool:
return bool(_character_flags(character) & _CJK)
def is_hiragana(character: str) -> bool:
return bool(_character_flags(character) & _HIRAGANA)
def is_katakana(character: str) -> bool:
return bool(_character_flags(character) & _KATAKANA)
def is_hangul(character: str) -> bool:
return bool(_character_flags(character) & _HANGUL)
def is_thai(character: str) -> bool:
return bool(_character_flags(character) & _THAI)
def is_arabic(character: str) -> bool:
return bool(_character_flags(character) & _ARABIC)
def is_arabic_isolated_form(character: str) -> bool:
return bool(_character_flags(character) & _ARABIC_ISOLATED_FORM)
def is_cjk_uncommon(character: str) -> bool:
return character not in COMMON_CJK_CHARACTERS
def is_unicode_range_secondary(range_name: str) -> bool:
return range_name in _SECONDARY_RANGE_NAMES
def is_unprintable(character: str) -> bool:
return (
not character.isspace() # includes \n \t \r \v
and not character.isprintable()
and character != "\x1a" # Why? Its the ASCII substitute character.
and character != "\ufeff" # bug discovered in Python,
# Zero Width No-Break Space located in Arabic Presentation Forms-B, Unicode 1.1 not acknowledged as space.
)
def any_specified_encoding(
sequence: bytes | bytearray, search_zone: int = 8192
) -> str | None:
"""
Extract using ASCII-only decoder any specified encoding in the first n-bytes.
"""
if not isinstance(sequence, (bytes, bytearray)):
raise TypeError
seq_len: int = len(sequence)
# Cheap literal pre-filter.
search_bytes = sequence[: min(seq_len, search_zone)]
lowered_bytes = search_bytes.lower()
if b"coding" not in lowered_bytes and b"charset" not in lowered_bytes:
return None
decoded_zone: str = search_bytes.decode("ascii", errors="ignore")
for match in RE_POSSIBLE_ENCODING_INDICATION.finditer(decoded_zone):
specified_encoding = match.group(1).lower().replace("-", "_")
encoding_iana = _IANA_NAMES.get(specified_encoding)
if encoding_iana is not None:
return encoding_iana
return None
@lru_cache(maxsize=None)
def is_multi_byte_encoding(name: str) -> bool:
"""
Verify is a specific encoding is a multi byte one based on it IANA name
"""
if name in _KNOWN_MB_DECODERS:
return True
# Besides the Unicode family above, every multibyte codec shipped with
# Python is implemented by _multibytecodec through exactly one of the six
# cjkcodecs providers below. Probing those providers directly (getcodec)
# classifies a name without importing its "encodings.<name>" module:
# classifying the whole IANA_SUPPORTED list would otherwise import many
# modules and dominate "import charset_normalizer" wall time.
# see https://github.com/jawah/charset_normalizer/issues/742
for provider in _KNOWN_MB_CLASSES:
try:
importlib.import_module(provider).getcodec(name) # type: ignore[attr-defined]
except (ImportError, AttributeError, LookupError): # Defensive: edge cases
continue
return True
return False
def identify_sig_or_bom(sequence: bytes | bytearray) -> tuple[str | None, bytes]:
"""
Identify and extract SIG/BOM in given sequence.
"""
for iana_encoding in ENCODING_MARKS:
marks: bytes | list[bytes] = ENCODING_MARKS[iana_encoding]
if isinstance(marks, bytes):
marks = [marks]
for mark in marks:
if sequence.startswith(mark):
return iana_encoding, mark
return None, b""
def should_strip_sig_or_bom(iana_encoding: str) -> bool:
return iana_encoding not in {"utf_16", "utf_32"}
def iana_name(cp_name: str, strict: bool = True) -> str:
"""Returns the Python normalized encoding name (Not the IANA official name)."""
cp_name = cp_name.lower().replace("-", "_")
encoding_iana = _IANA_NAMES.get(cp_name)
if encoding_iana is not None:
return encoding_iana
if strict:
raise ValueError(f"Unable to retrieve IANA for '{cp_name}'")
return cp_name
def cp_similarity(iana_name_a: str, iana_name_b: str) -> float:
if is_multi_byte_encoding(iana_name_a) or is_multi_byte_encoding(iana_name_b):
return 0.0
decoder_a = importlib.import_module(f"encodings.{iana_name_a}").IncrementalDecoder
decoder_b = importlib.import_module(f"encodings.{iana_name_b}").IncrementalDecoder
id_a: IncrementalDecoder = decoder_a(errors="ignore")
id_b: IncrementalDecoder = decoder_b(errors="ignore")
character_match_count: int = 0
for i in range(256):
to_be_decoded: bytes = bytes([i])
if id_a.decode(to_be_decoded) == id_b.decode(to_be_decoded):
character_match_count += 1
return character_match_count / 256
def is_cp_similar(iana_name_a: str, iana_name_b: str) -> bool:
"""
Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using
the function cp_similarity.
"""
return (
iana_name_a in IANA_SUPPORTED_SIMILAR
and iana_name_b in IANA_SUPPORTED_SIMILAR[iana_name_a]
)
def set_logging_handler(
name: str = "charset_normalizer",
level: int = logging.INFO,
format_string: str = "%(asctime)s | %(levelname)s | %(message)s",
) -> None:
logger = logging.getLogger(name)
logger.setLevel(level)
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter(format_string))
logger.addHandler(handler)
def cut_sequence_chunks(
sequences: bytes | bytearray,
encoding_iana: str,
offsets: range,
chunk_size: int,
bom_or_sig_available: bool,
strip_sig_or_bom: bool,
sig_payload: bytes,
is_multi_byte_decoder: bool,
decoded_payload: str | None = None,
deferred_decoding: bool = False,
) -> Generator[str, None, None]:
# iso2022 codec is stateful, generic mb cuter isn't going to cut it!
if decoded_payload and encoding_iana.startswith("iso2022_"):
decoded_length = len(decoded_payload)
sequence_length = len(sequences)
for i in offsets:
decoded_offset = i * decoded_length // sequence_length
chunk = decoded_payload[decoded_offset : decoded_offset + chunk_size]
if not chunk:
break
yield chunk
elif decoded_payload and not is_multi_byte_decoder:
for i in offsets:
chunk = decoded_payload[i : i + chunk_size]
if not chunk:
break
yield chunk
elif deferred_decoding:
# Deferred single-byte probing: the whole payload is not decoded
# yet. Single-byte codecs are stateless (1 byte == 1 char), hence
# decode(base)[i:j] == decode(base[i:j]): slicing the raw bytes
# yields exactly the chunks the branch above would have produced,
# short trailing chunks included, and raises UnicodeDecodeError on
# invalid bytes just like the whole-payload decode would.
base_bytes = (
sequences if not strip_sig_or_bom else sequences[len(sig_payload) :]
)
for i in offsets:
cut_sequence = base_bytes[i : i + chunk_size]
if not cut_sequence:
break
yield str(cut_sequence, encoding_iana)
else:
for i in offsets:
chunk_end = i + chunk_size
if chunk_end > len(sequences) + 8:
continue
cut_sequence = sequences[i : i + chunk_size]
if bom_or_sig_available and not strip_sig_or_bom:
cut_sequence = sig_payload + cut_sequence
chunk = cut_sequence.decode(
encoding_iana,
errors="ignore" if is_multi_byte_decoder else "strict",
)
# multi-byte bad cutting detector and adjustment
# not the cleanest way to perform that fix but clever enough for now.
if is_multi_byte_decoder and i > 0:
chunk_partial_size_chk: int = min(chunk_size, 16)
chunk_prefix = chunk[:chunk_partial_size_chk]
found_nearby = False
if decoded_payload:
decoded_length = len(decoded_payload)
expected_offset = i * decoded_length // len(sequences)
search_start = max(0, expected_offset - _MULTIBYTE_SEARCH_RADIUS)
search_end = min(
decoded_length, expected_offset + _MULTIBYTE_SEARCH_RADIUS
)
found_nearby = (
decoded_payload.find(chunk_prefix, search_start, search_end)
>= 0
)
if (
decoded_payload
and not found_nearby
and chunk_prefix not in decoded_payload
):
for j in range(i, i - 4, -1):
cut_sequence = sequences[j:chunk_end]
if bom_or_sig_available and not strip_sig_or_bom:
cut_sequence = sig_payload + cut_sequence
chunk = cut_sequence.decode(encoding_iana, errors="ignore")
if chunk[:chunk_partial_size_chk] in decoded_payload:
break
yield chunk
@@ -0,0 +1,8 @@
"""
Expose version
"""
from __future__ import annotations
__version__ = "3.5.1"
VERSION = __version__.split(".")
@@ -0,0 +1,154 @@
# defusedxml
#
# Copyright (c) 2013 by Christian Heimes <[email protected]>
# Licensed to PSF under a Contributor Agreement.
# See https://www.python.org/psf/license for licensing details.
"""Defused xml.etree.ElementTree facade
"""
from __future__ import print_function, absolute_import
import sys
import warnings
from xml.etree.ElementTree import ParseError
from xml.etree.ElementTree import TreeBuilder as _TreeBuilder
from xml.etree.ElementTree import parse as _parse
from xml.etree.ElementTree import tostring
from .common import PY3
if PY3:
import importlib
else:
from xml.etree.ElementTree import XMLParser as _XMLParser
from xml.etree.ElementTree import iterparse as _iterparse
from .common import (
DTDForbidden,
EntitiesForbidden,
ExternalReferenceForbidden,
_generate_etree_functions,
)
__origin__ = "xml.etree.ElementTree"
def _get_py3_cls():
"""Python 3.3 hides the pure Python code but defusedxml requires it.
The code is based on test.support.import_fresh_module().
"""
pymodname = "xml.etree.ElementTree"
cmodname = "_elementtree"
pymod = sys.modules.pop(pymodname, None)
cmod = sys.modules.pop(cmodname, None)
sys.modules[cmodname] = None
try:
pure_pymod = importlib.import_module(pymodname)
finally:
# restore module
sys.modules[pymodname] = pymod
if cmod is not None:
sys.modules[cmodname] = cmod
else:
sys.modules.pop(cmodname, None)
# restore attribute on original package
etree_pkg = sys.modules["xml.etree"]
if pymod is not None:
etree_pkg.ElementTree = pymod
elif hasattr(etree_pkg, "ElementTree"):
del etree_pkg.ElementTree
_XMLParser = pure_pymod.XMLParser
_iterparse = pure_pymod.iterparse
# patch pure module to use ParseError from C extension
pure_pymod.ParseError = ParseError
return _XMLParser, _iterparse
if PY3:
_XMLParser, _iterparse = _get_py3_cls()
_sentinel = object()
class DefusedXMLParser(_XMLParser):
def __init__(
self,
html=_sentinel,
target=None,
encoding=None,
forbid_dtd=False,
forbid_entities=True,
forbid_external=True,
):
# Python 2.x old style class
_XMLParser.__init__(self, target=target, encoding=encoding)
if html is not _sentinel:
# the 'html' argument has been deprecated and ignored in all
# supported versions of Python. Python 3.8 finally removed it.
if html:
raise TypeError("'html=True' is no longer supported.")
else:
warnings.warn(
"'html' keyword argument is no longer supported. Pass "
"in arguments as keyword arguments.",
category=DeprecationWarning,
)
self.forbid_dtd = forbid_dtd
self.forbid_entities = forbid_entities
self.forbid_external = forbid_external
if PY3:
parser = self.parser
else:
parser = self._parser
if self.forbid_dtd:
parser.StartDoctypeDeclHandler = self.defused_start_doctype_decl
if self.forbid_entities:
parser.EntityDeclHandler = self.defused_entity_decl
parser.UnparsedEntityDeclHandler = self.defused_unparsed_entity_decl
if self.forbid_external:
parser.ExternalEntityRefHandler = self.defused_external_entity_ref_handler
def defused_start_doctype_decl(self, name, sysid, pubid, has_internal_subset):
raise DTDForbidden(name, sysid, pubid)
def defused_entity_decl(
self, name, is_parameter_entity, value, base, sysid, pubid, notation_name
):
raise EntitiesForbidden(name, value, base, sysid, pubid, notation_name)
def defused_unparsed_entity_decl(self, name, base, sysid, pubid, notation_name):
# expat 1.2
raise EntitiesForbidden(name, None, base, sysid, pubid, notation_name) # pragma: no cover
def defused_external_entity_ref_handler(self, context, base, sysid, pubid):
raise ExternalReferenceForbidden(context, base, sysid, pubid)
# aliases
# XMLParse is a typo, keep it for backwards compatibility
XMLTreeBuilder = XMLParse = XMLParser = DefusedXMLParser
parse, iterparse, fromstring = _generate_etree_functions(
DefusedXMLParser, _TreeBuilder, _parse, _iterparse
)
XML = fromstring
__all__ = [
"ParseError",
"XML",
"XMLParse",
"XMLParser",
"XMLTreeBuilder",
"fromstring",
"iterparse",
"parse",
"tostring",
]
@@ -0,0 +1,67 @@
# defusedxml
#
# Copyright (c) 2013 by Christian Heimes <[email protected]>
# Licensed to PSF under a Contributor Agreement.
# See https://www.python.org/psf/license for licensing details.
"""Defuse XML bomb denial of service vulnerabilities
"""
from __future__ import print_function, absolute_import
import warnings
from .common import (
DefusedXmlException,
DTDForbidden,
EntitiesForbidden,
ExternalReferenceForbidden,
NotSupportedError,
_apply_defusing,
)
def defuse_stdlib():
"""Monkey patch and defuse all stdlib packages
:warning: The monkey patch is an EXPERIMETNAL feature.
"""
defused = {}
with warnings.catch_warnings():
from . import cElementTree
from . import ElementTree
from . import minidom
from . import pulldom
from . import sax
from . import expatbuilder
from . import expatreader
from . import xmlrpc
xmlrpc.monkey_patch()
defused[xmlrpc] = None
defused_mods = [
cElementTree,
ElementTree,
minidom,
pulldom,
sax,
expatbuilder,
expatreader,
]
for defused_mod in defused_mods:
stdlib_mod = _apply_defusing(defused_mod)
defused[defused_mod] = stdlib_mod
return defused
__version__ = "0.7.1"
__all__ = [
"DefusedXmlException",
"DTDForbidden",
"EntitiesForbidden",
"ExternalReferenceForbidden",
"NotSupportedError",
]
@@ -0,0 +1,62 @@
# defusedxml
#
# Copyright (c) 2013 by Christian Heimes <[email protected]>
# Licensed to PSF under a Contributor Agreement.
# See https://www.python.org/psf/license for licensing details.
"""Defused xml.etree.cElementTree
"""
from __future__ import absolute_import
import warnings
from .common import _generate_etree_functions
from xml.etree.cElementTree import TreeBuilder as _TreeBuilder
from xml.etree.cElementTree import parse as _parse
from xml.etree.cElementTree import tostring
# iterparse from ElementTree!
from xml.etree.ElementTree import iterparse as _iterparse
# This module is an alias for ElementTree just like xml.etree.cElementTree
from .ElementTree import (
XML,
XMLParse,
XMLParser,
XMLTreeBuilder,
fromstring,
iterparse,
parse,
tostring,
DefusedXMLParser,
ParseError,
)
__origin__ = "xml.etree.cElementTree"
warnings.warn(
"defusedxml.cElementTree is deprecated, import from defusedxml.ElementTree instead.",
category=DeprecationWarning,
stacklevel=2,
)
# XMLParse is a typo, keep it for backwards compatibility
XMLTreeBuilder = XMLParse = XMLParser = DefusedXMLParser
parse, iterparse, fromstring = _generate_etree_functions(
DefusedXMLParser, _TreeBuilder, _parse, _iterparse
)
XML = fromstring
__all__ = [
"ParseError",
"XML",
"XMLParse",
"XMLParser",
"XMLTreeBuilder",
"fromstring",
"iterparse",
"parse",
"tostring",
]
@@ -0,0 +1,129 @@
# defusedxml
#
# Copyright (c) 2013 by Christian Heimes <[email protected]>
# Licensed to PSF under a Contributor Agreement.
# See https://www.python.org/psf/license for licensing details.
"""Common constants, exceptions and helpe functions
"""
import sys
import xml.parsers.expat
PY3 = sys.version_info[0] == 3
# Fail early when pyexpat is not installed correctly
if not hasattr(xml.parsers.expat, "ParserCreate"):
raise ImportError("pyexpat") # pragma: no cover
class DefusedXmlException(ValueError):
"""Base exception"""
def __repr__(self):
return str(self)
class DTDForbidden(DefusedXmlException):
"""Document type definition is forbidden"""
def __init__(self, name, sysid, pubid):
super(DTDForbidden, self).__init__()
self.name = name
self.sysid = sysid
self.pubid = pubid
def __str__(self):
tpl = "DTDForbidden(name='{}', system_id={!r}, public_id={!r})"
return tpl.format(self.name, self.sysid, self.pubid)
class EntitiesForbidden(DefusedXmlException):
"""Entity definition is forbidden"""
def __init__(self, name, value, base, sysid, pubid, notation_name):
super(EntitiesForbidden, self).__init__()
self.name = name
self.value = value
self.base = base
self.sysid = sysid
self.pubid = pubid
self.notation_name = notation_name
def __str__(self):
tpl = "EntitiesForbidden(name='{}', system_id={!r}, public_id={!r})"
return tpl.format(self.name, self.sysid, self.pubid)
class ExternalReferenceForbidden(DefusedXmlException):
"""Resolving an external reference is forbidden"""
def __init__(self, context, base, sysid, pubid):
super(ExternalReferenceForbidden, self).__init__()
self.context = context
self.base = base
self.sysid = sysid
self.pubid = pubid
def __str__(self):
tpl = "ExternalReferenceForbidden(system_id='{}', public_id={})"
return tpl.format(self.sysid, self.pubid)
class NotSupportedError(DefusedXmlException):
"""The operation is not supported"""
def _apply_defusing(defused_mod):
assert defused_mod is sys.modules[defused_mod.__name__]
stdlib_name = defused_mod.__origin__
__import__(stdlib_name, {}, {}, ["*"])
stdlib_mod = sys.modules[stdlib_name]
stdlib_names = set(dir(stdlib_mod))
for name, obj in vars(defused_mod).items():
if name.startswith("_") or name not in stdlib_names:
continue
setattr(stdlib_mod, name, obj)
return stdlib_mod
def _generate_etree_functions(DefusedXMLParser, _TreeBuilder, _parse, _iterparse):
"""Factory for functions needed by etree, dependent on whether
cElementTree or ElementTree is used."""
def parse(source, parser=None, forbid_dtd=False, forbid_entities=True, forbid_external=True):
if parser is None:
parser = DefusedXMLParser(
target=_TreeBuilder(),
forbid_dtd=forbid_dtd,
forbid_entities=forbid_entities,
forbid_external=forbid_external,
)
return _parse(source, parser)
def iterparse(
source,
events=None,
parser=None,
forbid_dtd=False,
forbid_entities=True,
forbid_external=True,
):
if parser is None:
parser = DefusedXMLParser(
target=_TreeBuilder(),
forbid_dtd=forbid_dtd,
forbid_entities=forbid_entities,
forbid_external=forbid_external,
)
return _iterparse(source, events, parser)
def fromstring(text, forbid_dtd=False, forbid_entities=True, forbid_external=True):
parser = DefusedXMLParser(
target=_TreeBuilder(),
forbid_dtd=forbid_dtd,
forbid_entities=forbid_entities,
forbid_external=forbid_external,
)
parser.feed(text)
return parser.close()
return parse, iterparse, fromstring
@@ -0,0 +1,107 @@
# defusedxml
#
# Copyright (c) 2013 by Christian Heimes <[email protected]>
# Licensed to PSF under a Contributor Agreement.
# See https://www.python.org/psf/license for licensing details.
"""Defused xml.dom.expatbuilder
"""
from __future__ import print_function, absolute_import
from xml.dom.expatbuilder import ExpatBuilder as _ExpatBuilder
from xml.dom.expatbuilder import Namespaces as _Namespaces
from .common import DTDForbidden, EntitiesForbidden, ExternalReferenceForbidden
__origin__ = "xml.dom.expatbuilder"
class DefusedExpatBuilder(_ExpatBuilder):
"""Defused document builder"""
def __init__(
self, options=None, forbid_dtd=False, forbid_entities=True, forbid_external=True
):
_ExpatBuilder.__init__(self, options)
self.forbid_dtd = forbid_dtd
self.forbid_entities = forbid_entities
self.forbid_external = forbid_external
def defused_start_doctype_decl(self, name, sysid, pubid, has_internal_subset):
raise DTDForbidden(name, sysid, pubid)
def defused_entity_decl(
self, name, is_parameter_entity, value, base, sysid, pubid, notation_name
):
raise EntitiesForbidden(name, value, base, sysid, pubid, notation_name)
def defused_unparsed_entity_decl(self, name, base, sysid, pubid, notation_name):
# expat 1.2
raise EntitiesForbidden(name, None, base, sysid, pubid, notation_name) # pragma: no cover
def defused_external_entity_ref_handler(self, context, base, sysid, pubid):
raise ExternalReferenceForbidden(context, base, sysid, pubid)
def install(self, parser):
_ExpatBuilder.install(self, parser)
if self.forbid_dtd:
parser.StartDoctypeDeclHandler = self.defused_start_doctype_decl
if self.forbid_entities:
# if self._options.entities:
parser.EntityDeclHandler = self.defused_entity_decl
parser.UnparsedEntityDeclHandler = self.defused_unparsed_entity_decl
if self.forbid_external:
parser.ExternalEntityRefHandler = self.defused_external_entity_ref_handler
class DefusedExpatBuilderNS(_Namespaces, DefusedExpatBuilder):
"""Defused document builder that supports namespaces."""
def install(self, parser):
DefusedExpatBuilder.install(self, parser)
if self._options.namespace_declarations:
parser.StartNamespaceDeclHandler = self.start_namespace_decl_handler
def reset(self):
DefusedExpatBuilder.reset(self)
self._initNamespaces()
def parse(file, namespaces=True, forbid_dtd=False, forbid_entities=True, forbid_external=True):
"""Parse a document, returning the resulting Document node.
'file' may be either a file name or an open file object.
"""
if namespaces:
build_builder = DefusedExpatBuilderNS
else:
build_builder = DefusedExpatBuilder
builder = build_builder(
forbid_dtd=forbid_dtd, forbid_entities=forbid_entities, forbid_external=forbid_external
)
if isinstance(file, str):
fp = open(file, "rb")
try:
result = builder.parseFile(fp)
finally:
fp.close()
else:
result = builder.parseFile(file)
return result
def parseString(
string, namespaces=True, forbid_dtd=False, forbid_entities=True, forbid_external=True
):
"""Parse a document from a string, returning the resulting
Document node.
"""
if namespaces:
build_builder = DefusedExpatBuilderNS
else:
build_builder = DefusedExpatBuilder
builder = build_builder(
forbid_dtd=forbid_dtd, forbid_entities=forbid_entities, forbid_external=forbid_external
)
return builder.parseString(string)
@@ -0,0 +1,61 @@
# defusedxml
#
# Copyright (c) 2013 by Christian Heimes <[email protected]>
# Licensed to PSF under a Contributor Agreement.
# See https://www.python.org/psf/license for licensing details.
"""Defused xml.sax.expatreader
"""
from __future__ import print_function, absolute_import
from xml.sax.expatreader import ExpatParser as _ExpatParser
from .common import DTDForbidden, EntitiesForbidden, ExternalReferenceForbidden
__origin__ = "xml.sax.expatreader"
class DefusedExpatParser(_ExpatParser):
"""Defused SAX driver for the pyexpat C module."""
def __init__(
self,
namespaceHandling=0,
bufsize=2 ** 16 - 20,
forbid_dtd=False,
forbid_entities=True,
forbid_external=True,
):
_ExpatParser.__init__(self, namespaceHandling, bufsize)
self.forbid_dtd = forbid_dtd
self.forbid_entities = forbid_entities
self.forbid_external = forbid_external
def defused_start_doctype_decl(self, name, sysid, pubid, has_internal_subset):
raise DTDForbidden(name, sysid, pubid)
def defused_entity_decl(
self, name, is_parameter_entity, value, base, sysid, pubid, notation_name
):
raise EntitiesForbidden(name, value, base, sysid, pubid, notation_name)
def defused_unparsed_entity_decl(self, name, base, sysid, pubid, notation_name):
# expat 1.2
raise EntitiesForbidden(name, None, base, sysid, pubid, notation_name) # pragma: no cover
def defused_external_entity_ref_handler(self, context, base, sysid, pubid):
raise ExternalReferenceForbidden(context, base, sysid, pubid)
def reset(self):
_ExpatParser.reset(self)
parser = self._parser
if self.forbid_dtd:
parser.StartDoctypeDeclHandler = self.defused_start_doctype_decl
if self.forbid_entities:
parser.EntityDeclHandler = self.defused_entity_decl
parser.UnparsedEntityDeclHandler = self.defused_unparsed_entity_decl
if self.forbid_external:
parser.ExternalEntityRefHandler = self.defused_external_entity_ref_handler
def create_parser(*args, **kwargs):
return DefusedExpatParser(*args, **kwargs)
@@ -0,0 +1,153 @@
# defusedxml
#
# Copyright (c) 2013 by Christian Heimes <[email protected]>
# Licensed to PSF under a Contributor Agreement.
# See https://www.python.org/psf/license for licensing details.
"""DEPRECATED Example code for lxml.etree protection
The code has NO protection against decompression bombs.
"""
from __future__ import print_function, absolute_import
import threading
import warnings
from lxml import etree as _etree
from .common import DTDForbidden, EntitiesForbidden, NotSupportedError
LXML3 = _etree.LXML_VERSION[0] >= 3
__origin__ = "lxml.etree"
tostring = _etree.tostring
warnings.warn(
"defusedxml.lxml is no longer supported and will be removed in a future release.",
category=DeprecationWarning,
stacklevel=2,
)
class RestrictedElement(_etree.ElementBase):
"""A restricted Element class that filters out instances of some classes"""
__slots__ = ()
# blacklist = (etree._Entity, etree._ProcessingInstruction, etree._Comment)
blacklist = _etree._Entity
def _filter(self, iterator):
blacklist = self.blacklist
for child in iterator:
if isinstance(child, blacklist):
continue
yield child
def __iter__(self):
iterator = super(RestrictedElement, self).__iter__()
return self._filter(iterator)
def iterchildren(self, tag=None, reversed=False):
iterator = super(RestrictedElement, self).iterchildren(tag=tag, reversed=reversed)
return self._filter(iterator)
def iter(self, tag=None, *tags):
iterator = super(RestrictedElement, self).iter(tag=tag, *tags)
return self._filter(iterator)
def iterdescendants(self, tag=None, *tags):
iterator = super(RestrictedElement, self).iterdescendants(tag=tag, *tags)
return self._filter(iterator)
def itersiblings(self, tag=None, preceding=False):
iterator = super(RestrictedElement, self).itersiblings(tag=tag, preceding=preceding)
return self._filter(iterator)
def getchildren(self):
iterator = super(RestrictedElement, self).__iter__()
return list(self._filter(iterator))
def getiterator(self, tag=None):
iterator = super(RestrictedElement, self).getiterator(tag)
return self._filter(iterator)
class GlobalParserTLS(threading.local):
"""Thread local context for custom parser instances"""
parser_config = {
"resolve_entities": False,
# 'remove_comments': True,
# 'remove_pis': True,
}
element_class = RestrictedElement
def createDefaultParser(self):
parser = _etree.XMLParser(**self.parser_config)
element_class = self.element_class
if self.element_class is not None:
lookup = _etree.ElementDefaultClassLookup(element=element_class)
parser.set_element_class_lookup(lookup)
return parser
def setDefaultParser(self, parser):
self._default_parser = parser
def getDefaultParser(self):
parser = getattr(self, "_default_parser", None)
if parser is None:
parser = self.createDefaultParser()
self.setDefaultParser(parser)
return parser
_parser_tls = GlobalParserTLS()
getDefaultParser = _parser_tls.getDefaultParser
def check_docinfo(elementtree, forbid_dtd=False, forbid_entities=True):
"""Check docinfo of an element tree for DTD and entity declarations
The check for entity declarations needs lxml 3 or newer. lxml 2.x does
not support dtd.iterentities().
"""
docinfo = elementtree.docinfo
if docinfo.doctype:
if forbid_dtd:
raise DTDForbidden(docinfo.doctype, docinfo.system_url, docinfo.public_id)
if forbid_entities and not LXML3:
# lxml < 3 has no iterentities()
raise NotSupportedError("Unable to check for entity declarations " "in lxml 2.x")
if forbid_entities:
for dtd in docinfo.internalDTD, docinfo.externalDTD:
if dtd is None:
continue
for entity in dtd.iterentities():
raise EntitiesForbidden(entity.name, entity.content, None, None, None, None)
def parse(source, parser=None, base_url=None, forbid_dtd=False, forbid_entities=True):
if parser is None:
parser = getDefaultParser()
elementtree = _etree.parse(source, parser, base_url=base_url)
check_docinfo(elementtree, forbid_dtd, forbid_entities)
return elementtree
def fromstring(text, parser=None, base_url=None, forbid_dtd=False, forbid_entities=True):
if parser is None:
parser = getDefaultParser()
rootelement = _etree.fromstring(text, parser, base_url=base_url)
elementtree = rootelement.getroottree()
check_docinfo(elementtree, forbid_dtd, forbid_entities)
return rootelement
XML = fromstring
def iterparse(*args, **kwargs):
raise NotSupportedError("defused lxml.etree.iterparse not available")
@@ -0,0 +1,63 @@
# defusedxml
#
# Copyright (c) 2013 by Christian Heimes <[email protected]>
# Licensed to PSF under a Contributor Agreement.
# See https://www.python.org/psf/license for licensing details.
"""Defused xml.dom.minidom
"""
from __future__ import print_function, absolute_import
from xml.dom.minidom import _do_pulldom_parse
from . import expatbuilder as _expatbuilder
from . import pulldom as _pulldom
__origin__ = "xml.dom.minidom"
def parse(
file, parser=None, bufsize=None, forbid_dtd=False, forbid_entities=True, forbid_external=True
):
"""Parse a file into a DOM by filename or file object."""
if parser is None and not bufsize:
return _expatbuilder.parse(
file,
forbid_dtd=forbid_dtd,
forbid_entities=forbid_entities,
forbid_external=forbid_external,
)
else:
return _do_pulldom_parse(
_pulldom.parse,
(file,),
{
"parser": parser,
"bufsize": bufsize,
"forbid_dtd": forbid_dtd,
"forbid_entities": forbid_entities,
"forbid_external": forbid_external,
},
)
def parseString(
string, parser=None, forbid_dtd=False, forbid_entities=True, forbid_external=True
):
"""Parse a file into a DOM from a string."""
if parser is None:
return _expatbuilder.parseString(
string,
forbid_dtd=forbid_dtd,
forbid_entities=forbid_entities,
forbid_external=forbid_external,
)
else:
return _do_pulldom_parse(
_pulldom.parseString,
(string,),
{
"parser": parser,
"forbid_dtd": forbid_dtd,
"forbid_entities": forbid_entities,
"forbid_external": forbid_external,
},
)
@@ -0,0 +1,41 @@
# defusedxml
#
# Copyright (c) 2013 by Christian Heimes <[email protected]>
# Licensed to PSF under a Contributor Agreement.
# See https://www.python.org/psf/license for licensing details.
"""Defused xml.dom.pulldom
"""
from __future__ import print_function, absolute_import
from xml.dom.pulldom import parse as _parse
from xml.dom.pulldom import parseString as _parseString
from .sax import make_parser
__origin__ = "xml.dom.pulldom"
def parse(
stream_or_string,
parser=None,
bufsize=None,
forbid_dtd=False,
forbid_entities=True,
forbid_external=True,
):
if parser is None:
parser = make_parser()
parser.forbid_dtd = forbid_dtd
parser.forbid_entities = forbid_entities
parser.forbid_external = forbid_external
return _parse(stream_or_string, parser, bufsize)
def parseString(
string, parser=None, forbid_dtd=False, forbid_entities=True, forbid_external=True
):
if parser is None:
parser = make_parser()
parser.forbid_dtd = forbid_dtd
parser.forbid_entities = forbid_entities
parser.forbid_external = forbid_external
return _parseString(string, parser)
@@ -0,0 +1,60 @@
# defusedxml
#
# Copyright (c) 2013 by Christian Heimes <[email protected]>
# Licensed to PSF under a Contributor Agreement.
# See https://www.python.org/psf/license for licensing details.
"""Defused xml.sax
"""
from __future__ import print_function, absolute_import
from xml.sax import InputSource as _InputSource
from xml.sax import ErrorHandler as _ErrorHandler
from . import expatreader
__origin__ = "xml.sax"
def parse(
source,
handler,
errorHandler=_ErrorHandler(),
forbid_dtd=False,
forbid_entities=True,
forbid_external=True,
):
parser = make_parser()
parser.setContentHandler(handler)
parser.setErrorHandler(errorHandler)
parser.forbid_dtd = forbid_dtd
parser.forbid_entities = forbid_entities
parser.forbid_external = forbid_external
parser.parse(source)
def parseString(
string,
handler,
errorHandler=_ErrorHandler(),
forbid_dtd=False,
forbid_entities=True,
forbid_external=True,
):
from io import BytesIO
if errorHandler is None:
errorHandler = _ErrorHandler()
parser = make_parser()
parser.setContentHandler(handler)
parser.setErrorHandler(errorHandler)
parser.forbid_dtd = forbid_dtd
parser.forbid_entities = forbid_entities
parser.forbid_external = forbid_external
inpsrc = _InputSource()
inpsrc.setByteStream(BytesIO(string))
parser.parse(inpsrc)
def make_parser(parser_list=[]):
return expatreader.create_parser()
@@ -0,0 +1,153 @@
# defusedxml
#
# Copyright (c) 2013 by Christian Heimes <[email protected]>
# Licensed to PSF under a Contributor Agreement.
# See https://www.python.org/psf/license for licensing details.
"""Defused xmlrpclib
Also defuses gzip bomb
"""
from __future__ import print_function, absolute_import
import io
from .common import DTDForbidden, EntitiesForbidden, ExternalReferenceForbidden, PY3
if PY3:
__origin__ = "xmlrpc.client"
from xmlrpc.client import ExpatParser
from xmlrpc import client as xmlrpc_client
from xmlrpc import server as xmlrpc_server
from xmlrpc.client import gzip_decode as _orig_gzip_decode
from xmlrpc.client import GzipDecodedResponse as _OrigGzipDecodedResponse
else:
__origin__ = "xmlrpclib"
from xmlrpclib import ExpatParser
import xmlrpclib as xmlrpc_client
xmlrpc_server = None
from xmlrpclib import gzip_decode as _orig_gzip_decode
from xmlrpclib import GzipDecodedResponse as _OrigGzipDecodedResponse
try:
import gzip
except ImportError: # pragma: no cover
gzip = None
# Limit maximum request size to prevent resource exhaustion DoS
# Also used to limit maximum amount of gzip decoded data in order to prevent
# decompression bombs
# A value of -1 or smaller disables the limit
MAX_DATA = 30 * 1024 * 1024 # 30 MB
def defused_gzip_decode(data, limit=None):
"""gzip encoded data -> unencoded data
Decode data using the gzip content encoding as described in RFC 1952
"""
if not gzip: # pragma: no cover
raise NotImplementedError
if limit is None:
limit = MAX_DATA
f = io.BytesIO(data)
gzf = gzip.GzipFile(mode="rb", fileobj=f)
try:
if limit < 0: # no limit
decoded = gzf.read()
else:
decoded = gzf.read(limit + 1)
except IOError: # pragma: no cover
raise ValueError("invalid data")
f.close()
gzf.close()
if limit >= 0 and len(decoded) > limit:
raise ValueError("max gzipped payload length exceeded")
return decoded
class DefusedGzipDecodedResponse(gzip.GzipFile if gzip else object):
"""a file-like object to decode a response encoded with the gzip
method, as described in RFC 1952.
"""
def __init__(self, response, limit=None):
# response doesn't support tell() and read(), required by
# GzipFile
if not gzip: # pragma: no cover
raise NotImplementedError
self.limit = limit = limit if limit is not None else MAX_DATA
if limit < 0: # no limit
data = response.read()
self.readlength = None
else:
data = response.read(limit + 1)
self.readlength = 0
if limit >= 0 and len(data) > limit:
raise ValueError("max payload length exceeded")
self.stringio = io.BytesIO(data)
gzip.GzipFile.__init__(self, mode="rb", fileobj=self.stringio)
def read(self, n):
if self.limit >= 0:
left = self.limit - self.readlength
n = min(n, left + 1)
data = gzip.GzipFile.read(self, n)
self.readlength += len(data)
if self.readlength > self.limit:
raise ValueError("max payload length exceeded")
return data
else:
return gzip.GzipFile.read(self, n)
def close(self):
gzip.GzipFile.close(self)
self.stringio.close()
class DefusedExpatParser(ExpatParser):
def __init__(self, target, forbid_dtd=False, forbid_entities=True, forbid_external=True):
ExpatParser.__init__(self, target)
self.forbid_dtd = forbid_dtd
self.forbid_entities = forbid_entities
self.forbid_external = forbid_external
parser = self._parser
if self.forbid_dtd:
parser.StartDoctypeDeclHandler = self.defused_start_doctype_decl
if self.forbid_entities:
parser.EntityDeclHandler = self.defused_entity_decl
parser.UnparsedEntityDeclHandler = self.defused_unparsed_entity_decl
if self.forbid_external:
parser.ExternalEntityRefHandler = self.defused_external_entity_ref_handler
def defused_start_doctype_decl(self, name, sysid, pubid, has_internal_subset):
raise DTDForbidden(name, sysid, pubid)
def defused_entity_decl(
self, name, is_parameter_entity, value, base, sysid, pubid, notation_name
):
raise EntitiesForbidden(name, value, base, sysid, pubid, notation_name)
def defused_unparsed_entity_decl(self, name, base, sysid, pubid, notation_name):
# expat 1.2
raise EntitiesForbidden(name, None, base, sysid, pubid, notation_name) # pragma: no cover
def defused_external_entity_ref_handler(self, context, base, sysid, pubid):
raise ExternalReferenceForbidden(context, base, sysid, pubid)
def monkey_patch():
xmlrpc_client.FastParser = DefusedExpatParser
xmlrpc_client.GzipDecodedResponse = DefusedGzipDecodedResponse
xmlrpc_client.gzip_decode = defused_gzip_decode
if xmlrpc_server:
xmlrpc_server.gzip_decode = defused_gzip_decode
def unmonkey_patch():
xmlrpc_client.FastParser = None
xmlrpc_client.GzipDecodedResponse = _OrigGzipDecodedResponse
xmlrpc_client.gzip_decode = _orig_gzip_decode
if xmlrpc_server:
xmlrpc_server.gzip_decode = _orig_gzip_decode
@@ -0,0 +1,47 @@
from .core import (
IDNABidiError,
IDNAError,
InvalidCodepoint,
InvalidCodepointContext,
alabel,
check_bidi,
check_hyphen_ok,
check_initial_combiner,
check_label,
check_nfc,
decode,
encode,
ulabel,
uts46_remap,
valid_contextj,
valid_contexto,
valid_label_length,
valid_string_length,
)
from .idnadata import __version__ as unicode_version
from .intranges import intranges_contain
from .package_data import __version__
__all__ = [
"__version__",
"unicode_version",
"IDNABidiError",
"IDNAError",
"InvalidCodepoint",
"InvalidCodepointContext",
"alabel",
"check_bidi",
"check_hyphen_ok",
"check_initial_combiner",
"check_label",
"check_nfc",
"decode",
"encode",
"intranges_contain",
"ulabel",
"uts46_remap",
"valid_contextj",
"valid_contexto",
"valid_label_length",
"valid_string_length",
]
@@ -0,0 +1,6 @@
import sys
from .cli import main
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,132 @@
"""Command-line interface for the :mod:`idna` package.
Invoked via ``python -m idna``. See :func:`main` for the entry point.
"""
from __future__ import annotations
import argparse
import sys
from itertools import chain
from typing import IO, TYPE_CHECKING
from . import IDNAError, decode, encode, unicode_version
from .core import _alabel_prefix, _unicode_dots_re
from .package_data import __version__
if TYPE_CHECKING:
from collections.abc import Iterable
def _looks_like_alabel(s: str) -> bool:
"""Return True if any label in ``s`` carries the ``xn--`` ACE prefix."""
prefix = _alabel_prefix.decode("ascii")
return any(label.lower().startswith(prefix) for label in _unicode_dots_re.split(s))
def _build_parser() -> argparse.ArgumentParser:
parser = argparse.ArgumentParser(
prog="python -m idna",
description=(
"Convert a domain name between its Unicode (U-label) and "
"ASCII-compatible (A-label) forms. With no mode flag, the "
"direction is chosen from the first input — if it contains "
"an xn-- label the stream is decoded, otherwise it is "
"encoded — and the same mode is applied to every remaining "
"input. UTS #46 mapping is applied by default; pass "
"--strict to disable it. When no domains are given on the "
"command line and stdin is piped, one domain per line is "
"read from stdin."
),
)
mode = parser.add_mutually_exclusive_group()
mode.add_argument(
"-e",
"--encode",
dest="mode",
action="store_const",
const="encode",
help="Encode the input to its ASCII A-label form.",
)
mode.add_argument(
"-d",
"--decode",
dest="mode",
action="store_const",
const="decode",
help="Decode the input from its ASCII A-label form.",
)
parser.add_argument(
"--strict",
action="store_true",
help="Disable the default UTS #46 mapping and apply IDNA 2008 rules verbatim.",
)
parser.add_argument(
"--version",
action="version",
version=f"idna {__version__} (Unicode {unicode_version})",
)
parser.add_argument(
"domain",
nargs="*",
help="One or more domain names to convert. Omit to read from stdin.",
)
return parser
def _iter_stdin(stream: IO[str]) -> Iterable[str]:
"""Yield non-empty stripped lines from ``stream``, ignoring blanks."""
for line in stream:
stripped = line.strip()
if stripped:
yield stripped
def _convert_one(domain: str, mode: str, uts46: bool) -> bool:
"""Convert ``domain`` and write the result; return ``False`` on failure."""
try:
if mode == "decode":
print(decode(domain, uts46=uts46))
else:
print(encode(domain, uts46=uts46).decode("ascii"))
except IDNAError as err:
print(f"idna: {mode} failed for {domain!r}: {err}", file=sys.stderr)
return False
return True
def main(argv: list[str] | None = None) -> int:
"""Entry point for ``python -m idna``.
When more than one domain is supplied (via positional arguments or
piped stdin) and no mode flag is given, the first input determines
the direction and that mode is applied uniformly to the rest.
:param argv: Argument list excluding the program name. Defaults to
:data:`sys.argv` when ``None``.
:returns: ``0`` on success, ``1`` if any conversion fails.
"""
parser = _build_parser()
args = parser.parse_args(argv)
uts46 = not args.strict
if args.domain:
domains: Iterable[str] = args.domain
elif not sys.stdin.isatty():
domains = _iter_stdin(sys.stdin)
else:
parser.error("a domain argument is required when stdin is a terminal")
iterator = iter(domains)
first = next(iterator, None)
if first is None:
return 0
mode = args.mode or ("decode" if _looks_like_alabel(first) else "encode")
results = [_convert_one(domain, mode, uts46) for domain in chain([first], iterator)]
return 0 if all(results) else 1
if __name__ == "__main__":
sys.exit(main())
@@ -0,0 +1,163 @@
from __future__ import annotations
import codecs
from typing import Any
from .core import IDNAError, _unicode_dots_re, alabel, decode, encode, ulabel
class Codec(codecs.Codec):
"""Stateless IDNA 2008 codec.
Implements the :class:`codecs.Codec` protocol so that the whole-domain
encoder (:func:`idna.encode`) and decoder (:func:`idna.decode`) are
accessible through the standard codec machinery as ``"idna2008"``.
Only the ``"strict"`` error handler is supported; any other handler
raises :exc:`~idna.IDNAError`.
"""
def encode(self, data: str, errors: str = "strict") -> tuple[bytes, int]: # ty: ignore[invalid-method-override]
if errors != "strict":
raise IDNAError(f'Unsupported error handling "{errors}"', code="unsupported_errors")
if not data:
return b"", 0
return encode(data), len(data)
def decode(self, data: bytes, errors: str = "strict") -> tuple[str, int]: # ty: ignore[invalid-method-override]
if errors != "strict":
raise IDNAError(f'Unsupported error handling "{errors}"', code="unsupported_errors")
if not data:
return "", 0
return decode(data), len(data)
class IncrementalEncoder(codecs.BufferedIncrementalEncoder):
"""Incremental IDNA 2008 encoder.
Buffers a partial trailing label across calls until either the next
label separator is seen or ``final=True``, so that streamed input is
encoded one whole label at a time. Any of the four Unicode label
separators (``U+002E``, ``U+3002``, ``U+FF0E``, ``U+FF61``) ends a
label; the result always uses ``U+002E`` as the separator.
Only the ``"strict"`` error handler is supported.
"""
def _buffer_encode(self, data: str, errors: str, final: bool) -> tuple[bytes, int]: # ty: ignore[invalid-method-override]
if errors != "strict":
raise IDNAError(f'Unsupported error handling "{errors}"', code="unsupported_errors")
if not data:
return b"", 0
labels = _unicode_dots_re.split(data)
trailing_dot = b""
if labels:
if not labels[-1]:
trailing_dot = b"."
del labels[-1]
elif not final:
# Keep potentially unfinished label until the next call
del labels[-1]
if labels:
trailing_dot = b"."
result = []
size = 0
for label in labels:
result.append(alabel(label))
if size:
size += 1
size += len(label)
result_bytes = b".".join(result) + trailing_dot
size += len(trailing_dot)
return result_bytes, size
class IncrementalDecoder(codecs.BufferedIncrementalDecoder):
"""Incremental IDNA 2008 decoder.
Buffers a partial trailing label across calls until either the next
label separator is seen or ``final=True``, so that streamed input is
decoded one whole label at a time.
Only the ``"strict"`` error handler is supported.
"""
def _buffer_decode(self, data: Any, errors: str, final: bool) -> tuple[str, int]: # ty: ignore[invalid-method-override]
if errors != "strict":
raise IDNAError(f'Unsupported error handling "{errors}"', code="unsupported_errors")
if not data:
return ("", 0)
if not isinstance(data, str):
try:
data = str(data, "ascii")
except UnicodeDecodeError as err:
raise IDNAError("Invalid ASCII in A-label", code="invalid_ascii") from err
labels = _unicode_dots_re.split(data)
trailing_dot = ""
if labels:
if not labels[-1]:
trailing_dot = "."
del labels[-1]
elif not final:
# Keep potentially unfinished label until the next call
del labels[-1]
if labels:
trailing_dot = "."
result = []
size = 0
for label in labels:
result.append(ulabel(label))
if size:
size += 1
size += len(label)
result_str = ".".join(result) + trailing_dot
size += len(trailing_dot)
return (result_str, size)
class StreamWriter(Codec, codecs.StreamWriter):
pass
class StreamReader(Codec, codecs.StreamReader):
pass
def search_function(name: str) -> codecs.CodecInfo | None:
"""Codec search function registered with :mod:`codecs`.
Returns a :class:`codecs.CodecInfo` for the ``"idna2008"`` codec name
so that ``str.encode("idna2008")`` and ``bytes.decode("idna2008")``
invoke the IDNA 2008 codec defined in this module.
:param name: The codec name being looked up.
:returns: A :class:`codecs.CodecInfo` instance if ``name`` is
``"idna2008"``, otherwise ``None``.
"""
if name != "idna2008":
return None
return codecs.CodecInfo(
name=name,
encode=Codec().encode,
decode=Codec().decode, # type: ignore
incrementalencoder=IncrementalEncoder,
incrementaldecoder=IncrementalDecoder,
streamwriter=StreamWriter,
streamreader=StreamReader,
)
codecs.register(search_function)
@@ -0,0 +1,43 @@
from __future__ import annotations
from typing import Any
from .core import decode, encode
def ToASCII(label: str) -> bytes:
"""Compatibility shim for :rfc:`3490` ``ToASCII``.
Delegates to :func:`idna.encode` (IDNA 2008). Provided to ease porting
of code written against the legacy :mod:`encodings.idna` API; new code
should call :func:`idna.encode` directly.
:param label: The label or domain to encode.
:returns: The encoded form as ASCII :class:`bytes`.
"""
return encode(label)
def ToUnicode(label: bytes | bytearray) -> str:
"""Compatibility shim for :rfc:`3490` ``ToUnicode``.
Delegates to :func:`idna.decode` (IDNA 2008). Provided to ease porting
of code written against the legacy :mod:`encodings.idna` API; new code
should call :func:`idna.decode` directly.
:param label: The label or domain to decode.
:returns: The decoded Unicode form.
"""
return decode(label)
def nameprep(s: Any) -> None:
"""Stub for :rfc:`3491` Nameprep, which is not used by IDNA 2008.
IDNA 2008 (:rfc:`5891`) replaces Nameprep with the per-codepoint
validity classes from :rfc:`5892`; this function exists only to
return a clear error if legacy code attempts to call it.
:raises NotImplementedError: Always.
"""
raise NotImplementedError("IDNA 2008 does not utilise nameprep protocol")
@@ -0,0 +1,862 @@
from __future__ import annotations
import bisect
import re
import unicodedata
import warnings
from typing import Literal
from . import idnadata
from .intranges import intranges_contain
_virama_combining_class = 9
_alabel_prefix = b"xn--"
_max_input_length = 1024
_STATUS_VALID, _STATUS_MAPPED, _STATUS_DEVIATION, _STATUS_IGNORED = b"VMDI"
_unicode_dots_re = re.compile("[\u002e\u3002\uff0e\uff61]")
_std3_disallowed_re = re.compile("[\x00-\x2c\x2f\x3a-\x40A-Z\x5b-\x60\x7b-\x7f]")
_bidi_rtl_first = frozenset({"R", "AL"})
_bidi_rtl_categories = frozenset({"R", "AL", "AN"})
_bidi_rtl_allowed = frozenset({"R", "AL", "AN", "EN", "ES", "CS", "ET", "ON", "BN", "NSM"})
_bidi_rtl_valid_ending = frozenset({"R", "AL", "EN", "AN"})
_bidi_rtl_numeric = frozenset({"AN", "EN"})
_bidi_ltr_allowed = frozenset({"L", "EN", "ES", "CS", "ET", "ON", "BN", "NSM"})
_bidi_ltr_valid_ending = frozenset({"L", "EN"})
_bidi_joiner_l_or_d = frozenset({"L", "D"})
_bidi_joiner_r_or_d = frozenset({"R", "D"})
def _joining_type(cp: int) -> str | None:
for jt, ranges in idnadata.joining_types.items():
if intranges_contain(cp, ranges):
return jt
return None
# Machine-readable identifiers for the rule an :class:`IDNAError` reports.
# These strings are stable and documented; exception message wording is not.
_ErrorCode = Literal[
"input_too_long",
"label_too_long",
"domain_too_long",
"empty_label",
"empty_domain",
"not_nfc",
"hyphen_3_4",
"hyphen_start_end",
"leading_combiner",
"disallowed_codepoint",
"contextj",
"contexto",
"unknown_codepoint",
"bidi_rule_1",
"bidi_rule_2",
"bidi_rule_3",
"bidi_rule_4",
"bidi_rule_5",
"bidi_rule_6",
"bidi_unknown_direction",
"invalid_alabel",
"non_canonical_alabel",
"invalid_ascii",
"invalid_utf8",
"uts46_disallowed",
"uts46_std3",
"unsupported_errors",
]
class IDNAError(UnicodeError):
"""Base exception for all IDNA-encoding related problems.
``str(err)`` is a human-readable description of the failure. The
exception also carries machine-readable attributes so callers do not
need to parse the message:
* ``code`` -- a short, stable identifier for the rule that failed, such
as ``"disallowed_codepoint"`` or ``"bidi_rule_2"``; the full list is
documented in the README. Message wording, by contrast, may change
between releases.
* ``text`` -- the label (or, for UTS #46 processing, the domain) that
was being validated;
* ``codepoint`` -- the offending codepoint, as an ``int``;
* ``position`` -- the 1-based index of the offending character within
``text``, matching the position quoted in the message.
Each is ``None`` when it does not apply.
"""
code: str | None
text: str | None
codepoint: int | None
position: int | None
def __init__(
self,
*args: object,
code: _ErrorCode | None = None,
text: str | None = None,
codepoint: int | None = None,
position: int | None = None,
) -> None:
super().__init__(*args)
self.code = code
self.text = text
self.codepoint = codepoint
self.position = position
class IDNABidiError(IDNAError):
"""Exception when bidirectional requirements are not satisfied"""
class InvalidCodepoint(IDNAError):
"""Exception when a disallowed or unallocated codepoint is used"""
class InvalidCodepointContext(IDNAError):
"""Exception when the codepoint is not valid in the context it is used"""
def _combining_class(cp: int) -> int:
v = unicodedata.combining(chr(cp))
if v == 0 and not unicodedata.name(chr(cp)):
raise ValueError("Unknown character in unicodedata")
return v
def _is_script(cp: str, script: str) -> bool:
return intranges_contain(ord(cp), idnadata.scripts[script])
def _punycode(s: str) -> bytes:
return s.encode("punycode")
def _unot(s: int) -> str:
return f"U+{s:04X}"
def valid_label_length(label: bytes | str) -> bool:
"""Check that a label does not exceed the maximum permitted length.
Per :rfc:`1035` (and :rfc:`5891` §4.2.4) a DNS label must not exceed
63 octets. The argument may be either a :class:`str` (a U-label, where
length is measured in characters) or :class:`bytes` (an A-label, where
length is measured in octets).
:param label: The label to check.
:returns: ``True`` if the label is within the length limit, otherwise
``False``.
"""
return len(label) <= 63
def valid_string_length(domain: bytes | str, trailing_dot: bool) -> bool:
"""Check that a full domain name does not exceed the maximum length.
Per :rfc:`1035`, a domain name is limited to 253 octets when no trailing
dot is present, or 254 octets when one is included.
:param domain: The full (possibly multi-label) domain name.
:param trailing_dot: ``True`` if ``domain`` includes a trailing ``.``.
:returns: ``True`` if the domain is within the length limit, otherwise
``False``.
"""
return len(domain) <= (254 if trailing_dot else 253)
def check_bidi(label: str, check_ltr: bool = False) -> bool:
"""Validate the Bidi Rule from :rfc:`5893` for a single label.
The Bidi Rule constrains how bidirectional characters (Hebrew, Arabic,
etc.) may appear within a label. By default the check is only applied
when the label contains at least one right-to-left character (Unicode
bidirectional categories ``R``, ``AL``, or ``AN``); set ``check_ltr``
to ``True`` to apply it to LTR-only labels as well.
:param label: The label to validate, as a Unicode string.
:param check_ltr: If ``True``, apply the rules even when the label
contains no RTL characters.
:returns: ``True`` if the label satisfies the Bidi Rule.
:raises IDNABidiError: If any of Bidi Rule conditions 1-6 are violated,
or if the directional category of a codepoint cannot be determined.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long", code="input_too_long")
# Bidi rules should only be applied if string contains RTL characters
bidi_label = False
for idx, cp in enumerate(label, 1):
direction = unicodedata.bidirectional(cp)
if direction == "":
# String likely comes from a newer version of Unicode
raise IDNABidiError(
f"Unknown directionality in label {label!r} at position {idx}",
code="bidi_unknown_direction",
text=label,
codepoint=ord(cp),
position=idx,
)
if direction in _bidi_rtl_categories:
bidi_label = True
if not bidi_label and not check_ltr:
return True
# Bidi rule 1
direction = unicodedata.bidirectional(label[0])
if direction in _bidi_rtl_first:
rtl = True
elif direction == "L":
rtl = False
else:
raise IDNABidiError(
f"First codepoint in label {label!r} must be directionality L, R or AL",
code="bidi_rule_1",
text=label,
codepoint=ord(label[0]),
position=1,
)
valid_ending = False
ending_idx = 1
number_type: str | None = None
for idx, cp in enumerate(label, 1):
direction = unicodedata.bidirectional(cp)
if rtl:
# Bidi rule 2
if direction not in _bidi_rtl_allowed:
raise IDNABidiError(
f"Invalid direction for codepoint at position {idx} in a right-to-left label",
code="bidi_rule_2",
text=label,
codepoint=ord(cp),
position=idx,
)
# Bidi rule 3
if direction in _bidi_rtl_valid_ending:
valid_ending = True
ending_idx = idx
elif direction != "NSM":
valid_ending = False
ending_idx = idx
# Bidi rule 4
if direction in _bidi_rtl_numeric:
if not number_type:
number_type = direction
elif number_type != direction:
raise IDNABidiError(
"Can not mix numeral types in a right-to-left label",
code="bidi_rule_4",
text=label,
codepoint=ord(cp),
position=idx,
)
else:
# Bidi rule 5
if direction not in _bidi_ltr_allowed:
raise IDNABidiError(
f"Invalid direction for codepoint at position {idx} in a left-to-right label",
code="bidi_rule_5",
text=label,
codepoint=ord(cp),
position=idx,
)
# Bidi rule 6
if direction in _bidi_ltr_valid_ending:
valid_ending = True
ending_idx = idx
elif direction != "NSM":
valid_ending = False
ending_idx = idx
if not valid_ending:
# Rules 3 and 6 concern the last character that is not a
# non-spacing mark, which is what ``ending_idx`` tracks.
raise IDNABidiError(
"Label ends with illegal codepoint directionality",
code="bidi_rule_3" if rtl else "bidi_rule_6",
text=label,
codepoint=ord(label[ending_idx - 1]),
position=ending_idx,
)
return True
def check_initial_combiner(label: str) -> bool:
"""Reject labels that begin with a combining mark.
Per :rfc:`5891` §4.2.3.2 a label must not start with a character of
Unicode general category ``M`` (Mark).
:param label: The label to check.
:returns: ``True`` if the first character is not a combining mark.
:raises IDNAError: If the label begins with a combining character.
"""
if label and unicodedata.category(label[0])[0] == "M":
raise IDNAError(
"Label begins with an illegal combining character",
code="leading_combiner",
text=label,
codepoint=ord(label[0]),
position=1,
)
return True
def check_hyphen_ok(label: str) -> bool:
"""Validate the hyphen restrictions for a label.
Per :rfc:`5891` §4.2.3.1 a label must not start or end with a hyphen
(``U+002D``), and must not have hyphens in both the third and fourth
positions (the prefix reserved for A-labels).
:param label: The label to check.
:returns: ``True`` if the hyphen restrictions are satisfied.
:raises IDNAError: If any of the hyphen restrictions are violated.
"""
if label[2:4] == "--":
raise IDNAError("Label has disallowed hyphens in 3rd and 4th position", code="hyphen_3_4")
if label.startswith("-") or label.endswith("-"):
raise IDNAError("Label must not start or end with a hyphen", code="hyphen_start_end")
return True
def check_nfc(label: str) -> None:
"""Require that a label is in Unicode Normalization Form C.
:param label: The label to check.
:raises IDNAError: If ``label`` differs from its NFC normalisation.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long", code="input_too_long")
if unicodedata.normalize("NFC", label) != label:
raise IDNAError("Label must be in Normalization Form C", code="not_nfc")
def valid_contextj(label: str, pos: int) -> bool:
"""Validate the CONTEXTJ rules from :rfc:`5892` Appendix A.
These rules govern the contextual use of the joiner codepoints
``U+200C`` (ZERO WIDTH NON-JOINER, Appendix A.1) and ``U+200D``
(ZERO WIDTH JOINER, Appendix A.2) within a label.
:param label: The label containing the codepoint.
:param pos: Index of the joiner codepoint within ``label``.
:returns: ``True`` if the codepoint at ``pos`` satisfies its CONTEXTJ
rule, ``False`` otherwise (including when the codepoint at
``pos`` is not a recognised joiner).
:raises ValueError: If an adjacent codepoint has no Unicode name when
determining its combining class.
:raises IDNAError: If ``label`` exceeds the defensive input length limit.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long", code="input_too_long")
cp_value = ord(label[pos])
if cp_value == 0x200C:
if pos > 0 and _combining_class(ord(label[pos - 1])) == _virama_combining_class:
return True
ok = False
for i in range(pos - 1, -1, -1):
joining_type = _joining_type(ord(label[i]))
if joining_type == "T":
continue
if joining_type in _bidi_joiner_l_or_d:
ok = True
break
break
if not ok:
return False
ok = False
for i in range(pos + 1, len(label)):
joining_type = _joining_type(ord(label[i]))
if joining_type == "T":
continue
if joining_type in _bidi_joiner_r_or_d:
ok = True
break
break
return ok
if cp_value == 0x200D:
return pos > 0 and _combining_class(ord(label[pos - 1])) == _virama_combining_class
return False
def valid_contexto(label: str, pos: int, exception: bool = False) -> bool:
"""Validate the CONTEXTO rules from :rfc:`5892` Appendix A.
Covers the contextual rules for codepoints such as MIDDLE DOT
(``U+00B7``), Greek lower numeral sign, Hebrew punctuation, Katakana
middle dot, and the Arabic-Indic / Extended Arabic-Indic digit ranges.
:param label: The label containing the codepoint.
:param pos: Index of the codepoint within ``label``.
:param exception: Reserved for forward compatibility; currently unused.
:returns: ``True`` if the codepoint at ``pos`` satisfies its CONTEXTO
rule, ``False`` otherwise (including when the codepoint is not a
recognised CONTEXTO codepoint).
:raises IDNAError: If ``label`` exceeds the defensive input length limit.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long", code="input_too_long")
cp_value = ord(label[pos])
if cp_value == 0x00B7:
return 0 < pos < len(label) - 1 and ord(label[pos - 1]) == 0x006C and ord(label[pos + 1]) == 0x006C
if cp_value == 0x0375:
if pos < len(label) - 1 and len(label) > 1:
return _is_script(label[pos + 1], "Greek")
return False
if cp_value in {0x05F3, 0x05F4}:
if pos > 0:
return _is_script(label[pos - 1], "Hebrew")
return False
if cp_value == 0x30FB:
for cp in label:
if cp == "\u30fb":
continue
if _is_script(cp, "Hiragana") or _is_script(cp, "Katakana") or _is_script(cp, "Han"):
return True
return False
if 0x660 <= cp_value <= 0x669:
return not any(0x6F0 <= ord(cp) <= 0x06F9 for cp in label)
if 0x6F0 <= cp_value <= 0x6F9:
return not any(0x660 <= ord(cp) <= 0x0669 for cp in label)
return False
def check_label(label: str | bytes | bytearray) -> None:
"""Run the full set of IDNA 2008 validity checks on a single label.
Applies, in order: NFC normalisation (:func:`check_nfc`), hyphen
restrictions (:func:`check_hyphen_ok`), the no-leading-combiner rule
(:func:`check_initial_combiner`), per-codepoint validity (PVALID,
CONTEXTJ, CONTEXTO classes from :rfc:`5892`), and the Bidi Rule
(:func:`check_bidi`).
:param label: The label to validate. ``bytes`` or ``bytearray`` input
is decoded as UTF-8 first.
:raises IDNAError: If the label is empty or fails a structural rule.
:raises InvalidCodepoint: If the label contains a DISALLOWED or
UNASSIGNED codepoint.
:raises InvalidCodepointContext: If a CONTEXTJ or CONTEXTO codepoint
is not valid in its context.
:raises IDNABidiError: If the Bidi Rule is violated.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long", code="input_too_long")
if isinstance(label, (bytes, bytearray)):
try:
label = label.decode("utf-8")
except UnicodeDecodeError as err:
raise IDNAError("Invalid UTF-8 in label", code="invalid_utf8") from err
if len(label) == 0:
raise IDNAError("Empty Label", code="empty_label")
# Check against the domain length rather than the label length to
# support some UTS #46 use cases, while still bounding the work done
# by the label contextual rules below.
if not valid_string_length(label, trailing_dot=True):
raise IDNAError("Label too long", code="label_too_long")
check_nfc(label)
check_hyphen_ok(label)
check_initial_combiner(label)
for pos, cp in enumerate(label):
cp_value = ord(cp)
if intranges_contain(cp_value, idnadata.codepoint_classes["PVALID"]):
continue
if intranges_contain(cp_value, idnadata.codepoint_classes["CONTEXTJ"]):
try:
contextj_ok = valid_contextj(label, pos)
except ValueError as err:
raise IDNAError(
f"Unknown codepoint adjacent to joiner {_unot(cp_value)} at position {pos + 1} in {label!r}",
code="unknown_codepoint",
text=label,
codepoint=cp_value,
position=pos + 1,
) from err
if not contextj_ok:
raise InvalidCodepointContext(
f"Joiner {_unot(cp_value)} not allowed at position {pos + 1} in {label!r}",
code="contextj",
text=label,
codepoint=cp_value,
position=pos + 1,
)
elif intranges_contain(cp_value, idnadata.codepoint_classes["CONTEXTO"]):
if not valid_contexto(label, pos):
raise InvalidCodepointContext(
f"Codepoint {_unot(cp_value)} not allowed at position {pos + 1} in {label!r}",
code="contexto",
text=label,
codepoint=cp_value,
position=pos + 1,
)
else:
raise InvalidCodepoint(
f"Codepoint {_unot(cp_value)} at position {pos + 1} of {label!r} not allowed",
code="disallowed_codepoint",
text=label,
codepoint=cp_value,
position=pos + 1,
)
check_bidi(label)
def alabel(label: str) -> bytes:
"""Convert a single U-label into its A-label form.
The result is the ASCII-Compatible Encoding (ACE) form per :rfc:`5891`
§4: the label is validated, Punycode-encoded, and prefixed with
``xn--``. Pure ASCII labels that are already valid IDNA labels are
returned unchanged (as :class:`bytes`).
:param label: The label to convert, as a Unicode string.
:returns: The A-label as ASCII-encoded :class:`bytes`.
:raises IDNAError: If the label is invalid or the resulting A-label
exceeds 63 octets.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long", code="input_too_long")
try:
label_bytes = label.encode("ascii")
except UnicodeEncodeError:
pass
else:
ulabel(label_bytes)
if not valid_label_length(label_bytes):
raise IDNAError("Label too long", code="label_too_long")
return label_bytes
check_label(label)
label_bytes = _alabel_prefix + _punycode(label)
if not valid_label_length(label_bytes):
raise IDNAError("Label too long", code="label_too_long")
return label_bytes
def ulabel(label: str | bytes | bytearray) -> str:
"""Convert a single A-label into its U-label form.
Performs the inverse of :func:`alabel`: an ``xn--``-prefixed label is
Punycode-decoded and validated, and is rejected unless it is the
canonical A-label for the decoded U-label (:rfc:`5891` §5.3). Labels
that are already Unicode (or plain ASCII without the ACE prefix) are
validated and returned as a Unicode string.
:param label: The label to convert. ``bytes`` or ``bytearray`` input
is treated as ASCII.
:returns: The U-label as a Unicode string.
:raises IDNAError: If the label is malformed or fails validation.
"""
if len(label) > _max_input_length:
raise IDNAError("Label too long", code="input_too_long")
if not isinstance(label, (bytes, bytearray)):
try:
label_bytes = label.encode("ascii")
except UnicodeEncodeError:
check_label(label)
return label
else:
label_bytes = bytes(label)
if not label_bytes.isascii():
raise IDNAError("Invalid ASCII in A-label", code="invalid_ascii")
label_bytes = label_bytes.lower()
if label_bytes.startswith(_alabel_prefix):
label_bytes = label_bytes[len(_alabel_prefix) :]
if not label_bytes:
raise IDNAError("Malformed A-label, no Punycode eligible content found", code="invalid_alabel")
if label_bytes.endswith(b"-"):
raise IDNAError("A-label must not end with a hyphen", code="invalid_alabel")
else:
check_label(label_bytes)
return label_bytes.decode("ascii")
try:
label = label_bytes.decode("punycode")
except UnicodeError as err:
raise IDNAError("Invalid A-label", code="invalid_alabel") from err
# RFC 5891 §5.3: the label is rejected unless re-encoding the decoded
# form reproduces the (lowercased) input. This catches "fake A-labels"
# (RFC 5890 §2.3.2.1) such as ``xn---bbk``, a non-canonical Punycode
# spelling of ``xn--bbk`` that would otherwise decode to the same
# U-label and so display identically to a different wire-format name.
if _punycode(label) != label_bytes:
raise IDNAError("A-label is not the canonical Punycode encoding of its U-label", code="non_canonical_alabel")
check_label(label)
return label
def _check_std3(text: str, domain: str, offset: int) -> None:
"""Raise if ``text``, a slice of ``domain`` starting at ``offset`` that
UTS #46 mapping left unchanged, contains an ASCII character disallowed
under ``UseSTD3ASCIIRules``."""
match = _std3_disallowed_re.search(text)
if match:
codepoint = ord(match.group())
position = offset + match.start() + 1
raise InvalidCodepoint(
f"Codepoint {_unot(codepoint)} not allowed at position {position} in {domain!r}",
code="uts46_std3",
text=domain,
codepoint=codepoint,
position=position,
)
def _warn_transitional() -> None:
warnings.warn(
"Transitional processing is deprecated in UTS #46 and has no effect. "
"The transitional argument will be removed in a future version.",
DeprecationWarning,
stacklevel=3,
)
def uts46_remap(domain: str, std3_rules: bool = True, transitional: bool = False) -> str:
"""Apply the UTS #46 character mapping to a domain string.
Implements the mapping table from `UTS #46 §4
<https://www.unicode.org/reports/tr46/>`_: each character is kept,
replaced, or rejected based on its status (``V``, ``M``, ``D``,
``I``, ``X``). The result is returned in Normalisation Form C.
:param domain: The full domain name to remap.
:param std3_rules: If ``True``, apply UTS #46's ``UseSTD3ASCIIRules``:
after mapping, any ASCII character other than a lowercase letter,
digit, hyphen or the label separator ``.`` is rejected, whether it
appeared in the input or was produced by a mapping (e.g. U+FF01
FULLWIDTH EXCLAMATION MARK maps to ``!``). If ``False``, such
characters are passed through.
:param transitional: Deprecated and ignored. UTS #46 deprecated
transitional processing in Unicode 15.1 and deviation (status
``D``) codepoints are now always kept, so this has no effect
beyond emitting a :class:`DeprecationWarning`. It will be removed
in a future version.
:returns: The remapped domain, in Normalisation Form C.
:raises InvalidCodepoint: If the domain contains a disallowed
codepoint under the chosen rules.
:raises IDNAError: If ``domain`` exceeds the defensive input length limit.
"""
if transitional:
_warn_transitional()
if len(domain) > _max_input_length:
raise IDNAError("Domain too long", code="input_too_long")
if domain.isascii():
# The only ASCII mapping in UTS #46 is upper- to lowercase, and
# ASCII is invariant under NFC, so lowercasing is the whole job.
result = domain.lower()
if std3_rules:
_check_std3(result, domain, 0)
return result
from .uts46data import uts46_replacements, uts46_starts, uts46_statuses
# ``start`` marks the run of unchanged input not yet copied; a run is
# only sliced out when a character must be replaced or dropped, so the
# common no-change case makes no copy. STD3 is checked per output piece
# to report a violation at its input position.
output: list[str] = []
start = 0
for pos, char in enumerate(domain):
code_point = ord(char)
i = code_point if code_point < 256 else bisect.bisect_right(uts46_starts, code_point) - 1
status = uts46_statuses[i]
# UTS #46 §4: V valid, D deviation (kept), M mapped, I ignored,
# anything else disallowed.
if status == _STATUS_VALID:
continue
if status == _STATUS_MAPPED:
replacement = uts46_replacements[i]
elif status == _STATUS_DEVIATION:
continue
elif status == _STATUS_IGNORED:
replacement = None
else:
raise InvalidCodepoint(
f"Codepoint {_unot(code_point)} not allowed at position {pos + 1} in {domain!r}",
code="uts46_disallowed",
text=domain,
codepoint=code_point,
position=pos + 1,
)
if start < pos:
run = domain[start:pos]
if std3_rules:
_check_std3(run, domain, start)
output.append(run)
if replacement:
if std3_rules and _std3_disallowed_re.search(replacement):
raise InvalidCodepoint(
f"Codepoint {_unot(code_point)} not allowed at position {pos + 1} in {domain!r}",
code="uts46_std3",
text=domain,
codepoint=code_point,
position=pos + 1,
)
output.append(replacement)
start = pos + 1
if start == 0:
if std3_rules:
_check_std3(domain, domain, 0)
return unicodedata.normalize("NFC", domain)
tail = domain[start:]
if std3_rules:
_check_std3(tail, domain, start)
output.append(tail)
return unicodedata.normalize("NFC", "".join(output))
def encode(
s: str | bytes | bytearray,
strict: bool = False,
uts46: bool = False,
std3_rules: bool = False,
transitional: bool = False,
) -> bytes:
"""Encode a Unicode domain name into its ASCII (A-label) form.
Splits the input on label separators (only ``U+002E`` if ``strict`` is
set; otherwise also IDEOGRAPHIC FULL STOP ``U+3002``, FULLWIDTH FULL
STOP ``U+FF0E``, and HALFWIDTH IDEOGRAPHIC FULL STOP ``U+FF61``),
encodes each label with :func:`alabel`, and rejoins them with ``.``.
Optionally pre-processes the input through :func:`uts46_remap`.
:param s: The domain name to encode.
:param strict: If ``True``, only ``U+002E`` is recognised as a label
separator.
:param uts46: If ``True``, apply UTS #46 mapping before encoding.
:param std3_rules: Forwarded to :func:`uts46_remap` when ``uts46`` is
``True``.
:param transitional: Deprecated and ignored (see :func:`uts46_remap`):
emits a :class:`DeprecationWarning` and will be removed in a
future version.
:returns: The encoded domain as ASCII :class:`bytes`.
:raises IDNAError: If the domain is empty, contains an invalid label,
or exceeds the maximum domain length.
"""
if transitional:
_warn_transitional()
if not isinstance(s, str):
try:
s = str(s, "ascii")
except (UnicodeDecodeError, TypeError) as err:
raise IDNAError(
"should pass a unicode string to the function rather than a byte string.", code="invalid_ascii"
) from err
if len(s) > _max_input_length:
raise IDNAError("Domain too long", code="input_too_long")
if uts46:
s = uts46_remap(s, std3_rules)
if not valid_string_length(s, trailing_dot=True):
raise IDNAError("Domain too long", code="domain_too_long")
trailing_dot = False
result = []
labels = s.split(".") if strict else _unicode_dots_re.split(s)
if not labels or labels == [""]:
raise IDNAError("Empty domain", code="empty_domain")
if labels[-1] == "":
del labels[-1]
trailing_dot = True
for label in labels:
s = alabel(label)
if s:
result.append(s)
else:
raise IDNAError("Empty label", code="empty_label")
if trailing_dot:
result.append(b"")
s = b".".join(result)
if not valid_string_length(s, trailing_dot):
raise IDNAError("Domain too long", code="domain_too_long")
return s
def decode(
s: str | bytes | bytearray,
strict: bool = False,
uts46: bool = False,
std3_rules: bool = False,
display: bool = False,
) -> str:
"""Decode an A-label-encoded domain name back to Unicode.
Splits the input on label separators (see :func:`encode` for the
rules), decodes each label with :func:`ulabel`, and rejoins them
with ``.``. Optionally pre-processes the input through
:func:`uts46_remap`.
:param s: The domain name to decode.
:param strict: If ``True``, only ``U+002E`` is recognised as a label
separator.
:param uts46: If ``True``, apply UTS #46 mapping before decoding.
:param std3_rules: Forwarded to :func:`uts46_remap` when ``uts46`` is
``True``.
:param display: If ``True``, any ``xn--`` label that fails IDNA
validation is passed through unchanged (lowercased) rather than
aborting the whole call. Intended for "decode for display"
consumers (e.g. URL libraries, HTTP clients) that want to show
the user the label as it appears on the wire when it cannot be
rendered as Unicode. Matches the per-label recovery prescribed
by UTS #46 §4 and the WHATWG URL "domain to Unicode" algorithm.
:returns: The decoded domain as a Unicode string.
:raises IDNAError: If the input is not valid ASCII, contains an
invalid label, or is empty.
"""
if not isinstance(s, str):
try:
s = str(s, "ascii")
except (UnicodeDecodeError, TypeError) as err:
raise IDNAError("Invalid ASCII in A-label", code="invalid_ascii") from err
if len(s) > _max_input_length:
raise IDNAError("Domain too long", code="input_too_long")
if uts46:
s = uts46_remap(s, std3_rules, False)
if not valid_string_length(s, trailing_dot=True):
raise IDNAError("Domain too long", code="domain_too_long")
trailing_dot = False
result = []
labels = s.split(".") if strict else _unicode_dots_re.split(s)
if not labels or labels == [""]:
raise IDNAError("Empty domain", code="empty_domain")
if not labels[-1]:
del labels[-1]
trailing_dot = True
for label in labels:
try:
u = ulabel(label)
except IDNAError:
if display and label[:4].lower() == "xn--":
u = label.lower()
else:
raise
if u:
result.append(u)
else:
raise IDNAError("Empty label", code="empty_label")
if trailing_dot:
result.append("")
return ".".join(result)
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,55 @@
"""
Given a list of integers, made up of (hopefully) a small number of long runs
of consecutive integers, compute a representation of the form
((start1, end1), (start2, end2) ...). Then answer the question "was x present
in the original list?" in time O(log(# runs)).
"""
import bisect
def intranges_from_list(list_: list[int]) -> tuple[int, ...]:
"""Represent a list of integers as a sequence of ranges:
((start_0, end_0), (start_1, end_1), ...), such that the original
integers are exactly those x such that start_i <= x < end_i for some i.
Ranges are encoded as single integers (start << 32 | end), not as tuples.
"""
sorted_list = sorted(list_)
ranges = []
last_write = -1
for i in range(len(sorted_list)):
if i + 1 < len(sorted_list) and sorted_list[i] == sorted_list[i + 1] - 1:
continue
current_range = sorted_list[last_write + 1 : i + 1]
ranges.append(_encode_range(current_range[0], current_range[-1] + 1))
last_write = i
return tuple(ranges)
def _encode_range(start: int, end: int) -> int:
return (start << 32) | end
def _decode_range(r: int) -> tuple[int, int]:
return (r >> 32), (r & ((1 << 32) - 1))
def intranges_contain(int_: int, ranges: tuple[int, ...]) -> bool:
"""Determine if `int_` falls into one of the ranges in `ranges`."""
tuple_ = _encode_range(int_, 0)
pos = bisect.bisect_left(ranges, tuple_)
# we could be immediately ahead of a tuple (start, end)
# with start < int_ <= end
if pos > 0:
left, right = _decode_range(ranges[pos - 1])
if left <= int_ < right:
return True
# or we could be immediately behind a tuple (int_, end)
if pos < len(ranges):
left, _ = _decode_range(ranges[pos])
if left == int_:
return True
return False
@@ -0,0 +1 @@
__version__ = "3.19"
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,219 @@
# __
# /__) _ _ _ _ _/ _
# / ( (- (/ (/ (- _) / _)
# /
"""
Requests HTTP Library
~~~~~~~~~~~~~~~~~~~~~
Requests is an HTTP library, written in Python, for human beings.
Basic GET usage:
>>> import requests
>>> r = requests.get('https://www.python.org')
>>> r.status_code
200
>>> b'Python is a programming language' in r.content
True
... or POST:
>>> payload = dict(key1='value1', key2='value2')
>>> r = requests.post('https://httpbin.org/post', data=payload)
>>> print(r.text)
{
...
"form": {
"key1": "value1",
"key2": "value2"
},
...
}
The other HTTP methods are supported - see `requests.api`. Full documentation
is at <https://requests.readthedocs.io>.
:copyright: (c) 2017 by Kenneth Reitz.
:license: Apache 2.0, see LICENSE for more details.
"""
from __future__ import annotations
import warnings
import urllib3
from .exceptions import RequestsDependencyWarning
try:
from charset_normalizer import __version__ as charset_normalizer_version
except ImportError:
charset_normalizer_version = None
try:
from chardet import __version__ as chardet_version # type: ignore[import-not-found]
except ImportError:
chardet_version = None
def check_compatibility(
urllib3_version: str,
chardet_version: str | None,
charset_normalizer_version: str | None,
) -> None:
urllib3_version_list = urllib3_version.split(".")[:3]
assert urllib3_version_list != ["dev"] # Verify urllib3 isn't installed from git.
# Sometimes, urllib3 only reports its version as 16.1.
if len(urllib3_version_list) == 2:
urllib3_version_list.append("0")
# Check urllib3 for compatibility.
major, minor, patch = urllib3_version_list # noqa: F811
major, minor, patch = int(major), int(minor), int(patch)
# urllib3 >= 1.21.1
assert major >= 1
if major == 1:
assert minor >= 21
# Check charset_normalizer for compatibility.
if chardet_version:
major, minor, patch = chardet_version.split(".")[:3]
major, minor, patch = int(major), int(minor), int(patch)
# chardet_version >= 3.0.2, < 8.0.0
assert (3, 0, 2) <= (major, minor, patch) < (8, 0, 0)
elif charset_normalizer_version:
major, minor, patch = charset_normalizer_version.split(".")[:3]
major, minor, patch = int(major), int(minor), int(patch)
# charset_normalizer >= 2.0.0 < 4.0.0
assert (2, 0, 0) <= (major, minor, patch) < (4, 0, 0)
else:
warnings.warn(
"Unable to find acceptable character detection dependency "
"(chardet or charset_normalizer).",
RequestsDependencyWarning,
)
def _check_cryptography(cryptography_version: str) -> None:
# cryptography < 1.3.4
try:
cryptography_version_list = list(map(int, cryptography_version.split(".")))
except ValueError:
return
if cryptography_version_list < [1, 3, 4]:
warning = f"Old version of cryptography ({cryptography_version_list}) may cause slowdown."
warnings.warn(warning, RequestsDependencyWarning)
# Check imported dependencies for compatibility.
try:
check_compatibility(
urllib3.__version__, # type: ignore[reportPrivateImportUsage]
chardet_version, # type: ignore[reportUnknownArgumentType]
charset_normalizer_version,
)
except (AssertionError, ValueError):
warnings.warn(
f"urllib3 ({urllib3.__version__}) or chardet " # type: ignore[reportPrivateImportUsage]
f"({chardet_version})/charset_normalizer ({charset_normalizer_version}) "
"doesn't match a supported version!",
RequestsDependencyWarning,
)
# Attempt to enable urllib3's fallback for SNI support
# if the standard library doesn't support SNI or the
# 'ssl' library isn't available.
try:
try:
import ssl
except ImportError:
ssl = None
if not getattr(ssl, "HAS_SNI", False):
from urllib3.contrib import pyopenssl
pyopenssl.inject_into_urllib3()
# Check cryptography version
from cryptography import ( # type: ignore[reportMissingImports]
__version__ as cryptography_version, # type: ignore[reportUnknownVariableType]
)
_check_cryptography(cryptography_version) # type: ignore[reportUnknownArgumentType]
except ImportError:
pass
# urllib3's DependencyWarnings should be silenced.
from urllib3.exceptions import DependencyWarning
warnings.simplefilter("ignore", DependencyWarning)
# Set default logging handler to avoid "No handler found" warnings.
import logging
from logging import NullHandler
from . import packages, utils
from .__version__ import (
__author__,
__author_email__,
__build__,
__cake__,
__copyright__,
__description__,
__license__,
__title__,
__url__,
__version__,
)
from .api import delete, get, head, options, patch, post, put, request
from .exceptions import (
ConnectionError,
ConnectTimeout,
FileModeWarning,
HTTPError,
JSONDecodeError,
ReadTimeout,
RequestException,
Timeout,
TooManyRedirects,
URLRequired,
)
from .models import PreparedRequest, Request, Response
from .sessions import Session, session
from .status_codes import codes
__all__ = (
"ConnectionError",
"ConnectTimeout",
"HTTPError",
"JSONDecodeError",
"PreparedRequest",
"ReadTimeout",
"Request",
"RequestException",
"Response",
"Session",
"Timeout",
"TooManyRedirects",
"URLRequired",
"codes",
"delete",
"get",
"head",
"options",
"packages",
"patch",
"post",
"put",
"request",
"session",
"utils",
)
logging.getLogger(__name__).addHandler(NullHandler())
# FileModeWarnings go off per the default.
warnings.simplefilter("default", FileModeWarning, append=True)
@@ -0,0 +1,14 @@
# .-. .-. .-. . . .-. .-. .-. .-.
# |( |- |.| | | |- `-. | `-.
# ' ' `-' `-`.`-' `-' `-' ' `-'
__title__ = "requests"
__description__ = "Python HTTP for Humans."
__url__ = "https://requests.readthedocs.io"
__version__ = "2.34.2"
__build__ = 0x023402
__author__ = "Kenneth Reitz"
__author_email__ = "[email protected]"
__license__ = "Apache-2.0"
__copyright__ = "Copyright Kenneth Reitz"
__cake__ = "\u2728 \U0001f370 \u2728"
@@ -0,0 +1,51 @@
"""
requests._internal_utils
~~~~~~~~~~~~~~
Provides utility functions that are consumed internally by Requests
which depend on extremely few external helpers (such as compat)
"""
import re
from .compat import builtin_str
_VALID_HEADER_NAME_RE_BYTE = re.compile(rb"^[^:\s][^:\r\n]*\Z")
_VALID_HEADER_NAME_RE_STR = re.compile(r"^[^:\s][^:\r\n]*\Z")
_VALID_HEADER_VALUE_RE_BYTE = re.compile(rb"^\S[^\r\n]*\Z|^\Z")
_VALID_HEADER_VALUE_RE_STR = re.compile(r"^\S[^\r\n]*\Z|^\Z")
_HEADER_VALIDATORS_STR = (_VALID_HEADER_NAME_RE_STR, _VALID_HEADER_VALUE_RE_STR)
_HEADER_VALIDATORS_BYTE = (_VALID_HEADER_NAME_RE_BYTE, _VALID_HEADER_VALUE_RE_BYTE)
HEADER_VALIDATORS = {
bytes: _HEADER_VALIDATORS_BYTE,
str: _HEADER_VALIDATORS_STR,
}
def to_native_string(string: str | bytes, encoding: str = "ascii") -> str:
"""Given a string object, regardless of type, returns a representation of
that string in the native string type, encoding and decoding where
necessary. This assumes ASCII unless told otherwise.
"""
if isinstance(string, builtin_str):
out = string
else:
out = string.decode(encoding)
return out
def unicode_is_ascii(u_string: str) -> bool:
"""Determine if unicode string only contains ASCII characters.
:param str u_string: unicode string to check. Must be unicode
and not Python 2 `str`.
:rtype: bool
"""
assert isinstance(u_string, str)
try:
u_string.encode("ascii")
return True
except UnicodeEncodeError:
return False
@@ -0,0 +1,183 @@
"""
requests._types
~~~~~~~~~~~~~~~
This module contains type aliases used internally by the Requests library.
These types are not part of the public API and must not be relied upon
by external code.
"""
from __future__ import annotations
from collections.abc import Callable, Iterable, Mapping, MutableMapping, Sequence
from typing import (
TYPE_CHECKING,
Any,
Protocol,
TypeAlias,
TypeVar,
runtime_checkable,
)
_T_co = TypeVar("_T_co", covariant=True)
_KT_co = TypeVar("_KT_co", covariant=True)
_VT_co = TypeVar("_VT_co", covariant=True)
@runtime_checkable
class SupportsRead(Protocol[_T_co]):
def read(self, length: int = ..., /) -> _T_co: ...
@runtime_checkable
class SupportsItems(Protocol[_KT_co, _VT_co]):
def items(self) -> Iterable[tuple[_KT_co, _VT_co]]: ...
# These are needed at runtime for default_hooks() return type
HookType: TypeAlias = Callable[["Response"], Any]
HooksInputType: TypeAlias = Mapping[str, Iterable[HookType] | HookType]
def is_prepared(request: PreparedRequest) -> TypeIs[_ValidatedRequest]:
"""Verify a PreparedRequest has been fully prepared."""
if TYPE_CHECKING:
return request.url is not None and request.method is not None
# noop at runtime to avoid AssertionError
return True
if TYPE_CHECKING:
from http.cookiejar import CookieJar
from typing import TypeAlias, TypedDict
from typing_extensions import (
Buffer, # TODO: move to collections.abc when Python >= 3.12
TypeIs, # TODO: move to typing when Python >= 3.13
)
from .auth import AuthBase
from .cookies import RequestsCookieJar
from .models import PreparedRequest, Response
from .structures import CaseInsensitiveDict
class _ValidatedRequest(PreparedRequest):
"""Subtype asserting a PreparedRequest has been fully prepared before calling.
The override suppression is required because mutable attribute types are
invariant (Liskov), but we only narrow after preparation is complete. This
is the explicit contract for Requests but Python's typing doesn't have a
better way to represent the requirement.
"""
url: str # type: ignore[reportIncompatibleVariableOverride]
method: str # type: ignore[reportIncompatibleVariableOverride]
# Type aliases for core API concepts (ordered by request() signature)
UriType: TypeAlias = str | bytes
_ParamsMappingKeyType: TypeAlias = str | bytes | int | float
_ParamsMappingValueType: TypeAlias = (
str | bytes | int | float | Iterable[str | bytes | int | float] | None
)
ParamsType: TypeAlias = (
SupportsItems[_ParamsMappingKeyType, _ParamsMappingValueType]
| tuple[tuple[_ParamsMappingKeyType, _ParamsMappingValueType], ...]
| Iterable[tuple[_ParamsMappingKeyType, _ParamsMappingValueType]]
| str
| bytes
| None
)
KVDataType: TypeAlias = Iterable[tuple[Any, Any]] | SupportsItems[Any, Any]
RawDataType: TypeAlias = KVDataType | str | bytes
StreamDataType: TypeAlias = SupportsRead[str | bytes]
EncodableDataType: TypeAlias = RawDataType | StreamDataType
DataType: TypeAlias = (
KVDataType
| Iterable[bytes | str]
| str
| bytes
| Buffer
| SupportsRead[str | bytes]
| None
)
BodyType: TypeAlias = (
bytes | str | Iterable[bytes | str] | SupportsRead[bytes | str] | None
)
HeadersType: TypeAlias = Mapping[str, str | bytes] | None
CookiesType: TypeAlias = RequestsCookieJar | Mapping[str, str]
# Building blocks for FilesType
_FileName: TypeAlias = str | None
_FileContent: TypeAlias = SupportsRead[str | bytes] | str | bytes
_FileSpecBasic: TypeAlias = tuple[_FileName, _FileContent]
_FileSpecWithContentType: TypeAlias = tuple[_FileName, _FileContent, str]
_FileSpecWithHeaders: TypeAlias = tuple[
_FileName, _FileContent, str, CaseInsensitiveDict[str] | Mapping[str, str]
]
_FileSpec: TypeAlias = (
_FileContent | _FileSpecBasic | _FileSpecWithContentType | _FileSpecWithHeaders
)
FilesType: TypeAlias = (
Mapping[str, _FileSpec] | Iterable[tuple[str, _FileSpec]] | None
)
AuthType: TypeAlias = (
tuple[str, str] | AuthBase | Callable[[PreparedRequest], PreparedRequest] | None
)
TimeoutType: TypeAlias = float | tuple[float | None, float | None] | None
ProxiesType: TypeAlias = MutableMapping[str, str]
HooksType: TypeAlias = dict[str, list[HookType]] | None
VerifyType: TypeAlias = bool | str
CertType: TypeAlias = str | tuple[str, str] | None
JsonType: TypeAlias = (
None
| bool
| int
| float
| str
| Sequence["JsonType"]
| Mapping[str, "JsonType"]
)
# TypedDicts for Unpack kwargs (PEP 692)
class BaseRequestKwargs(TypedDict, total=False):
headers: HeadersType
cookies: RequestsCookieJar | CookieJar | dict[str, str] | None
files: FilesType
auth: AuthType
timeout: TimeoutType
allow_redirects: bool
proxies: dict[str, str] | None
hooks: HooksInputType | None
stream: bool | None
verify: VerifyType | None
cert: CertType
class RequestKwargs(BaseRequestKwargs, total=False):
"""kwargs for request(), options(), head(), delete()."""
params: ParamsType
data: DataType
json: JsonType
class GetKwargs(BaseRequestKwargs, total=False):
data: DataType
json: JsonType
class PostKwargs(BaseRequestKwargs, total=False):
params: ParamsType
class DataKwargs(BaseRequestKwargs, total=False):
"""kwargs for put(), patch()."""
params: ParamsType
json: JsonType
@@ -0,0 +1,748 @@
"""
requests.adapters
~~~~~~~~~~~~~~~~~
This module contains the transport adapters that Requests uses to define
and maintain connections.
"""
from __future__ import annotations
import os.path
import socket # noqa: F401 # type: ignore[reportUnusedImport]
import typing
import warnings
from typing import Any
from urllib3.exceptions import (
ClosedPoolError,
ConnectTimeoutError,
LocationValueError,
MaxRetryError,
NewConnectionError,
ProtocolError,
ReadTimeoutError,
ResponseError,
)
from urllib3.exceptions import HTTPError as _HTTPError
from urllib3.exceptions import InvalidHeader as _InvalidHeader
from urllib3.exceptions import ProxyError as _ProxyError
from urllib3.exceptions import SSLError as _SSLError
from urllib3.poolmanager import PoolManager, proxy_from_url
from urllib3.util import Timeout as TimeoutSauce
from urllib3.util import parse_url
from urllib3.util.retry import Retry
from .auth import _basic_auth_str # type: ignore[reportPrivateUsage]
from .compat import basestring, urlparse
from .cookies import extract_cookies_to_jar
from .exceptions import (
ConnectionError,
ConnectTimeout,
InvalidHeader,
InvalidProxyURL,
InvalidSchema,
InvalidURL,
ProxyError,
ReadTimeout,
RetryError,
SSLError,
)
from .models import Response
from .structures import CaseInsensitiveDict
from .utils import (
DEFAULT_CA_BUNDLE_PATH,
get_auth_from_url,
get_encoding_from_headers,
prepend_scheme_if_needed,
select_proxy,
urldefragauth,
)
try:
from urllib3.contrib.socks import SOCKSProxyManager # type: ignore[assignment]
except ImportError:
def SOCKSProxyManager(*args: Any, **kwargs: Any) -> None:
raise InvalidSchema("Missing dependencies for SOCKS support.")
if typing.TYPE_CHECKING:
from urllib3.connectionpool import HTTPConnectionPool
from urllib3.poolmanager import PoolManager as _PoolManager
from . import _types as _t
from .models import PreparedRequest
from ._types import is_prepared as _is_prepared
DEFAULT_POOLBLOCK = False
DEFAULT_POOLSIZE = 10
DEFAULT_RETRIES = 0
DEFAULT_POOL_TIMEOUT = None
def _urllib3_request_context(
request: PreparedRequest,
verify: bool | str | None,
client_cert: tuple[str, str] | str | None,
poolmanager: PoolManager,
) -> tuple[dict[str, Any], dict[str, Any]]:
host_params: dict[str, Any] = {}
pool_kwargs: dict[str, Any] = {}
parsed_request_url = urlparse(request.url)
scheme = parsed_request_url.scheme.lower()
port = parsed_request_url.port
cert_reqs = "CERT_REQUIRED"
if verify is False:
cert_reqs = "CERT_NONE"
elif isinstance(verify, str):
if not os.path.isdir(verify):
pool_kwargs["ca_certs"] = verify
else:
pool_kwargs["ca_cert_dir"] = verify
pool_kwargs["cert_reqs"] = cert_reqs
if client_cert is not None:
if isinstance(client_cert, tuple) and len(client_cert) == 2:
pool_kwargs["cert_file"] = client_cert[0]
pool_kwargs["key_file"] = client_cert[1]
else:
# According to our docs, we allow users to specify just the client
# cert path
pool_kwargs["cert_file"] = client_cert
host_params = {
"scheme": scheme,
"host": parsed_request_url.hostname,
"port": port,
}
return host_params, pool_kwargs
class BaseAdapter:
"""The Base Transport Adapter"""
def __init__(self) -> None:
super().__init__()
def send(
self,
request: PreparedRequest,
stream: bool = False,
timeout: _t.TimeoutType = None,
verify: _t.VerifyType = True,
cert: _t.CertType = None,
proxies: dict[str, str] | None = None,
) -> Response:
"""Sends PreparedRequest object. Returns Response object.
:param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
:param stream: (optional) Whether to stream the request content.
:param timeout: (optional) How long to wait for the server to send
data before giving up, as a float, or a :ref:`(connect timeout,
read timeout) <timeouts>` tuple.
:type timeout: float or tuple
:param verify: (optional) Either a boolean, in which case it controls whether we verify
the server's TLS certificate, or a string, in which case it must be a path
to a CA bundle to use
:param cert: (optional) Any user-provided SSL certificate to be trusted.
:param proxies: (optional) The proxies dictionary to apply to the request.
"""
raise NotImplementedError
def close(self) -> None:
"""Cleans up adapter specific items."""
raise NotImplementedError
class HTTPAdapter(BaseAdapter):
"""The built-in HTTP Adapter for urllib3.
Provides a general-case interface for Requests sessions to contact HTTP and
HTTPS urls by implementing the Transport Adapter interface. This class will
usually be created by the :class:`Session <Session>` class under the
covers.
:param pool_connections: The number of urllib3 connection pools to cache.
:param pool_maxsize: The maximum number of connections to save in the pool.
:param max_retries: The maximum number of retries each connection
should attempt. Note, this applies only to failed DNS lookups, socket
connections and connection timeouts, never to requests where data has
made it to the server. By default, Requests does not retry failed
connections. If you need granular control over the conditions under
which we retry a request, import urllib3's ``Retry`` class and pass
that instead.
:param pool_block: Whether the connection pool should block for connections.
Usage::
>>> import requests
>>> s = requests.Session()
>>> a = requests.adapters.HTTPAdapter(max_retries=3)
>>> s.mount('http://', a)
"""
__attrs__: list[str] = [
"max_retries",
"config",
"_pool_connections",
"_pool_maxsize",
"_pool_block",
]
max_retries: Retry
config: dict[str, Any]
proxy_manager: dict[str, Any]
_pool_connections: int
_pool_maxsize: int
_pool_block: bool
poolmanager: _PoolManager
def __init__(
self,
pool_connections: int = DEFAULT_POOLSIZE,
pool_maxsize: int = DEFAULT_POOLSIZE,
max_retries: int | Retry = DEFAULT_RETRIES,
pool_block: bool = DEFAULT_POOLBLOCK,
) -> None:
if max_retries == DEFAULT_RETRIES:
self.max_retries = Retry(0, read=False)
else:
self.max_retries = Retry.from_int(max_retries)
self.config = {}
self.proxy_manager = {}
super().__init__()
self._pool_connections = pool_connections
self._pool_maxsize = pool_maxsize
self._pool_block = pool_block
self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block)
def __getstate__(self) -> dict[str, Any]:
return {attr: getattr(self, attr, None) for attr in self.__attrs__}
def __setstate__(self, state: dict[str, Any]) -> None:
# Can't handle by adding 'proxy_manager' to self.__attrs__ because
# self.poolmanager uses a lambda function, which isn't pickleable.
self.proxy_manager = {}
self.config = {}
for attr, value in state.items():
setattr(self, attr, value)
self.init_poolmanager(
self._pool_connections, self._pool_maxsize, block=self._pool_block
)
def init_poolmanager(
self,
connections: int,
maxsize: int,
block: bool = DEFAULT_POOLBLOCK,
**pool_kwargs: Any,
) -> None:
"""Initializes a urllib3 PoolManager.
This method should not be called from user code, and is only
exposed for use when subclassing the
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param connections: The number of urllib3 connection pools to cache.
:param maxsize: The maximum number of connections to save in the pool.
:param block: Block when no free connections are available.
:param pool_kwargs: Extra keyword arguments used to initialize the Pool Manager.
"""
# save these values for pickling
self._pool_connections = connections
self._pool_maxsize = maxsize
self._pool_block = block
self.poolmanager = PoolManager(
num_pools=connections,
maxsize=maxsize,
block=block,
**pool_kwargs,
)
def proxy_manager_for(self, proxy: str, **proxy_kwargs: Any) -> Any:
"""Return urllib3 ProxyManager for the given proxy.
This method should not be called from user code, and is only
exposed for use when subclassing the
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param proxy: The proxy to return a urllib3 ProxyManager for.
:param proxy_kwargs: Extra keyword arguments used to configure the Proxy Manager.
:returns: ProxyManager
:rtype: urllib3.ProxyManager
"""
if proxy in self.proxy_manager:
manager = self.proxy_manager[proxy]
elif proxy.lower().startswith("socks"):
username, password = get_auth_from_url(proxy)
manager = self.proxy_manager[proxy] = SOCKSProxyManager(
proxy,
username=username,
password=password,
num_pools=self._pool_connections,
maxsize=self._pool_maxsize,
block=self._pool_block,
**proxy_kwargs,
)
else:
proxy_headers = self.proxy_headers(proxy)
manager = self.proxy_manager[proxy] = proxy_from_url(
proxy,
proxy_headers=proxy_headers,
num_pools=self._pool_connections,
maxsize=self._pool_maxsize,
block=self._pool_block,
**proxy_kwargs,
)
return manager
def cert_verify(
self, conn: Any, url: str, verify: _t.VerifyType, cert: _t.CertType
) -> None:
"""Verify a SSL certificate. This method should not be called from user
code, and is only exposed for use when subclassing the
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param conn: The urllib3 connection object associated with the cert.
:param url: The requested URL.
:param verify: Either a boolean, in which case it controls whether we verify
the server's TLS certificate, or a string, in which case it must be a path
to a CA bundle to use
:param cert: The SSL certificate to verify.
"""
if url.lower().startswith("https") and verify:
cert_loc = None
# Allow self-specified cert location.
if verify is not True:
cert_loc = verify
if not cert_loc:
cert_loc = DEFAULT_CA_BUNDLE_PATH
if not cert_loc or not os.path.exists(cert_loc):
raise OSError(
f"Could not find a suitable TLS CA certificate bundle, "
f"invalid path: {cert_loc}"
)
conn.cert_reqs = "CERT_REQUIRED"
if not os.path.isdir(cert_loc):
conn.ca_certs = cert_loc
else:
conn.ca_cert_dir = cert_loc
else:
conn.cert_reqs = "CERT_NONE"
conn.ca_certs = None
conn.ca_cert_dir = None
if cert:
if not isinstance(cert, basestring):
conn.cert_file = cert[0]
conn.key_file = cert[1]
else:
conn.cert_file = cert
conn.key_file = None
if conn.cert_file and not os.path.exists(conn.cert_file):
raise OSError(
f"Could not find the TLS certificate file, "
f"invalid path: {conn.cert_file}"
)
if conn.key_file and not os.path.exists(conn.key_file):
raise OSError(
f"Could not find the TLS key file, invalid path: {conn.key_file}"
)
def build_response(self, req: PreparedRequest, resp: Any) -> Response:
"""Builds a :class:`Response <requests.Response>` object from a urllib3
response. This should not be called from user code, and is only exposed
for use when subclassing the
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`
:param req: The :class:`PreparedRequest <PreparedRequest>` used to generate the response.
:param resp: The urllib3 response object.
:rtype: requests.Response
"""
assert _is_prepared(req)
response = Response()
# Fallback to None if there's no status_code, for whatever reason.
response.status_code = getattr(resp, "status", None) # type: ignore[assignment]
# Make headers case-insensitive.
response.headers = CaseInsensitiveDict(getattr(resp, "headers", {}))
# Set encoding.
response.encoding = get_encoding_from_headers(response.headers)
response.raw = resp
response.reason = response.raw.reason
if isinstance(req.url, bytes):
response.url = req.url.decode("utf-8")
else:
response.url = req.url
# Add new cookies from the server.
extract_cookies_to_jar(response.cookies, req, resp)
# Give the Response some context.
response.request = req
response.connection = self
return response
def build_connection_pool_key_attributes(
self, request: PreparedRequest, verify: _t.VerifyType, cert: _t.CertType = None
) -> tuple[dict[str, Any], dict[str, Any]]:
"""Build the PoolKey attributes used by urllib3 to return a connection.
This looks at the PreparedRequest, the user-specified verify value,
and the value of the cert parameter to determine what PoolKey values
to use to select a connection from a given urllib3 Connection Pool.
The SSL related pool key arguments are not consistently set. As of
this writing, use the following to determine what keys may be in that
dictionary:
* If ``verify`` is ``True``, ``"ssl_context"`` will be set and will be the
default Requests SSL Context
* If ``verify`` is ``False``, ``"ssl_context"`` will not be set but
``"cert_reqs"`` will be set
* If ``verify`` is a string, (i.e., it is a user-specified trust bundle)
``"ca_certs"`` will be set if the string is not a directory recognized
by :py:func:`os.path.isdir`, otherwise ``"ca_cert_dir"`` will be
set.
* If ``"cert"`` is specified, ``"cert_file"`` will always be set. If
``"cert"`` is a tuple with a second item, ``"key_file"`` will also
be present
To override these settings, one may subclass this class, call this
method and use the above logic to change parameters as desired. For
example, if one wishes to use a custom :py:class:`ssl.SSLContext` one
must both set ``"ssl_context"`` and based on what else they require,
alter the other keys to ensure the desired behaviour.
:param request:
The PreparedRequest being sent over the connection.
:type request:
:class:`~requests.models.PreparedRequest`
:param verify:
Either a boolean, in which case it controls whether
we verify the server's TLS certificate, or a string, in which case it
must be a path to a CA bundle to use.
:param cert:
(optional) Any user-provided SSL certificate for client
authentication (a.k.a., mTLS). This may be a string (i.e., just
the path to a file which holds both certificate and key) or a
tuple of length 2 with the certificate file path and key file
path.
:returns:
A tuple of two dictionaries. The first is the "host parameters"
portion of the Pool Key including scheme, hostname, and port. The
second is a dictionary of SSLContext related parameters.
"""
return _urllib3_request_context(request, verify, cert, self.poolmanager)
def get_connection_with_tls_context(
self,
request: PreparedRequest,
verify: _t.VerifyType,
proxies: dict[str, str] | None = None,
cert: _t.CertType = None,
) -> HTTPConnectionPool:
"""Returns a urllib3 connection for the given request and TLS settings.
This should not be called from user code, and is only exposed for use
when subclassing the :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param request:
The :class:`PreparedRequest <PreparedRequest>` object to be sent
over the connection.
:param verify:
Either a boolean, in which case it controls whether we verify the
server's TLS certificate, or a string, in which case it must be a
path to a CA bundle to use.
:param proxies:
(optional) The proxies dictionary to apply to the request.
:param cert:
(optional) Any user-provided SSL certificate to be used for client
authentication (a.k.a., mTLS).
:rtype:
urllib3.HTTPConnectionPool
"""
assert _is_prepared(request)
proxy = select_proxy(request.url, proxies)
try:
host_params, pool_kwargs = self.build_connection_pool_key_attributes(
request,
verify,
cert,
)
except ValueError as e:
raise InvalidURL(e, request=request)
if proxy:
proxy = prepend_scheme_if_needed(proxy, "http")
proxy_url = parse_url(proxy)
if not proxy_url.host:
raise InvalidProxyURL(
"Please check proxy URL. It is malformed "
"and could be missing the host."
)
proxy_manager = self.proxy_manager_for(proxy)
conn = proxy_manager.connection_from_host(
**host_params, pool_kwargs=pool_kwargs
)
else:
# Only scheme should be lower case
conn = self.poolmanager.connection_from_host(
**host_params, pool_kwargs=pool_kwargs
)
return conn
def get_connection(
self, url: str, proxies: dict[str, str] | None = None
) -> HTTPConnectionPool:
"""DEPRECATED: Users should move to `get_connection_with_tls_context`
for all subclasses of HTTPAdapter using Requests>=2.32.2.
Returns a urllib3 connection for the given URL. This should not be
called from user code, and is only exposed for use when subclassing the
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param url: The URL to connect to.
:param proxies: (optional) A Requests-style dictionary of proxies used on this request.
:rtype: urllib3.HTTPConnectionPool
"""
warnings.warn(
(
"`get_connection` has been deprecated in favor of "
"`get_connection_with_tls_context`. Custom HTTPAdapter subclasses "
"will need to migrate for Requests>=2.32.2. Please see "
"https://github.com/psf/requests/pull/6710 for more details."
),
DeprecationWarning,
)
proxy = select_proxy(url, proxies)
if proxy:
proxy = prepend_scheme_if_needed(proxy, "http")
proxy_url = parse_url(proxy)
if not proxy_url.host:
raise InvalidProxyURL(
"Please check proxy URL. It is malformed "
"and could be missing the host."
)
proxy_manager = self.proxy_manager_for(proxy)
conn = proxy_manager.connection_from_url(url)
else:
# Only scheme should be lower case
parsed = urlparse(url)
url = parsed.geturl()
conn = self.poolmanager.connection_from_url(url)
return conn
def close(self) -> None:
"""Disposes of any internal state.
Currently, this closes the PoolManager and any active ProxyManager,
which closes any pooled connections.
"""
self.poolmanager.clear()
for proxy in self.proxy_manager.values():
proxy.clear()
def request_url(
self, request: PreparedRequest, proxies: dict[str, str] | None
) -> str:
"""Obtain the url to use when making the final request.
If the message is being sent through a HTTP proxy, the full URL has to
be used. Otherwise, we should only use the path portion of the URL.
This should not be called from user code, and is only exposed for use
when subclassing the
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
:param proxies: A dictionary of schemes or schemes and hosts to proxy URLs.
:rtype: str
"""
assert _is_prepared(request)
proxy = select_proxy(request.url, proxies)
scheme = urlparse(request.url).scheme
is_proxied_http_request = proxy and scheme != "https"
using_socks_proxy = False
if proxy:
proxy_scheme = urlparse(proxy).scheme.lower()
using_socks_proxy = proxy_scheme.startswith("socks")
url = request.path_url
if is_proxied_http_request and not using_socks_proxy:
url = urldefragauth(request.url)
return url
def add_headers(self, request: PreparedRequest, **kwargs: Any) -> None:
"""Add any headers needed by the connection. As of v2.0 this does
nothing by default, but is left for overriding by users that subclass
the :class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
This should not be called from user code, and is only exposed for use
when subclassing the
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param request: The :class:`PreparedRequest <PreparedRequest>` to add headers to.
:param kwargs: The keyword arguments from the call to send().
"""
pass
def proxy_headers(self, proxy: str) -> dict[str, str]:
"""Returns a dictionary of the headers to add to any request sent
through a proxy. This works with urllib3 magic to ensure that they are
correctly sent to the proxy, rather than in a tunnelled request if
CONNECT is being used.
This should not be called from user code, and is only exposed for use
when subclassing the
:class:`HTTPAdapter <requests.adapters.HTTPAdapter>`.
:param proxy: The url of the proxy being used for this request.
:rtype: dict
"""
headers: dict[str, str] = {}
username, password = get_auth_from_url(proxy)
if username:
headers["Proxy-Authorization"] = _basic_auth_str(username, password)
return headers
def send(
self,
request: PreparedRequest,
stream: bool = False,
timeout: _t.TimeoutType = None,
verify: _t.VerifyType = True,
cert: _t.CertType = None,
proxies: dict[str, str] | None = None,
) -> Response:
"""Sends PreparedRequest object. Returns Response object.
:param request: The :class:`PreparedRequest <PreparedRequest>` being sent.
:param stream: (optional) Whether to stream the request content.
:param timeout: (optional) How long to wait for the server to send
data before giving up, as a float, or a :ref:`(connect timeout,
read timeout) <timeouts>` tuple.
:type timeout: float or tuple or urllib3 Timeout object
:param verify: (optional) Either a boolean, in which case it controls whether
we verify the server's TLS certificate, or a string, in which case it
must be a path to a CA bundle to use
:param cert: (optional) Any user-provided SSL certificate to be trusted.
:param proxies: (optional) The proxies dictionary to apply to the request.
:rtype: requests.Response
"""
assert _is_prepared(request)
try:
conn = self.get_connection_with_tls_context(
request, verify, proxies=proxies, cert=cert
)
except LocationValueError as e:
raise InvalidURL(e, request=request)
self.cert_verify(conn, request.url, verify, cert)
url = self.request_url(request, proxies)
self.add_headers(
request,
stream=stream,
timeout=timeout,
verify=verify,
cert=cert,
proxies=proxies,
)
chunked = not (request.body is None or "Content-Length" in request.headers)
if isinstance(timeout, tuple):
try:
connect, read = timeout
resolved_timeout = TimeoutSauce(connect=connect, read=read)
except ValueError:
raise ValueError(
f"Invalid timeout {timeout}. Pass a (connect, read) timeout tuple, "
f"or a single float to set both timeouts to the same value."
)
elif isinstance(timeout, TimeoutSauce):
resolved_timeout = timeout
else:
resolved_timeout = TimeoutSauce(connect=timeout, read=timeout)
try:
resp = conn.urlopen(
method=request.method,
url=url,
body=request.body, # type: ignore[arg-type] # urllib3 stubs don't accept Iterable[bytes | str]
headers=request.headers, # type: ignore[arg-type] # urllib3#3072
redirect=False,
assert_same_host=False,
preload_content=False,
decode_content=False,
retries=self.max_retries,
timeout=resolved_timeout,
chunked=chunked,
)
except (ProtocolError, OSError) as err:
raise ConnectionError(err, request=request)
except MaxRetryError as e:
if isinstance(e.reason, ConnectTimeoutError):
# TODO: Remove this in 3.0.0: see #2811
if not isinstance(e.reason, NewConnectionError):
raise ConnectTimeout(e, request=request)
if isinstance(e.reason, ResponseError):
raise RetryError(e, request=request)
if isinstance(e.reason, _ProxyError):
raise ProxyError(e, request=request)
if isinstance(e.reason, _SSLError):
# This branch is for urllib3 v1.22 and later.
raise SSLError(e, request=request)
raise ConnectionError(e, request=request)
except ClosedPoolError as e:
raise ConnectionError(e, request=request)
except _ProxyError as e:
raise ProxyError(e)
except (_SSLError, _HTTPError) as e:
if isinstance(e, _SSLError):
# This branch is for urllib3 versions earlier than v1.22
raise SSLError(e, request=request)
elif isinstance(e, ReadTimeoutError):
raise ReadTimeout(e, request=request)
elif isinstance(e, _InvalidHeader):
raise InvalidHeader(e, request=request)
else:
raise
return self.build_response(request, resp)
@@ -0,0 +1,180 @@
"""
requests.api
~~~~~~~~~~~~
This module implements the Requests API.
:copyright: (c) 2012 by Kenneth Reitz.
:license: Apache2, see LICENSE for more details.
"""
from __future__ import annotations
from typing import TYPE_CHECKING
from . import sessions
from .models import Response
if TYPE_CHECKING:
from typing_extensions import Unpack
from . import _types as _t
def request(
method: str, url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]
) -> Response:
"""Constructs and sends a :class:`Request <Request>`.
:param method: method for the new :class:`Request` object: ``GET``, ``OPTIONS``, ``HEAD``, ``POST``, ``PUT``, ``PATCH``, or ``DELETE``.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary, list of tuples or bytes to send
in the query string for the :class:`Request`.
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
object to send in the body of the :class:`Request`.
:param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`.
:param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
:param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
:param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': file-tuple}``) for multipart encoding upload.
``file-tuple`` can be a 2-tuple ``('filename', fileobj)``, 3-tuple ``('filename', fileobj, 'content_type')``
or a 4-tuple ``('filename', fileobj, 'content_type', custom_headers)``, where ``'content_type'`` is a string
defining the content type of the given file and ``custom_headers`` a dict-like object containing additional headers
to add for the file.
:param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
:param timeout: (optional) How many seconds to wait for the server to send data
before giving up, as a float, or a :ref:`(connect timeout, read
timeout) <timeouts>` tuple.
:type timeout: float or tuple
:param allow_redirects: (optional) Boolean. Enable/disable GET/OPTIONS/POST/PUT/PATCH/DELETE/HEAD redirection. Defaults to ``True``.
:type allow_redirects: bool
:param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
:param verify: (optional) Either a boolean, in which case it controls whether we verify
the server's TLS certificate, or a string, in which case it must be a path
to a CA bundle to use. Defaults to ``True``.
:param stream: (optional) if ``False``, the response content will be immediately downloaded.
:param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
:return: :class:`Response <Response>` object
:rtype: requests.Response
Usage::
>>> import requests
>>> req = requests.request('GET', 'https://httpbin.org/get')
>>> req
<Response [200]>
"""
# By using the 'with' statement we are sure the session is closed, thus we
# avoid leaving sockets open which can trigger a ResourceWarning in some
# cases, and look like a memory leak in others.
with sessions.Session() as session:
return session.request(method=method, url=url, **kwargs)
def get(
url: _t.UriType, params: _t.ParamsType = None, **kwargs: Unpack[_t.GetKwargs]
) -> Response:
r"""Sends a GET request.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary, list of tuples or bytes to send
in the query string for the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
return request("get", url, params=params, **kwargs)
def options(url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]) -> Response:
r"""Sends an OPTIONS request.
:param url: URL for the new :class:`Request` object.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
return request("options", url, **kwargs)
def head(url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]) -> Response:
r"""Sends a HEAD request.
:param url: URL for the new :class:`Request` object.
:param \*\*kwargs: Optional arguments that ``request`` takes. If
`allow_redirects` is not provided, it will be set to `False` (as
opposed to the default :meth:`request` behavior).
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
kwargs.setdefault("allow_redirects", False)
return request("head", url, **kwargs)
def post(
url: _t.UriType,
data: _t.DataType = None,
json: _t.JsonType = None,
**kwargs: Unpack[_t.PostKwargs],
) -> Response:
r"""Sends a POST request.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
object to send in the body of the :class:`Request`.
:param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
return request("post", url, data=data, json=json, **kwargs)
def put(
url: _t.UriType, data: _t.DataType = None, **kwargs: Unpack[_t.DataKwargs]
) -> Response:
r"""Sends a PUT request.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
object to send in the body of the :class:`Request`.
:param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
return request("put", url, data=data, **kwargs)
def patch(
url: _t.UriType, data: _t.DataType = None, **kwargs: Unpack[_t.DataKwargs]
) -> Response:
r"""Sends a PATCH request.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
object to send in the body of the :class:`Request`.
:param json: (optional) A JSON serializable Python object to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
return request("patch", url, data=data, **kwargs)
def delete(url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]) -> Response:
r"""Sends a DELETE request.
:param url: URL for the new :class:`Request` object.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:return: :class:`Response <Response>` object
:rtype: requests.Response
"""
return request("delete", url, **kwargs)
@@ -0,0 +1,354 @@
"""
requests.auth
~~~~~~~~~~~~~
This module contains the authentication handlers for Requests.
"""
from __future__ import annotations
import hashlib
import os
import re
import threading
import time
import warnings
from base64 import b64encode
from typing import TYPE_CHECKING, Any, Final, cast, overload
from ._internal_utils import to_native_string
from .compat import basestring, str, urlparse
from .cookies import extract_cookies_to_jar
from .utils import parse_dict_header
if TYPE_CHECKING:
from http.cookiejar import CookieJar
from typing import Any
from .models import PreparedRequest, Response
CONTENT_TYPE_FORM_URLENCODED: Final = "application/x-www-form-urlencoded"
CONTENT_TYPE_MULTI_PART: Final = "multipart/form-data"
def _basic_auth_str(username: bytes | str, password: bytes | str) -> str:
"""Returns a Basic Auth string."""
# "I want us to put a big-ol' comment on top of it that
# says that this behaviour is dumb but we need to preserve
# it because people are relying on it."
# - Lukasa
#
# These are here solely to maintain backwards compatibility
# for things like ints. This will be removed in 3.0.0.
if not isinstance(username, basestring): # type: ignore[reportUnnecessaryIsInstance] # runtime guard for non-str/bytes
warnings.warn(
"Non-string usernames will no longer be supported in Requests "
f"3.0.0. Please convert the object you've passed in ({username!r}) to "
"a string or bytes object in the near future to avoid "
"problems.",
category=DeprecationWarning,
)
username = str(username)
if not isinstance(password, basestring): # type: ignore[reportUnnecessaryIsInstance] # runtime guard for non-str/bytes
warnings.warn(
"Non-string passwords will no longer be supported in Requests "
f"3.0.0. Please convert the object you've passed in ({type(password)!r}) to "
"a string or bytes object in the near future to avoid "
"problems.",
category=DeprecationWarning,
)
password = str(password)
# -- End Removal --
if isinstance(username, str):
username = username.encode("latin1")
if isinstance(password, str):
password = password.encode("latin1")
authstr = "Basic " + to_native_string(
b64encode(b":".join((username, password))).strip()
)
return authstr
class AuthBase:
"""Base class that all auth implementations derive from"""
def __call__(self, r: PreparedRequest) -> PreparedRequest:
raise NotImplementedError("Auth hooks must be callable.")
class HTTPBasicAuth(AuthBase):
"""Attaches HTTP Basic Authentication to the given Request object."""
username: bytes | str
password: bytes | str
@overload
def __init__(self, username: str, password: str) -> None: ...
@overload
def __init__(self, username: bytes, password: bytes) -> None: ...
def __init__(self, username: bytes | str, password: bytes | str) -> None:
self.username = username
self.password = password
def __eq__(self, other: object) -> bool:
return all(
[
self.username == getattr(other, "username", None),
self.password == getattr(other, "password", None),
]
)
def __ne__(self, other: Any) -> bool:
return not self == other
def __call__(self, r: PreparedRequest) -> PreparedRequest:
r.headers["Authorization"] = _basic_auth_str(self.username, self.password)
return r
class HTTPProxyAuth(HTTPBasicAuth):
"""Attaches HTTP Proxy Authentication to a given Request object."""
def __call__(self, r: PreparedRequest) -> PreparedRequest:
r.headers["Proxy-Authorization"] = _basic_auth_str(self.username, self.password)
return r
class HTTPDigestAuth(AuthBase):
"""Attaches HTTP Digest Authentication to the given Request object."""
username: bytes | str
password: bytes | str
_thread_local: threading.local
last_nonce: str
nonce_count: int
chal: dict[str, str]
pos: int | None
num_401_calls: int | None
@overload
def __init__(self, username: str, password: str) -> None: ...
@overload
def __init__(self, username: bytes, password: bytes) -> None: ...
def __init__(self, username: bytes | str, password: bytes | str) -> None:
self.username = username
self.password = password
# Keep state in per-thread local storage
self._thread_local = threading.local()
def init_per_thread_state(self) -> None:
# Ensure state is initialized just once per-thread
if not hasattr(self._thread_local, "init"):
self._thread_local.init = True
self._thread_local.last_nonce = ""
self._thread_local.nonce_count = 0
self._thread_local.chal = {}
self._thread_local.pos = None
self._thread_local.num_401_calls = None
def build_digest_header(self, method: str, url: str) -> str | None:
"""
:rtype: str
"""
realm = self._thread_local.chal["realm"]
nonce = self._thread_local.chal["nonce"]
qop = self._thread_local.chal.get("qop")
algorithm = self._thread_local.chal.get("algorithm")
opaque = self._thread_local.chal.get("opaque")
hash_utf8 = None
if algorithm is None:
_algorithm = "MD5"
else:
_algorithm = algorithm.upper()
# lambdas assume digest modules are imported at the top level
if _algorithm == "MD5" or _algorithm == "MD5-SESS":
def md5_utf8(x: str | bytes) -> str:
if isinstance(x, str):
x = x.encode("utf-8")
return hashlib.md5(x, usedforsecurity=False).hexdigest()
hash_utf8 = md5_utf8
elif _algorithm == "SHA":
def sha_utf8(x: str | bytes) -> str:
if isinstance(x, str):
x = x.encode("utf-8")
return hashlib.sha1(x, usedforsecurity=False).hexdigest()
hash_utf8 = sha_utf8
elif _algorithm == "SHA-256":
def sha256_utf8(x: str | bytes) -> str:
if isinstance(x, str):
x = x.encode("utf-8")
return hashlib.sha256(x, usedforsecurity=False).hexdigest()
hash_utf8 = sha256_utf8
elif _algorithm == "SHA-512":
def sha512_utf8(x: str | bytes) -> str:
if isinstance(x, str):
x = x.encode("utf-8")
return hashlib.sha512(x, usedforsecurity=False).hexdigest()
hash_utf8 = sha512_utf8
if hash_utf8 is None:
return None
def KD(s: str, d: str) -> str:
return hash_utf8(f"{s}:{d}")
# XXX not implemented yet
entdig = None
p_parsed = urlparse(url)
#: path is request-uri defined in RFC 2616 which should not be empty
path = p_parsed.path or "/"
if p_parsed.query:
path += f"?{p_parsed.query}"
A1 = f"{self.username}:{realm}:{self.password}"
A2 = f"{method}:{path}"
HA1 = hash_utf8(A1)
HA2 = hash_utf8(A2)
if nonce == self._thread_local.last_nonce:
self._thread_local.nonce_count += 1
else:
self._thread_local.nonce_count = 1
ncvalue = f"{self._thread_local.nonce_count:08x}"
s = str(self._thread_local.nonce_count).encode("utf-8")
s += nonce.encode("utf-8")
s += time.ctime().encode("utf-8")
s += os.urandom(8)
cnonce = hashlib.sha1(s, usedforsecurity=False).hexdigest()[:16]
if _algorithm == "MD5-SESS":
HA1 = hash_utf8(f"{HA1}:{nonce}:{cnonce}") # type: ignore[reportConstantRedefinition] # RFC 2617 terminology
if not qop:
respdig = KD(HA1, f"{nonce}:{HA2}")
elif qop == "auth" or "auth" in qop.split(","):
noncebit = f"{nonce}:{ncvalue}:{cnonce}:auth:{HA2}"
respdig = KD(HA1, noncebit)
else:
# XXX handle auth-int.
return None
self._thread_local.last_nonce = nonce
# XXX should the partial digests be encoded too?
base = (
f'username="{self.username}", realm="{realm}", nonce="{nonce}", '
f'uri="{path}", response="{respdig}"'
)
if opaque:
base += f', opaque="{opaque}"'
if algorithm:
base += f', algorithm="{algorithm}"'
if entdig:
base += f', digest="{entdig}"'
if qop:
base += f', qop="auth", nc={ncvalue}, cnonce="{cnonce}"'
return f"Digest {base}"
def handle_redirect(self, r: Response, **kwargs: Any) -> None:
"""Reset num_401_calls counter on redirects."""
if r.is_redirect:
self._thread_local.num_401_calls = 1
def handle_401(self, r: Response, **kwargs: Any) -> Response:
"""
Takes the given response and tries digest-auth, if needed.
:rtype: requests.Response
"""
# If response is not 4xx, do not auth
# See https://github.com/psf/requests/issues/3772
if not 400 <= r.status_code < 500:
self._thread_local.num_401_calls = 1
return r
if self._thread_local.pos is not None:
# Rewind the file position indicator of the body to where
# it was to resend the request.
if (seek := getattr(r.request.body, "seek", None)) is not None:
seek(self._thread_local.pos)
s_auth = r.headers.get("www-authenticate", "")
if "digest" in s_auth.lower() and self._thread_local.num_401_calls < 2:
self._thread_local.num_401_calls += 1
pat = re.compile(r"digest ", flags=re.IGNORECASE)
self._thread_local.chal = parse_dict_header(pat.sub("", s_auth, count=1))
# Consume content and release the original connection
# to allow our new request to reuse the same one.
r.content
r.close()
prep = r.request.copy()
cookie_jar = cast("CookieJar", prep._cookies) # type: ignore[reportPrivateUsage]
extract_cookies_to_jar(cookie_jar, r.request, r.raw)
prep.prepare_cookies(cookie_jar)
_digest_auth = self.build_digest_header(
cast(str, prep.method), cast(str, prep.url)
)
if _digest_auth:
prep.headers["Authorization"] = _digest_auth
_r = r.connection.send(prep, **kwargs)
_r.history.append(r)
_r.request = prep
return _r
self._thread_local.num_401_calls = 1
return r
def __call__(self, r: PreparedRequest) -> PreparedRequest:
# Initialize per-thread state, if needed
self.init_per_thread_state()
# If we have a saved nonce, skip the 401
if self._thread_local.last_nonce:
_digest_auth = self.build_digest_header(
cast(str, r.method), cast(str, r.url)
)
if _digest_auth:
r.headers["Authorization"] = _digest_auth
if (tell := getattr(r.body, "tell", None)) is not None:
self._thread_local.pos = tell()
else:
# In the case of HTTPDigestAuth being reused and the body of
# the previous request was a file-like object, pos has the
# file position of the previous body. Ensure it's set to
# None.
self._thread_local.pos = None
r.register_hook("response", self.handle_401)
r.register_hook("response", self.handle_redirect)
self._thread_local.num_401_calls = 1
return r
def __eq__(self, other: object) -> bool:
return all(
[
self.username == getattr(other, "username", None),
self.password == getattr(other, "password", None),
]
)
def __ne__(self, other: Any) -> bool:
return not self == other
@@ -0,0 +1,18 @@
#!/usr/bin/env python
"""
requests.certs
~~~~~~~~~~~~~~
This module returns the preferred default CA certificate bundle. There is
only one — the one from the certifi package.
If you are packaging Requests, e.g., for a Linux distribution or a managed
environment, you can change the definition of where() to return a separately
packaged CA bundle.
"""
from certifi import where
if __name__ == "__main__":
print(where())
@@ -0,0 +1,113 @@
"""
requests.compat
~~~~~~~~~~~~~~~
This module previously handled import compatibility issues
between Python 2 and Python 3. It remains for backwards
compatibility until the next major version.
"""
# pyright: reportUnusedImport=false
from __future__ import annotations
import importlib
import sys
from types import ModuleType
# -------
# urllib3
# -------
from urllib3 import (
__version__ as urllib3_version, # type: ignore[reportPrivateImportUsage]
)
# Detect which major version of urllib3 is being used.
try:
is_urllib3_1 = int(urllib3_version.split(".")[0]) == 1
except (TypeError, AttributeError):
# If we can't discern a version, prefer old functionality.
is_urllib3_1 = True
# -------------------
# Character Detection
# -------------------
def _resolve_char_detection() -> ModuleType | None:
"""Find supported character detection libraries."""
chardet = None
for lib in ("chardet", "charset_normalizer"):
if chardet is None:
try:
chardet = importlib.import_module(lib)
except ImportError:
pass
return chardet
chardet = _resolve_char_detection()
# -------
# Pythons
# -------
# Syntax sugar.
_ver = sys.version_info
#: Python 2.x?
is_py2 = _ver[0] == 2
#: Python 3.x?
is_py3 = _ver[0] == 3
# json/simplejson module import resolution
has_simplejson = False
try:
import simplejson as json # type: ignore[import-not-found]
has_simplejson = True
except ImportError:
import json
if has_simplejson:
from simplejson import JSONDecodeError # type: ignore[import-not-found]
else:
from json import JSONDecodeError
# Keep OrderedDict for backwards compatibility.
from collections import OrderedDict
from collections.abc import Callable, Mapping, MutableMapping
from http import cookiejar as cookielib
from http.cookies import Morsel
from io import StringIO
# --------------
# Legacy Imports
# --------------
from urllib.parse import (
quote,
quote_plus,
unquote,
unquote_plus,
urldefrag,
urlencode,
urljoin,
urlparse,
urlsplit,
urlunparse,
)
from urllib.request import (
getproxies,
getproxies_environment,
parse_http_list,
proxy_bypass,
proxy_bypass_environment, # type: ignore[attr-defined] # https://github.com/python/cpython/issues/145331
)
builtin_str = str
str = str
bytes = bytes
basestring = (str, bytes)
numeric_types = (int, float)
integer_types = (int,)
@@ -0,0 +1,625 @@
"""
requests.cookies
~~~~~~~~~~~~~~~~
Compatibility code to be able to use `http.cookiejar.CookieJar` with requests.
requests.utils imports from here, so be careful with imports.
"""
from __future__ import annotations
import calendar
import copy
import time
from collections.abc import Iterator, MutableMapping
from http.cookiejar import Cookie, CookieJar, CookiePolicy
from typing import TYPE_CHECKING, Any, TypeVar, overload
from ._internal_utils import to_native_string
from ._types import is_prepared as _is_prepared
from .compat import Morsel, cookielib, urlparse, urlunparse
if TYPE_CHECKING:
from _typeshed import SupportsKeysAndGetItem
from .models import PreparedRequest
import threading
class MockRequest:
"""Wraps a `requests.PreparedRequest` to mimic a `urllib2.Request`.
The code in `http.cookiejar.CookieJar` expects this interface in order to correctly
manage cookie policies, i.e., determine whether a cookie can be set, given the
domains of the request and the cookie.
The original request object is read-only. The client is responsible for collecting
the new headers via `get_new_headers()` and interpreting them appropriately. You
probably want `get_cookie_header`, defined below.
"""
type: str
def __init__(self, request: PreparedRequest) -> None:
assert _is_prepared(request)
self._r = request
self._new_headers: dict[str, str] = {}
self.type = urlparse(self._r.url).scheme
def get_type(self) -> str:
return self.type
def get_host(self) -> str:
return urlparse(self._r.url).netloc
def get_origin_req_host(self) -> str:
return self.get_host()
def get_full_url(self) -> str:
# Only return the response's URL if the user hadn't set the Host
# header
if not self._r.headers.get("Host"):
return self._r.url
# If they did set it, retrieve it and reconstruct the expected domain
host = to_native_string(self._r.headers["Host"], encoding="utf-8")
parsed = urlparse(self._r.url)
# Reconstruct the URL as we expect it
return urlunparse(
[
parsed.scheme,
host,
parsed.path,
parsed.params,
parsed.query,
parsed.fragment,
]
)
def is_unverifiable(self) -> bool:
return True
def has_header(self, name: str) -> bool:
return name in self._r.headers or name in self._new_headers
def get_header(self, name: str, default: str | None = None) -> str | None:
return self._r.headers.get(name, self._new_headers.get(name, default)) # type: ignore[return-value]
def add_header(self, key: str, val: str) -> None:
"""cookiejar has no legitimate use for this method; add it back if you find one."""
raise NotImplementedError(
"Cookie headers should be added with add_unredirected_header()"
)
def add_unredirected_header(self, name: str, value: str) -> None:
self._new_headers[name] = value
def get_new_headers(self) -> dict[str, str]:
return self._new_headers
@property
def unverifiable(self) -> bool:
return self.is_unverifiable()
@property
def origin_req_host(self) -> str:
return self.get_origin_req_host()
@property
def host(self) -> str:
return self.get_host()
class MockResponse:
"""Wraps a `httplib.HTTPMessage` to mimic a `urllib.addinfourl`.
...what? Basically, expose the parsed HTTP headers from the server response
the way `http.cookiejar` expects to see them.
"""
def __init__(self, headers: Any) -> None:
"""Make a MockResponse for `cookiejar` to read.
:param headers: a httplib.HTTPMessage or analogous carrying the headers
"""
self._headers = headers
def info(self) -> Any:
return self._headers
def getheaders(self, name: str) -> Any:
self._headers.getheaders(name)
def extract_cookies_to_jar(
jar: CookieJar, request: PreparedRequest, response: Any
) -> None:
"""Extract the cookies from the response into a CookieJar.
:param jar: http.cookiejar.CookieJar (not necessarily a RequestsCookieJar)
:param request: our own requests.Request object
:param response: urllib3.HTTPResponse object
"""
if not (hasattr(response, "_original_response") and response._original_response):
return
# the _original_response field is the wrapped httplib.HTTPResponse object,
req = MockRequest(request)
# pull out the HTTPMessage with the headers and put it in the mock:
res = MockResponse(response._original_response.msg)
jar.extract_cookies(res, req) # type: ignore[arg-type]
def get_cookie_header(jar: CookieJar, request: PreparedRequest) -> str | None:
"""
Produce an appropriate Cookie header string to be sent with `request`, or None.
:rtype: str
"""
r = MockRequest(request)
jar.add_cookie_header(r) # type: ignore[arg-type]
return r.get_new_headers().get("Cookie")
def remove_cookie_by_name(
cookiejar: CookieJar, name: str, domain: str | None = None, path: str | None = None
) -> None:
"""Unsets a cookie by name, by default over all domains and paths.
Wraps CookieJar.clear(), is O(n).
"""
clearables: list[tuple[str, str, str]] = []
for cookie in cookiejar:
if cookie.name != name:
continue
if domain is not None and domain != cookie.domain:
continue
if path is not None and path != cookie.path:
continue
clearables.append((cookie.domain, cookie.path, cookie.name))
for domain, path, name in clearables:
cookiejar.clear(domain, path, name)
class CookieConflictError(RuntimeError):
"""There are two cookies that meet the criteria specified in the cookie jar.
Use .get and .set and include domain and path args in order to be more specific.
"""
class RequestsCookieJar(CookieJar, MutableMapping[str, str | None]): # type: ignore[misc]
"""Compatibility class; is a http.cookiejar.CookieJar, but exposes a dict
interface.
This is the CookieJar we create by default for requests and sessions that
don't specify one, since some clients may expect response.cookies and
session.cookies to support dict operations.
Requests does not use the dict interface internally; it's just for
compatibility with external client code. All requests code should work
out of the box with externally provided instances of ``CookieJar``, e.g.
``LWPCookieJar`` and ``FileCookieJar``.
Unlike a regular CookieJar, this class is pickleable.
.. warning:: dictionary operations that are normally O(1) may be O(n).
"""
_policy: CookiePolicy
def get( # type: ignore[override]
self,
name: str,
default: str | None = None,
domain: str | None = None,
path: str | None = None,
) -> str | None:
"""Dict-like get() that also supports optional domain and path args in
order to resolve naming collisions from using one cookie jar over
multiple domains.
.. warning:: operation is O(n), not O(1).
"""
try:
return self._find_no_duplicates(name, domain, path)
except KeyError:
return default
def set(
self, name: str, value: str | Morsel[dict[str, str]] | None, **kwargs: Any
) -> Cookie | None:
"""Dict-like set() that also supports optional domain and path args in
order to resolve naming collisions from using one cookie jar over
multiple domains.
"""
# support client code that unsets cookies by assignment of a None value:
if value is None:
remove_cookie_by_name(
self, name, domain=kwargs.get("domain"), path=kwargs.get("path")
)
return
if isinstance(value, Morsel):
c = morsel_to_cookie(value)
else:
c = create_cookie(name, value, **kwargs)
self.set_cookie(c)
return c
def iterkeys(self) -> Iterator[str]:
"""Dict-like iterkeys() that returns an iterator of names of cookies
from the jar.
.. seealso:: itervalues() and iteritems().
"""
for cookie in iter(self):
yield cookie.name
def keys(self) -> list[str]: # type: ignore[override]
"""Dict-like keys() that returns a list of names of cookies from the
jar.
.. seealso:: values() and items().
"""
return list(self.iterkeys())
def itervalues(self) -> Iterator[str | None]:
"""Dict-like itervalues() that returns an iterator of values of cookies
from the jar.
.. seealso:: iterkeys() and iteritems().
"""
for cookie in iter(self):
yield cookie.value
def values(self) -> list[str | None]: # type: ignore[override]
"""Dict-like values() that returns a list of values of cookies from the
jar.
.. seealso:: keys() and items().
"""
return list(self.itervalues())
def iteritems(self) -> Iterator[tuple[str, str | None]]:
"""Dict-like iteritems() that returns an iterator of name-value tuples
from the jar.
.. seealso:: iterkeys() and itervalues().
"""
for cookie in iter(self):
yield cookie.name, cookie.value
def items(self) -> list[tuple[str, str | None]]: # type: ignore[override]
"""Dict-like items() that returns a list of name-value tuples from the
jar. Allows client-code to call ``dict(RequestsCookieJar)`` and get a
vanilla python dict of key value pairs.
.. seealso:: keys() and values().
"""
return list(self.iteritems())
def list_domains(self) -> list[str]:
"""Utility method to list all the domains in the jar."""
domains: list[str] = []
for cookie in iter(self):
if cookie.domain not in domains:
domains.append(cookie.domain)
return domains
def list_paths(self) -> list[str]:
"""Utility method to list all the paths in the jar."""
paths: list[str] = []
for cookie in iter(self):
if cookie.path not in paths:
paths.append(cookie.path)
return paths
def multiple_domains(self) -> bool:
"""Returns True if there are multiple domains in the jar.
Returns False otherwise.
:rtype: bool
"""
domains: list[str] = []
for cookie in iter(self):
if cookie.domain is not None and cookie.domain in domains: # type: ignore[reportUnnecessaryComparison] # defensive check
return True
domains.append(cookie.domain)
return False # there is only one domain in jar
def get_dict(
self, domain: str | None = None, path: str | None = None
) -> dict[str, str | None]:
"""Takes as an argument an optional domain and path and returns a plain
old Python dict of name-value pairs of cookies that meet the
requirements.
:rtype: dict
"""
dictionary: dict[str, str | None] = {}
for cookie in iter(self):
if (domain is None or cookie.domain == domain) and (
path is None or cookie.path == path
):
dictionary[cookie.name] = cookie.value
return dictionary
def __iter__(self) -> Iterator[Cookie]: # type: ignore[override]
"""RequestCookieJar's __iter__ comes from CookieJar not MutableMapping."""
return super().__iter__()
def __contains__(self, name: object) -> bool:
try:
return super().__contains__(name)
except CookieConflictError:
return True
def __getitem__(self, name: str) -> str | None:
"""Dict-like __getitem__() for compatibility with client code. Throws
exception if there are more than one cookie with name. In that case,
use the more explicit get() method instead.
.. warning:: operation is O(n), not O(1).
"""
return self._find_no_duplicates(name)
def __setitem__(
self, name: str, value: str | Morsel[dict[str, str]] | None
) -> None:
"""Dict-like __setitem__ for compatibility with client code. Throws
exception if there is already a cookie of that name in the jar. In that
case, use the more explicit set() method instead.
"""
self.set(name, value)
def __delitem__(self, name: str) -> None:
"""Deletes a cookie given a name. Wraps ``http.cookiejar.CookieJar``'s
``remove_cookie_by_name()``.
"""
remove_cookie_by_name(self, name)
def set_cookie(self, cookie: Cookie, *args: Any, **kwargs: Any) -> None:
if (
(value := cookie.value) is not None
and value.startswith('"')
and value.endswith('"')
):
cookie.value = value.replace('\\"', "")
return super().set_cookie(cookie, *args, **kwargs)
def update( # type: ignore[override]
self, other: CookieJar | SupportsKeysAndGetItem[str, str]
) -> None:
"""Updates this jar with cookies from another CookieJar or dict-like"""
if isinstance(other, cookielib.CookieJar):
for cookie in other:
self.set_cookie(copy.copy(cookie))
else:
super().update(other)
def _find(
self, name: str, domain: str | None = None, path: str | None = None
) -> str | None:
"""Requests uses this method internally to get cookie values.
If there are conflicting cookies, _find arbitrarily chooses one.
See _find_no_duplicates if you want an exception thrown if there are
conflicting cookies.
:param name: a string containing name of cookie
:param domain: (optional) string containing domain of cookie
:param path: (optional) string containing path of cookie
:return: cookie.value
"""
for cookie in iter(self):
if cookie.name == name:
if domain is None or cookie.domain == domain:
if path is None or cookie.path == path:
return cookie.value
raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
def _find_no_duplicates(
self, name: str, domain: str | None = None, path: str | None = None
) -> str:
"""Both ``__get_item__`` and ``get`` call this function: it's never
used elsewhere in Requests.
:param name: a string containing name of cookie
:param domain: (optional) string containing domain of cookie
:param path: (optional) string containing path of cookie
:raises KeyError: if cookie is not found
:raises CookieConflictError: if there are multiple cookies
that match name and optionally domain and path
:return: cookie.value
"""
toReturn = None
for cookie in iter(self):
if cookie.name == name:
if domain is None or cookie.domain == domain:
if path is None or cookie.path == path:
if toReturn is not None:
# if there are multiple cookies that meet passed in criteria
raise CookieConflictError(
f"There are multiple cookies with name, {name!r}"
)
# we will eventually return this as long as no cookie conflict
toReturn = cookie.value
if toReturn is not None:
return toReturn
raise KeyError(f"name={name!r}, domain={domain!r}, path={path!r}")
def __getstate__(self) -> dict[str, Any]:
"""Unlike a normal CookieJar, this class is pickleable."""
state = self.__dict__.copy()
# remove the unpickleable RLock object
state.pop("_cookies_lock")
return state
def __setstate__(self, state: dict[str, Any]) -> None:
"""Unlike a normal CookieJar, this class is pickleable."""
self.__dict__.update(state)
if "_cookies_lock" not in self.__dict__:
self._cookies_lock = threading.RLock()
def copy(self) -> RequestsCookieJar:
"""Return a copy of this RequestsCookieJar."""
new_cj = RequestsCookieJar()
new_cj.set_policy(self.get_policy())
new_cj.update(self)
return new_cj
def get_policy(self) -> CookiePolicy:
"""Return the CookiePolicy instance used."""
return self._policy
def _copy_cookie_jar(jar: CookieJar | None) -> CookieJar | None: # type: ignore[reportUnusedFunction] # cross-module usage in models.py
if jar is None:
return None
if copy_method := getattr(jar, "copy", None):
# We're dealing with an instance of RequestsCookieJar
return copy_method()
# We're dealing with a generic CookieJar instance
new_jar = copy.copy(jar)
new_jar.clear()
for cookie in jar:
new_jar.set_cookie(copy.copy(cookie))
return new_jar
def create_cookie(name: str, value: str, **kwargs: Any) -> Cookie:
"""Make a cookie from underspecified parameters.
By default, the pair of `name` and `value` will be set for the domain ''
and sent on every request (this is sometimes called a "supercookie").
"""
result: dict[str, Any] = {
"version": 0,
"name": name,
"value": value,
"port": None,
"domain": "",
"path": "/",
"secure": False,
"expires": None,
"discard": True,
"comment": None,
"comment_url": None,
"rest": {"HttpOnly": None},
"rfc2109": False,
}
badargs = set(kwargs) - set(result)
if badargs:
raise TypeError(
f"create_cookie() got unexpected keyword arguments: {list(badargs)}"
)
result.update(kwargs)
result["port_specified"] = bool(result["port"])
result["domain_specified"] = bool(result["domain"])
result["domain_initial_dot"] = result["domain"].startswith(".")
result["path_specified"] = bool(result["path"])
return cookielib.Cookie(**result)
def morsel_to_cookie(morsel: Morsel[Any]) -> Cookie:
"""Convert a Morsel object into a Cookie containing the one k/v pair."""
expires: int | None = None
if morsel["max-age"]:
try:
expires = int(time.time() + int(morsel["max-age"]))
except ValueError:
raise TypeError(f"max-age: {morsel['max-age']} must be integer")
elif morsel["expires"]:
time_template = "%a, %d-%b-%Y %H:%M:%S GMT"
expires = calendar.timegm(time.strptime(morsel["expires"], time_template))
return create_cookie(
comment=morsel["comment"],
comment_url=bool(morsel["comment"]),
discard=False,
domain=morsel["domain"],
expires=expires,
name=morsel.key,
path=morsel["path"],
port=None,
rest={"HttpOnly": morsel["httponly"]},
rfc2109=False,
secure=bool(morsel["secure"]),
value=morsel.value,
version=morsel["version"] or 0,
)
_CookieJarT = TypeVar("_CookieJarT", bound=CookieJar)
@overload
def cookiejar_from_dict(
cookie_dict: dict[str, str] | None,
cookiejar: None = None,
overwrite: bool = True,
) -> RequestsCookieJar: ...
@overload
def cookiejar_from_dict(
cookie_dict: dict[str, str] | None,
cookiejar: _CookieJarT,
overwrite: bool = True,
) -> _CookieJarT: ...
def cookiejar_from_dict(
cookie_dict: dict[str, str] | None,
cookiejar: CookieJar | None = None,
overwrite: bool = True,
) -> CookieJar:
"""Returns a CookieJar from a key/value dictionary.
:param cookie_dict: Dict of key/values to insert into CookieJar.
:param cookiejar: (optional) A cookiejar to add the cookies to.
:param overwrite: (optional) If False, will not replace cookies
already in the jar with new ones.
:rtype: CookieJar
"""
if cookiejar is None:
cookiejar = RequestsCookieJar()
if cookie_dict is not None:
names_from_jar = [cookie.name for cookie in cookiejar]
for name in cookie_dict:
if overwrite or (name not in names_from_jar):
cookiejar.set_cookie(create_cookie(name, cookie_dict[name]))
return cookiejar
def merge_cookies(
cookiejar: CookieJar, cookies: dict[str, str] | CookieJar | None
) -> CookieJar:
"""Add cookies to cookiejar and returns a merged CookieJar.
:param cookiejar: CookieJar object to add the cookies to.
:param cookies: Dictionary or CookieJar object to be added.
:rtype: CookieJar
"""
if not isinstance(cookiejar, cookielib.CookieJar): # type: ignore[reportUnnecessaryIsInstance] # runtime guard
raise ValueError("You can only merge into CookieJar")
if isinstance(cookies, dict):
cookiejar = cookiejar_from_dict(cookies, cookiejar=cookiejar, overwrite=False)
elif isinstance(cookies, cookielib.CookieJar):
if update_method := getattr(cookiejar, "update", None):
update_method(cookies)
else:
for cookie_in_jar in cookies:
cookiejar.set_cookie(cookie_in_jar)
return cookiejar
@@ -0,0 +1,162 @@
"""
requests.exceptions
~~~~~~~~~~~~~~~~~~~
This module contains the set of Requests' exceptions.
"""
from __future__ import annotations
from typing import TYPE_CHECKING, Any
from urllib3.exceptions import HTTPError as BaseHTTPError
from .compat import JSONDecodeError as CompatJSONDecodeError
if TYPE_CHECKING:
from .models import PreparedRequest, Request, Response
class RequestException(IOError):
"""There was an ambiguous exception that occurred while handling your
request.
"""
response: Response | None
request: Request | PreparedRequest | None
def __init__(self, *args: Any, **kwargs: Any) -> None:
"""Initialize RequestException with `request` and `response` objects."""
response: Response | None = kwargs.pop("response", None)
self.response = response
self.request = kwargs.pop("request", None)
if response is not None and not self.request and hasattr(response, "request"):
self.request = response.request
super().__init__(*args, **kwargs)
class InvalidJSONError(RequestException):
"""A JSON error occurred."""
class JSONDecodeError(InvalidJSONError, CompatJSONDecodeError):
"""Couldn't decode the text into json"""
def __init__(self, *args: Any, **kwargs: Any) -> None:
"""
Construct the JSONDecodeError instance first with all
args. Then use it's args to construct the IOError so that
the json specific args aren't used as IOError specific args
and the error message from JSONDecodeError is preserved.
"""
CompatJSONDecodeError.__init__(self, *args)
InvalidJSONError.__init__(self, *self.args, **kwargs)
def __reduce__(self) -> tuple[Any, ...] | str:
"""
The __reduce__ method called when pickling the object must
be the one from the JSONDecodeError (be it json/simplejson)
as it expects all the arguments for instantiation, not just
one like the IOError, and the MRO would by default call the
__reduce__ method from the IOError due to the inheritance order.
"""
return CompatJSONDecodeError.__reduce__(self)
class HTTPError(RequestException):
"""An HTTP error occurred."""
class ConnectionError(RequestException):
"""A Connection error occurred."""
class ProxyError(ConnectionError):
"""A proxy error occurred."""
class SSLError(ConnectionError):
"""An SSL error occurred."""
class Timeout(RequestException):
"""The request timed out.
Catching this error will catch both
:exc:`~requests.exceptions.ConnectTimeout` and
:exc:`~requests.exceptions.ReadTimeout` errors.
"""
class ConnectTimeout(ConnectionError, Timeout):
"""The request timed out while trying to connect to the remote server.
Requests that produced this error are safe to retry.
"""
class ReadTimeout(Timeout):
"""The server did not send any data in the allotted amount of time."""
class URLRequired(RequestException):
"""A valid URL is required to make a request."""
class TooManyRedirects(RequestException):
"""Too many redirects."""
class MissingSchema(RequestException, ValueError):
"""The URL scheme (e.g. http or https) is missing."""
class InvalidSchema(RequestException, ValueError):
"""The URL scheme provided is either invalid or unsupported."""
class InvalidURL(RequestException, ValueError):
"""The URL provided was somehow invalid."""
class InvalidHeader(RequestException, ValueError):
"""The header value provided was somehow invalid."""
class InvalidProxyURL(InvalidURL):
"""The proxy URL provided is invalid."""
class ChunkedEncodingError(RequestException):
"""The server declared chunked encoding but sent an invalid chunk."""
class ContentDecodingError(RequestException, BaseHTTPError):
"""Failed to decode response content."""
class StreamConsumedError(RequestException, TypeError):
"""The content for this response was already consumed."""
class RetryError(RequestException):
"""Custom retries logic failed"""
class UnrewindableBodyError(RequestException):
"""Requests encountered an error when trying to rewind a body."""
# Warnings
class RequestsWarning(Warning):
"""Base warning for Requests."""
class FileModeWarning(RequestsWarning, DeprecationWarning):
"""A file was opened in text mode, but Requests determined its binary length."""
class RequestsDependencyWarning(RequestsWarning):
"""An imported dependency doesn't match the expected version range."""
@@ -0,0 +1,134 @@
"""Module containing bug report helper(s)."""
# pyright: reportUnknownMemberType=false
import json
import platform
import ssl
import sys
from typing import Any
import idna
import urllib3
from . import __version__ as requests_version
try:
import charset_normalizer
except ImportError:
charset_normalizer = None
try:
import chardet # type: ignore[import-not-found]
except ImportError:
chardet = None
try:
from urllib3.contrib import pyopenssl
except ImportError:
pyopenssl = None
OpenSSL = None
cryptography = None
else:
import cryptography # type: ignore[import-not-found]
import OpenSSL # type: ignore[import-not-found]
def _implementation():
"""Return a dict with the Python implementation and version.
Provide both the name and the version of the Python implementation
currently running. For example, on CPython 3.10.3 it will return
{'name': 'CPython', 'version': '3.10.3'}.
This function works best on CPython and PyPy: in particular, it probably
doesn't work for Jython or IronPython. Future investigation should be done
to work out the correct shape of the code for those platforms.
"""
implementation = platform.python_implementation()
if implementation == "CPython":
implementation_version = platform.python_version()
elif implementation == "PyPy":
pypy = sys.pypy_version_info # type: ignore[attr-defined]
implementation_version = f"{pypy.major}.{pypy.minor}.{pypy.micro}"
if sys.pypy_version_info.releaselevel != "final": # type: ignore[attr-defined]
implementation_version = "".join(
[implementation_version, sys.pypy_version_info.releaselevel] # type: ignore[attr-defined]
)
elif implementation == "Jython":
implementation_version = platform.python_version() # Complete Guess
elif implementation == "IronPython":
implementation_version = platform.python_version() # Complete Guess
else:
implementation_version = "Unknown"
return {"name": implementation, "version": implementation_version}
def info() -> dict[str, Any]:
"""Generate information for a bug report."""
try:
platform_info = {
"system": platform.system(),
"release": platform.release(),
}
except OSError:
platform_info = {
"system": "Unknown",
"release": "Unknown",
}
implementation_info = _implementation()
urllib3_info = {"version": urllib3.__version__} # type: ignore[reportPrivateImportUsage]
charset_normalizer_info = {"version": None}
chardet_info: dict[str, str | None] = {"version": None}
if charset_normalizer:
charset_normalizer_info = {"version": charset_normalizer.__version__}
if chardet:
chardet_info = {"version": chardet.__version__}
pyopenssl_info: dict[str, str | None] = {
"version": None,
"openssl_version": "",
}
if OpenSSL:
pyopenssl_info = {
"version": OpenSSL.__version__,
"openssl_version": f"{OpenSSL.SSL.OPENSSL_VERSION_NUMBER:x}",
}
cryptography_info = {
"version": getattr(cryptography, "__version__", ""),
}
idna_info = {
"version": getattr(idna, "__version__", ""),
}
system_ssl = ssl.OPENSSL_VERSION_NUMBER
system_ssl_info = {"version": f"{system_ssl:x}" if system_ssl is not None else ""} # type: ignore[reportUnnecessaryComparison]
return {
"platform": platform_info,
"implementation": implementation_info,
"system_ssl": system_ssl_info,
"using_pyopenssl": pyopenssl is not None,
"using_charset_normalizer": chardet is None,
"pyOpenSSL": pyopenssl_info,
"urllib3": urllib3_info,
"chardet": chardet_info,
"charset_normalizer": charset_normalizer_info,
"cryptography": cryptography_info,
"idna": idna_info,
"requests": {
"version": requests_version,
},
}
def main():
"""Pretty-print the bug information as JSON."""
print(json.dumps(info(), sort_keys=True, indent=2))
if __name__ == "__main__":
main()
@@ -0,0 +1,48 @@
"""
requests.hooks
~~~~~~~~~~~~~~
This module provides the capabilities for the Requests hooks system.
Available hooks:
``response``:
The response generated from a Request.
"""
from __future__ import annotations
from collections.abc import Callable, Iterable
from typing import TYPE_CHECKING, Any
if TYPE_CHECKING:
from . import _types as _t
from .models import Response
HOOKS: list[str] = ["response"]
def default_hooks() -> dict[str, list[_t.HookType]]:
return {event: [] for event in HOOKS}
# TODO: response is the only one
def dispatch_hook(
key: str,
hooks: _t.HooksInputType | None,
hook_data: Response,
**kwargs: Any,
) -> Response:
"""Dispatches a hook dictionary on a given piece of data."""
hooks_dict = hooks or {}
hook_list: Iterable[_t.HookType] | _t.HookType | None = hooks_dict.get(key)
if hook_list:
if isinstance(hook_list, Callable):
hook_list = [hook_list]
for hook in hook_list:
_hook_data = hook(hook_data, **kwargs)
if _hook_data is not None:
hook_data = _hook_data
return hook_data
@@ -0,0 +1,23 @@
import sys
from .compat import chardet
# This code exists for backwards compatibility reasons.
# I don't like it either. Just look the other way. :)
for package in ("urllib3", "idna"):
locals()[package] = __import__(package)
# This traversal is apparently necessary such that the identities are
# preserved (requests.packages.urllib3.* is urllib3.*)
for mod in list(sys.modules):
if mod == package or mod.startswith(f"{package}."):
sys.modules[f"requests.packages.{mod}"] = sys.modules[mod]
if chardet is not None:
target = chardet.__name__
for mod in list(sys.modules):
if mod == target or mod.startswith(f"{target}."):
imported_mod = sys.modules[mod]
sys.modules[f"requests.packages.{mod}"] = imported_mod
mod = mod.replace(target, "chardet")
sys.modules[f"requests.packages.{mod}"] = imported_mod
@@ -0,0 +1,920 @@
"""
requests.sessions
~~~~~~~~~~~~~~~~~
This module provides a Session object to manage and persist settings across
requests (cookies, auth, proxies).
"""
from __future__ import annotations
import os
import sys
import time
from collections import OrderedDict
from collections.abc import Generator, Mapping, MutableMapping
from datetime import timedelta
from typing import TYPE_CHECKING, Any, cast
from ._internal_utils import to_native_string
from ._types import is_prepared as _is_prepared
from .adapters import HTTPAdapter
from .auth import _basic_auth_str # type: ignore[reportPrivateUsage]
from .compat import cookielib, urljoin, urlparse
from .cookies import (
RequestsCookieJar,
cookiejar_from_dict,
extract_cookies_to_jar,
merge_cookies,
)
from .exceptions import (
ChunkedEncodingError,
ContentDecodingError,
InvalidSchema,
TooManyRedirects,
)
from .hooks import default_hooks, dispatch_hook
# formerly defined here, reexposed here for backward compatibility
from .models import ( # noqa: F401
DEFAULT_REDIRECT_LIMIT,
REDIRECT_STATI, # type: ignore[reportUnusedImport]
PreparedRequest,
Request,
Response,
)
from .status_codes import codes
from .structures import CaseInsensitiveDict
from .utils import ( # noqa: F401
DEFAULT_PORTS,
default_headers,
get_auth_from_url,
get_environ_proxies,
get_netrc_auth,
requote_uri,
resolve_proxies,
rewind_body,
should_bypass_proxies, # type: ignore[reportUnusedImport] # re-export for external consumers
to_key_val_list,
)
if TYPE_CHECKING:
from http.cookiejar import CookieJar
from typing_extensions import Self, Unpack
from . import _types as _t
from .adapters import BaseAdapter
# Preferred clock, based on which one is more accurate on a given system.
if sys.platform == "win32":
preferred_clock = time.perf_counter
else:
preferred_clock = time.time
def merge_setting(
request_setting: Any, session_setting: Any, dict_class: type = OrderedDict
) -> Any:
"""Determines appropriate setting for a given request, taking into account
the explicit setting on that request, and the setting in the session. If a
setting is a dictionary, they will be merged together using `dict_class`
"""
if session_setting is None:
return request_setting
if request_setting is None:
return session_setting
# Bypass if not a dictionary (e.g. verify)
if not (
isinstance(session_setting, Mapping) and isinstance(request_setting, Mapping)
):
return request_setting
merged_setting = dict_class(to_key_val_list(session_setting)) # type: ignore[arg-type] # isinstance narrows Any to Mapping[Unknown]
merged_setting.update(to_key_val_list(request_setting)) # type: ignore[arg-type]
# Remove keys that are set to None. Extract keys first to avoid altering
# the dictionary during iteration.
none_keys = [k for (k, v) in merged_setting.items() if v is None]
for key in none_keys:
del merged_setting[key]
return merged_setting
def merge_hooks(
request_hooks: _t.HooksType,
session_hooks: _t.HooksType,
dict_class: type = OrderedDict,
) -> _t.HooksType:
"""Properly merges both requests and session hooks.
This is necessary because when request_hooks == {'response': []}, the
merge breaks Session hooks entirely.
"""
if session_hooks is None or session_hooks.get("response") == []:
return request_hooks
if request_hooks is None or request_hooks.get("response") == []:
return session_hooks
return merge_setting(request_hooks, session_hooks, dict_class)
class SessionRedirectMixin:
max_redirects: int
trust_env: bool
cookies: RequestsCookieJar
def send(self, request: PreparedRequest, **kwargs: Any) -> Response: ...
def get_redirect_target(self, resp: Response) -> str | None:
"""Receives a Response. Returns a redirect URI or ``None``"""
# Due to the nature of how requests processes redirects this method will
# be called at least once upon the original response and at least twice
# on each subsequent redirect response (if any).
# If a custom mixin is used to handle this logic, it may be advantageous
# to cache the redirect location onto the response object as a private
# attribute.
if resp.is_redirect:
location = resp.headers["location"]
# Currently the underlying http module on py3 decode headers
# in latin1, but empirical evidence suggests that latin1 is very
# rarely used with non-ASCII characters in HTTP headers.
# It is more likely to get UTF8 header rather than latin1.
# This causes incorrect handling of UTF8 encoded location headers.
# To solve this, we re-encode the location in latin1.
location = location.encode("latin1")
return to_native_string(location, "utf8")
return None
def should_strip_auth(self, old_url: str, new_url: str) -> bool:
"""Decide whether Authorization header should be removed when redirecting"""
old_parsed = urlparse(old_url)
new_parsed = urlparse(new_url)
if old_parsed.hostname != new_parsed.hostname:
return True
# Special case: allow http -> https redirect when using the standard
# ports. This isn't specified by RFC 7235, but is kept to avoid
# breaking backwards compatibility with older versions of requests
# that allowed any redirects on the same host.
if (
old_parsed.scheme == "http"
and old_parsed.port in (80, None)
and new_parsed.scheme == "https"
and new_parsed.port in (443, None)
):
return False
# Handle default port usage corresponding to scheme.
changed_port = old_parsed.port != new_parsed.port
changed_scheme = old_parsed.scheme != new_parsed.scheme
default_port = (DEFAULT_PORTS.get(old_parsed.scheme, None), None)
if (
not changed_scheme
and old_parsed.port in default_port
and new_parsed.port in default_port
):
return False
# Standard case: root URI must match
return changed_port or changed_scheme
def resolve_redirects(
self,
resp: Response,
req: PreparedRequest,
stream: bool = False,
timeout: _t.TimeoutType = None,
verify: _t.VerifyType = True,
cert: _t.CertType = None,
proxies: dict[str, str] | None = None,
yield_requests: bool = False,
**adapter_kwargs: Any,
) -> Generator[Response, None, None]:
"""Receives a Response. Returns a generator of Responses or Requests."""
hist: list[Response] = [] # keep track of history
url = self.get_redirect_target(resp)
previous_fragment = urlparse(req.url).fragment
while url:
prepared_request = req.copy()
# Update history and keep track of redirects.
resp.history = hist[:]
hist.append(resp)
try:
resp.content # Consume socket so it can be released
except (ChunkedEncodingError, ContentDecodingError, RuntimeError):
resp.raw.read(decode_content=False)
if len(resp.history) >= self.max_redirects:
raise TooManyRedirects(
f"Exceeded {self.max_redirects} redirects.", response=resp
)
# Release the connection back into the pool.
resp.close()
# Handle redirection without scheme (see: RFC 1808 Section 4)
if url.startswith("//"):
parsed_rurl = urlparse(resp.url)
url = ":".join([to_native_string(parsed_rurl.scheme), url])
# Normalize url case and attach previous fragment if needed (RFC 7231 7.1.2)
parsed = urlparse(url)
if parsed.fragment == "" and previous_fragment:
parsed = parsed._replace(fragment=previous_fragment)
elif parsed.fragment:
previous_fragment = parsed.fragment
url = parsed.geturl()
# Facilitate relative 'location' headers, as allowed by RFC 7231.
# (e.g. '/path/to/resource' instead of 'http://domain.tld/path/to/resource')
# Compliant with RFC3986, we percent encode the url.
if not parsed.netloc:
url = urljoin(resp.url, requote_uri(url))
else:
url = requote_uri(url)
prepared_request.url = to_native_string(url)
self.rebuild_method(prepared_request, resp)
# https://github.com/psf/requests/issues/1084
if resp.status_code not in (
codes.temporary_redirect,
codes.permanent_redirect,
):
# https://github.com/psf/requests/issues/3490
purged_headers = ("Content-Length", "Content-Type", "Transfer-Encoding")
for header in purged_headers:
prepared_request.headers.pop(header, None)
prepared_request.body = None
headers = prepared_request.headers
headers.pop("Cookie", None)
# Extract any cookies sent on the response to the cookiejar
# in the new request. Because we've mutated our copied prepared
# request, use the old one that we haven't yet touched.
cookie_jar = cast("CookieJar", prepared_request._cookies) # type: ignore[reportPrivateUsage]
extract_cookies_to_jar(cookie_jar, req, resp.raw)
merge_cookies(cookie_jar, self.cookies)
prepared_request.prepare_cookies(cookie_jar)
# Rebuild auth and proxy information.
proxies = self.rebuild_proxies(prepared_request, proxies)
self.rebuild_auth(prepared_request, resp)
# A failed tell() sets `_body_position` to `object()`. This non-None
# value ensures `rewindable` will be True, allowing us to raise an
# UnrewindableBodyError, instead of hanging the connection.
rewindable = prepared_request._body_position is not None and ( # type: ignore[reportPrivateUsage]
"Content-Length" in headers or "Transfer-Encoding" in headers
)
# Attempt to rewind consumed file-like object.
if rewindable:
rewind_body(prepared_request)
# Override the original request.
req = prepared_request
if yield_requests:
yield req # type: ignore[misc] # Internal use only, returns PreparedRequest
else:
resp = self.send(
req,
stream=stream,
timeout=timeout,
verify=verify,
cert=cert,
proxies=proxies,
allow_redirects=False,
**adapter_kwargs,
)
extract_cookies_to_jar(self.cookies, prepared_request, resp.raw)
# extract redirect url, if any, for the next loop
url = self.get_redirect_target(resp)
yield resp
def rebuild_auth(
self, prepared_request: PreparedRequest, response: Response
) -> None:
"""When being redirected we may want to strip authentication from the
request to avoid leaking credentials. This method intelligently removes
and reapplies authentication where possible to avoid credential loss.
"""
original_request = response.request
assert _is_prepared(original_request)
assert _is_prepared(prepared_request)
headers = prepared_request.headers
original_url = original_request.url
url = prepared_request.url
if "Authorization" in headers and self.should_strip_auth(original_url, url):
# If we get redirected to a new host, we should strip out any
# authentication headers.
del headers["Authorization"]
# .netrc might have more auth for us on our new host.
new_auth = get_netrc_auth(url) if self.trust_env else None
if new_auth is not None:
prepared_request.prepare_auth(new_auth)
def rebuild_proxies(
self,
prepared_request: PreparedRequest,
proxies: dict[str, str] | None,
) -> dict[str, str]:
"""This method re-evaluates the proxy configuration by considering the
environment variables. If we are redirected to a URL covered by
NO_PROXY, we strip the proxy configuration. Otherwise, we set missing
proxy keys for this URL (in case they were stripped by a previous
redirect).
This method also replaces the Proxy-Authorization header where
necessary.
:rtype: dict
"""
assert _is_prepared(prepared_request)
headers = prepared_request.headers
scheme = urlparse(prepared_request.url).scheme
new_proxies = resolve_proxies(prepared_request, proxies, self.trust_env)
if "Proxy-Authorization" in headers:
del headers["Proxy-Authorization"]
try:
username, password = get_auth_from_url(new_proxies[scheme])
except KeyError:
username, password = None, None
# urllib3 handles proxy authorization for us in the standard adapter.
# Avoid appending this to TLS tunneled requests where it may be leaked.
if not scheme.startswith("https") and username and password:
headers["Proxy-Authorization"] = _basic_auth_str(username, password)
return new_proxies
def rebuild_method(
self, prepared_request: PreparedRequest, response: Response
) -> None:
"""When being redirected we may want to change the method of the request
based on certain specs or browser behavior.
"""
method = prepared_request.method
# https://tools.ietf.org/html/rfc7231#section-6.4.4
if response.status_code == codes.see_other and method != "HEAD":
method = "GET"
# Do what the browsers do, despite standards...
# First, turn 302s into GETs.
if response.status_code == codes.found and method != "HEAD":
method = "GET"
# Second, if a POST is responded to with a 301, turn it into a GET.
# This bizarre behaviour is explained in Issue 1704.
if response.status_code == codes.moved and method == "POST":
method = "GET"
prepared_request.method = method
class Session(SessionRedirectMixin):
"""A Requests session.
Provides cookie persistence, connection-pooling, and configuration.
Basic Usage::
>>> import requests
>>> s = requests.Session()
>>> s.get('https://httpbin.org/get')
<Response [200]>
Or as a context manager::
>>> with requests.Session() as s:
... s.get('https://httpbin.org/get')
<Response [200]>
"""
headers: CaseInsensitiveDict[str]
auth: _t.AuthType
proxies: dict[str, str]
hooks: dict[str, list[_t.HookType]]
params: MutableMapping[str, Any]
stream: bool
verify: _t.VerifyType
cert: _t.CertType
max_redirects: int
trust_env: bool
cookies: RequestsCookieJar
adapters: MutableMapping[str, BaseAdapter]
__attrs__: list[str] = [
"headers",
"cookies",
"auth",
"proxies",
"hooks",
"params",
"verify",
"cert",
"adapters",
"stream",
"trust_env",
"max_redirects",
]
def __init__(self) -> None:
#: A case-insensitive dictionary of headers to be sent on each
#: :class:`Request <Request>` sent from this
#: :class:`Session <Session>`.
self.headers = default_headers()
#: Default Authentication tuple or object to attach to
#: :class:`Request <Request>`.
self.auth = None
#: Dictionary mapping protocol or protocol and host to the URL of the proxy
#: (e.g. {'http': 'foo.bar:3128', 'http://host.name': 'foo.bar:4012'}) to
#: be used on each :class:`Request <Request>`.
self.proxies = {}
#: Event-handling hooks.
self.hooks = default_hooks()
#: Dictionary of querystring data to attach to each
#: :class:`Request <Request>`. The dictionary values may be lists for
#: representing multivalued query parameters.
self.params = {}
#: Stream response content default.
self.stream = False
#: SSL Verification default.
#: Defaults to `True`, requiring requests to verify the TLS certificate at the
#: remote end.
#: If verify is set to `False`, requests will accept any TLS certificate
#: presented by the server, and will ignore hostname mismatches and/or
#: expired certificates, which will make your application vulnerable to
#: man-in-the-middle (MitM) attacks.
#: Only set this to `False` for testing.
#: If verify is set to a string, it must be the path to a CA bundle file
#: that will be used to verify the TLS certificate.
self.verify = True
#: SSL client certificate default, if String, path to ssl client
#: cert file (.pem). If Tuple, ('cert', 'key') pair.
self.cert = None
#: Maximum number of redirects allowed. If the request exceeds this
#: limit, a :class:`TooManyRedirects` exception is raised.
#: This defaults to requests.models.DEFAULT_REDIRECT_LIMIT, which is
#: 30.
self.max_redirects = DEFAULT_REDIRECT_LIMIT
#: Trust environment settings for proxy configuration, default
#: authentication and similar.
self.trust_env = True
#: A CookieJar containing all currently outstanding cookies set on this
#: session. By default it is a
#: :class:`RequestsCookieJar <requests.cookies.RequestsCookieJar>`, but
#: may be any other ``cookielib.CookieJar`` compatible object.
self.cookies = cookiejar_from_dict({})
# Default connection adapters.
self.adapters = OrderedDict()
self.mount("https://", HTTPAdapter())
self.mount("http://", HTTPAdapter())
def __enter__(self) -> Self:
return self
def __exit__(self, *args: Any) -> None:
self.close()
def prepare_request(self, request: Request) -> PreparedRequest:
"""Constructs a :class:`PreparedRequest <PreparedRequest>` for
transmission and returns it. The :class:`PreparedRequest` has settings
merged from the :class:`Request <Request>` instance and those of the
:class:`Session`.
:param request: :class:`Request` instance to prepare with this
session's settings.
:rtype: requests.PreparedRequest
"""
url = cast("_t.UriType", request.url)
method = cast(str, request.method)
cookies = request.cookies or {}
# Bootstrap CookieJar.
if not isinstance(cookies, cookielib.CookieJar):
cookies = cookiejar_from_dict(cookies)
# Merge with session cookies
merged_cookies = merge_cookies(
merge_cookies(RequestsCookieJar(), self.cookies), cookies
)
# Set environment's basic authentication if not explicitly set.
auth = request.auth
if self.trust_env and not auth and not self.auth:
auth = get_netrc_auth(url)
p = PreparedRequest()
p.prepare(
method=method.upper(),
url=url,
files=request.files,
data=request.data,
json=request.json,
headers=merge_setting(
request.headers, self.headers, dict_class=CaseInsensitiveDict
),
params=merge_setting(request.params, self.params),
auth=merge_setting(auth, self.auth),
cookies=merged_cookies,
hooks=merge_hooks(request.hooks, self.hooks),
)
return p
def request(
self,
method: str,
url: _t.UriType,
params: _t.ParamsType = None,
data: _t.DataType = None,
headers: _t.HeadersType = None,
cookies: RequestsCookieJar | CookieJar | dict[str, str] | None = None,
files: _t.FilesType = None,
auth: _t.AuthType = None,
timeout: _t.TimeoutType = None,
allow_redirects: bool = True,
proxies: dict[str, str] | None = None,
hooks: _t.HooksInputType | None = None,
stream: bool | None = None,
verify: _t.VerifyType | None = None,
cert: _t.CertType = None,
json: _t.JsonType = None,
) -> Response:
"""Constructs a :class:`Request <Request>`, prepares it and sends it.
Returns :class:`Response <Response>` object.
:param method: method for the new :class:`Request` object.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary or bytes to be sent in the query
string for the :class:`Request`.
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
object to send in the body of the :class:`Request`.
:param json: (optional) json to send in the body of the
:class:`Request`.
:param headers: (optional) Dictionary of HTTP Headers to send with the
:class:`Request`.
:param cookies: (optional) Dict or CookieJar object to send with the
:class:`Request`.
:param files: (optional) Dictionary of ``'filename': file-like-objects``
for multipart encoding upload.
:param auth: (optional) Auth tuple or callable to enable
Basic/Digest/Custom HTTP Auth.
:param timeout: (optional) How many seconds to wait for the server to send
data before giving up, as a float, or a :ref:`(connect timeout,
read timeout) <timeouts>` tuple.
:type timeout: float or tuple
:param allow_redirects: (optional) Set to True by default.
:type allow_redirects: bool
:param proxies: (optional) Dictionary mapping protocol or protocol and
hostname to the URL of the proxy.
:param hooks: (optional) Dictionary mapping hook name to one event or
list of events, event must be callable.
:param stream: (optional) whether to immediately download the response
content. Defaults to ``False``.
:param verify: (optional) Either a boolean, in which case it controls whether we verify
the server's TLS certificate, or a string, in which case it must be a path
to a CA bundle to use. Defaults to ``True``. When set to
``False``, requests will accept any TLS certificate presented by
the server, and will ignore hostname mismatches and/or expired
certificates, which will make your application vulnerable to
man-in-the-middle (MitM) attacks. Setting verify to ``False``
may be useful during local development or testing.
:param cert: (optional) if String, path to ssl client cert file (.pem).
If Tuple, ('cert', 'key') pair.
:rtype: requests.Response
"""
if isinstance(url, bytes):
url = url.decode("utf-8")
# Create the Request.
req = Request(
method=method.upper(),
url=url,
headers=headers,
files=files,
data=data or {},
json=json,
params=params or {},
auth=auth,
cookies=cookies,
hooks=hooks,
)
prep = self.prepare_request(req)
assert _is_prepared(prep)
proxies = proxies or {}
settings = self.merge_environment_settings(
prep.url, proxies, stream, verify, cert
)
# Send the request.
send_kwargs = {
"timeout": timeout,
"allow_redirects": allow_redirects,
}
send_kwargs.update(settings)
resp = self.send(prep, **send_kwargs)
return resp
def get(
self,
url: _t.UriType,
params: _t.ParamsType = None,
**kwargs: Unpack[_t.GetKwargs],
) -> Response:
r"""Sends a GET request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param params: (optional) Dictionary, list of tuples or bytes to send
in the query string for the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:rtype: requests.Response
"""
kwargs.setdefault("allow_redirects", True)
return self.request("GET", url, params=params, **kwargs)
def options(self, url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]) -> Response:
r"""Sends a OPTIONS request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:rtype: requests.Response
"""
kwargs.setdefault("allow_redirects", True)
return self.request("OPTIONS", url, **kwargs)
def head(self, url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]) -> Response:
r"""Sends a HEAD request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:rtype: requests.Response
"""
kwargs.setdefault("allow_redirects", False)
return self.request("HEAD", url, **kwargs)
def post(
self,
url: _t.UriType,
data: _t.DataType = None,
json: _t.JsonType = None,
**kwargs: Unpack[_t.PostKwargs],
) -> Response:
r"""Sends a POST request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
object to send in the body of the :class:`Request`.
:param json: (optional) json to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:rtype: requests.Response
"""
return self.request("POST", url, data=data, json=json, **kwargs)
def put(
self, url: _t.UriType, data: _t.DataType = None, **kwargs: Unpack[_t.DataKwargs]
) -> Response:
r"""Sends a PUT request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
object to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:rtype: requests.Response
"""
return self.request("PUT", url, data=data, **kwargs)
def patch(
self, url: _t.UriType, data: _t.DataType = None, **kwargs: Unpack[_t.DataKwargs]
) -> Response:
r"""Sends a PATCH request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param data: (optional) Dictionary, list of tuples, bytes, or file-like
object to send in the body of the :class:`Request`.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:rtype: requests.Response
"""
return self.request("PATCH", url, data=data, **kwargs)
def delete(self, url: _t.UriType, **kwargs: Unpack[_t.RequestKwargs]) -> Response:
r"""Sends a DELETE request. Returns :class:`Response` object.
:param url: URL for the new :class:`Request` object.
:param \*\*kwargs: Optional arguments that ``request`` takes.
:rtype: requests.Response
"""
return self.request("DELETE", url, **kwargs)
def send(self, request: PreparedRequest, **kwargs: Any) -> Response:
"""Send a given PreparedRequest.
:rtype: requests.Response
"""
# Set defaults that the hooks can utilize to ensure they always have
# the correct parameters to reproduce the previous request.
kwargs.setdefault("stream", self.stream)
kwargs.setdefault("verify", self.verify)
kwargs.setdefault("cert", self.cert)
if "proxies" not in kwargs:
kwargs["proxies"] = resolve_proxies(request, self.proxies, self.trust_env)
# It's possible that users might accidentally send a Request object.
# Guard against that specific failure case.
if isinstance(request, Request):
raise ValueError("You can only send PreparedRequests.")
assert _is_prepared(request)
# Set up variables needed for resolve_redirects and dispatching of hooks
allow_redirects = kwargs.pop("allow_redirects", True)
stream = kwargs.get("stream")
hooks = request.hooks
# Get the appropriate adapter to use
adapter = self.get_adapter(url=request.url)
# Start time (approximately) of the request
start = preferred_clock()
# Send the request
r = adapter.send(request, **kwargs)
# Total elapsed time of the request (approximately)
elapsed = preferred_clock() - start
r.elapsed = timedelta(seconds=elapsed)
# Response manipulation hooks
r = dispatch_hook("response", hooks, r, **kwargs)
# Persist cookies
if r.history:
# If the hooks create history then we want those cookies too
for resp in r.history:
extract_cookies_to_jar(self.cookies, resp.request, resp.raw)
extract_cookies_to_jar(self.cookies, request, r.raw)
# Resolve redirects if allowed.
if allow_redirects:
# Redirect resolving generator.
gen = self.resolve_redirects(r, request, **kwargs)
history = [resp for resp in gen]
else:
history = []
# Shuffle things around if there's history.
if history:
# Insert the first (original) request at the start
history.insert(0, r)
# Get the last request made
r = history.pop()
r.history = history
# If redirects aren't being followed, store the response on the Request for Response.next().
if not allow_redirects:
try:
r._next = next( # type: ignore[assignment] # yield_requests=True returns PreparedRequest
self.resolve_redirects(r, request, yield_requests=True, **kwargs)
)
except StopIteration:
pass
if not stream:
r.content
return r
def merge_environment_settings(
self,
url: str,
proxies: dict[str, str] | None,
stream: bool | None,
verify: _t.VerifyType | None,
cert: _t.CertType,
) -> dict[str, Any]:
"""
Check the environment and merge it with some settings.
:rtype: dict
"""
# Gather clues from the surrounding environment.
if self.trust_env:
# Set environment's proxies.
no_proxy = proxies.get("no_proxy") if proxies is not None else None
env_proxies = get_environ_proxies(url, no_proxy=no_proxy)
if proxies is not None:
for k, v in env_proxies.items():
proxies.setdefault(k, v)
# Look for requests environment configuration
# and be compatible with cURL.
if verify is True or verify is None:
verify = (
os.environ.get("REQUESTS_CA_BUNDLE")
or os.environ.get("CURL_CA_BUNDLE")
or verify
)
# Merge all the kwargs.
proxies = merge_setting(proxies, self.proxies)
stream = merge_setting(stream, self.stream)
verify = merge_setting(verify, self.verify)
cert = merge_setting(cert, self.cert)
return {"proxies": proxies, "stream": stream, "verify": verify, "cert": cert}
def get_adapter(self, url: str) -> BaseAdapter:
"""
Returns the appropriate connection adapter for the given URL.
:rtype: requests.adapters.BaseAdapter
"""
for prefix, adapter in self.adapters.items():
if url.lower().startswith(prefix.lower()):
return adapter
# Nothing matches :-/
raise InvalidSchema(f"No connection adapters were found for {url!r}")
def close(self) -> None:
"""Closes all adapters and as such the session"""
for v in self.adapters.values():
v.close()
def mount(self, prefix: str, adapter: BaseAdapter) -> None:
"""Registers a connection adapter to a prefix.
Adapters are sorted in descending order by prefix length.
"""
self.adapters[prefix] = adapter
keys_to_move = [k for k in self.adapters if len(k) < len(prefix)]
for key in keys_to_move:
self.adapters[key] = self.adapters.pop(key)
def __getstate__(self) -> dict[str, Any]:
state = {attr: getattr(self, attr, None) for attr in self.__attrs__}
return state
def __setstate__(self, state: dict[str, Any]) -> None:
for attr, value in state.items():
setattr(self, attr, value)
def session() -> Session:
"""
Returns a :class:`Session` for context-management.
.. deprecated:: 1.0.0
This method has been deprecated since version 1.0.0 and is only kept for
backwards compatibility. New code should use :class:`~requests.sessions.Session`
to create a session. This may be removed at a future date.
:rtype: Session
"""
return Session()
@@ -0,0 +1,128 @@
r"""
The ``codes`` object defines a mapping from common names for HTTP statuses
to their numerical codes, accessible either as attributes or as dictionary
items.
Example::
>>> import requests
>>> requests.codes['temporary_redirect']
307
>>> requests.codes.teapot
418
>>> requests.codes['\o/']
200
Some codes have multiple names, and both upper- and lower-case versions of
the names are allowed. For example, ``codes.ok``, ``codes.OK``, and
``codes.okay`` all correspond to the HTTP status code 200.
"""
from .structures import LookupDict
_codes = {
# Informational.
100: ("continue",),
101: ("switching_protocols",),
102: ("processing", "early-hints"),
103: ("checkpoint",),
122: ("uri_too_long", "request_uri_too_long"),
200: ("ok", "okay", "all_ok", "all_okay", "all_good", "\\o/", "✓"),
201: ("created",),
202: ("accepted",),
203: ("non_authoritative_info", "non_authoritative_information"),
204: ("no_content",),
205: ("reset_content", "reset"),
206: ("partial_content", "partial"),
207: ("multi_status", "multiple_status", "multi_stati", "multiple_stati"),
208: ("already_reported",),
226: ("im_used",),
# Redirection.
300: ("multiple_choices",),
301: ("moved_permanently", "moved", "\\o-"),
302: ("found",),
303: ("see_other", "other"),
304: ("not_modified",),
305: ("use_proxy",),
306: ("switch_proxy",),
307: ("temporary_redirect", "temporary_moved", "temporary"),
308: (
"permanent_redirect",
"resume_incomplete",
"resume",
), # "resume" and "resume_incomplete" to be removed in 3.0
# Client Error.
400: ("bad_request", "bad"),
401: ("unauthorized",),
402: ("payment_required", "payment"),
403: ("forbidden",),
404: ("not_found", "-o-"),
405: ("method_not_allowed", "not_allowed"),
406: ("not_acceptable",),
407: ("proxy_authentication_required", "proxy_auth", "proxy_authentication"),
408: ("request_timeout", "timeout"),
409: ("conflict",),
410: ("gone",),
411: ("length_required",),
412: ("precondition_failed", "precondition"),
413: ("request_entity_too_large", "content_too_large"),
414: ("request_uri_too_large", "uri_too_long"),
415: ("unsupported_media_type", "unsupported_media", "media_type"),
416: (
"requested_range_not_satisfiable",
"requested_range",
"range_not_satisfiable",
),
417: ("expectation_failed",),
418: ("im_a_teapot", "teapot", "i_am_a_teapot"),
421: ("misdirected_request",),
422: ("unprocessable_entity", "unprocessable", "unprocessable_content"),
423: ("locked",),
424: ("failed_dependency", "dependency"),
425: ("unordered_collection", "unordered", "too_early"),
426: ("upgrade_required", "upgrade"),
428: ("precondition_required", "precondition"),
429: ("too_many_requests", "too_many"),
431: ("header_fields_too_large", "fields_too_large"),
444: ("no_response", "none"),
449: ("retry_with", "retry"),
450: ("blocked_by_windows_parental_controls", "parental_controls"),
451: ("unavailable_for_legal_reasons", "legal_reasons"),
499: ("client_closed_request",),
# Server Error.
500: ("internal_server_error", "server_error", "/o\\", "✗"),
501: ("not_implemented",),
502: ("bad_gateway",),
503: ("service_unavailable", "unavailable"),
504: ("gateway_timeout",),
505: ("http_version_not_supported", "http_version"),
506: ("variant_also_negotiates",),
507: ("insufficient_storage",),
509: ("bandwidth_limit_exceeded", "bandwidth"),
510: ("not_extended",),
511: ("network_authentication_required", "network_auth", "network_authentication"),
}
codes: LookupDict[int] = LookupDict(name="status_codes")
def _init():
for code, titles in _codes.items():
for title in titles:
setattr(codes, title, code)
if not title.startswith(("\\", "/")):
setattr(codes, title.upper(), code)
def doc(code: int) -> str:
names = ", ".join(f"``{n}``" for n in _codes[code])
return "* %d: %s" % (code, names)
global __doc__
__doc__ = (
__doc__ + "\n" + "\n".join(doc(code) for code in sorted(_codes))
if __doc__ is not None
else None
)
_init()
@@ -0,0 +1,130 @@
"""
requests.structures
~~~~~~~~~~~~~~~~~~~
Data structures that power Requests.
"""
from __future__ import annotations
from collections import OrderedDict
from collections.abc import Iterable, Iterator, Mapping
from typing import Any, Generic, TypeVar, overload
from .compat import MutableMapping
_VT = TypeVar("_VT")
_D = TypeVar("_D")
class CaseInsensitiveDict(MutableMapping[str, _VT], Generic[_VT]):
"""A case-insensitive ``dict``-like object.
Implements all methods and operations of
``MutableMapping`` as well as dict's ``copy``. Also
provides ``lower_items``.
All keys are expected to be strings. The structure remembers the
case of the last key to be set, and ``iter(instance)``,
``keys()``, ``items()``, ``iterkeys()``, and ``iteritems()``
will contain case-sensitive keys. However, querying and contains
testing is case insensitive::
cid = CaseInsensitiveDict()
cid['Accept'] = 'application/json'
cid['aCCEPT'] == 'application/json' # True
list(cid) == ['Accept'] # True
For example, ``headers['content-encoding']`` will return the
value of a ``'Content-Encoding'`` response header, regardless
of how the header name was originally stored.
If the constructor, ``.update``, or equality comparison
operations are given keys that have equal ``.lower()``s, the
behavior is undefined.
"""
_store: OrderedDict[str, tuple[str, _VT]]
def __init__(
self,
data: Mapping[str, _VT] | Iterable[tuple[str, _VT]] | None = None,
**kwargs: _VT,
) -> None:
self._store = OrderedDict()
if data is None:
data = {}
self.update(data, **kwargs)
def __setitem__(self, key: str, value: _VT) -> None:
# Use the lowercased key for lookups, but store the actual
# key alongside the value.
self._store[key.lower()] = (key, value)
def __getitem__(self, key: str) -> _VT:
return self._store[key.lower()][1]
def __delitem__(self, key: str) -> None:
del self._store[key.lower()]
def __iter__(self) -> Iterator[str]:
return (casedkey for casedkey, _ in self._store.values())
def __len__(self) -> int:
return len(self._store)
def lower_items(self) -> Iterator[tuple[str, _VT]]:
"""Like iteritems(), but with all lowercase keys."""
return ((lowerkey, keyval[1]) for (lowerkey, keyval) in self._store.items())
def __eq__(self, other: object) -> bool:
if isinstance(other, Mapping):
other_dict: CaseInsensitiveDict[Any] = CaseInsensitiveDict(other) # type: ignore[reportUnknownArgumentType]
else:
return NotImplemented
# Compare insensitively
return dict(self.lower_items()) == dict(other_dict.lower_items())
# Copy is required
def copy(self) -> CaseInsensitiveDict[_VT]:
return CaseInsensitiveDict(self._store.values())
def __repr__(self) -> str:
return str(dict(self.items()))
class LookupDict(dict[str, _VT]):
"""Dictionary lookup object."""
name: Any
def __init__(self, name: Any = None) -> None:
self.name = name
super().__init__()
def __repr__(self) -> str:
return f"<lookup '{self.name}'>"
def __getattr__(self, key: str) -> _VT | None:
# We need this for type checkers to infer typing
# on attribute access with status_codes.py
if key in self.__dict__:
return self.__dict__[key]
else:
raise AttributeError(
f"'{type(self).__name__}' object has no attribute '{key}'"
)
def __getitem__(self, key: str) -> _VT | None: # type: ignore[override]
# We allow fall-through here, so values default to None
return self.__dict__.get(key, None)
@overload
def get(self, key: str, default: None = None) -> _VT | None: ...
@overload
def get(self, key: str, default: _D | _VT) -> _D | _VT: ...
def get(self, key: str, default: _D | None = None) -> _VT | _D | None:
return self.__dict__.get(key, default)
@@ -0,0 +1,211 @@
"""
Python HTTP library with thread-safe connection pooling, file post support, user friendly, and more
"""
from __future__ import annotations
# Set default logging handler to avoid "No handler found" warnings.
import logging
import sys
import typing
import warnings
from logging import NullHandler
from . import exceptions
from ._base_connection import _TYPE_BODY
from ._collections import HTTPHeaderDict
from ._version import __version__
from .connectionpool import HTTPConnectionPool, HTTPSConnectionPool, connection_from_url
from .filepost import _TYPE_FIELDS, encode_multipart_formdata
from .poolmanager import PoolManager, ProxyManager, proxy_from_url
from .response import BaseHTTPResponse, HTTPResponse
from .util.request import make_headers
from .util.retry import Retry
from .util.timeout import Timeout
# Ensure that Python is compiled with OpenSSL 1.1.1+
# If the 'ssl' module isn't available at all that's
# fine, we only care if the module is available.
try:
import ssl
except ImportError:
pass
else:
if not ssl.OPENSSL_VERSION.startswith("OpenSSL "): # Defensive:
warnings.warn(
"urllib3 v2 only supports OpenSSL 1.1.1+, currently "
f"the 'ssl' module is compiled with {ssl.OPENSSL_VERSION!r}. "
"See: https://github.com/urllib3/urllib3/issues/3020",
exceptions.NotOpenSSLWarning,
)
elif ssl.OPENSSL_VERSION_INFO < (1, 1, 1): # Defensive:
raise ImportError(
"urllib3 v2 only supports OpenSSL 1.1.1+, currently "
f"the 'ssl' module is compiled with {ssl.OPENSSL_VERSION!r}. "
"See: https://github.com/urllib3/urllib3/issues/2168"
)
__author__ = "Andrey Petrov ([email protected])"
__license__ = "MIT"
__version__ = __version__
__all__ = (
"HTTPConnectionPool",
"HTTPHeaderDict",
"HTTPSConnectionPool",
"PoolManager",
"ProxyManager",
"HTTPResponse",
"Retry",
"Timeout",
"add_stderr_logger",
"connection_from_url",
"disable_warnings",
"encode_multipart_formdata",
"make_headers",
"proxy_from_url",
"request",
"BaseHTTPResponse",
)
logging.getLogger(__name__).addHandler(NullHandler())
def add_stderr_logger(
level: int = logging.DEBUG,
) -> logging.StreamHandler[typing.TextIO]:
"""
Helper for quickly adding a StreamHandler to the logger. Useful for
debugging.
Returns the handler after adding it.
"""
# This method needs to be in this __init__.py to get the __name__ correct
# even if urllib3 is vendored within another package.
logger = logging.getLogger(__name__)
handler = logging.StreamHandler()
handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(message)s"))
logger.addHandler(handler)
logger.setLevel(level)
logger.debug("Added a stderr logging handler to logger: %s", __name__)
return handler
# ... Clean up.
del NullHandler
# All warning filters *must* be appended unless you're really certain that they
# shouldn't be: otherwise, it's very hard for users to use most Python
# mechanisms to silence them.
# SecurityWarning's always go off by default.
warnings.simplefilter("always", exceptions.SecurityWarning, append=True)
# InsecurePlatformWarning's don't vary between requests, so we keep it default.
warnings.simplefilter("default", exceptions.InsecurePlatformWarning, append=True)
def disable_warnings(category: type[Warning] = exceptions.HTTPWarning) -> None:
"""
Helper for quickly disabling all urllib3 warnings.
"""
warnings.simplefilter("ignore", category)
_DEFAULT_POOL = PoolManager()
def request(
method: str,
url: str,
*,
body: _TYPE_BODY | None = None,
fields: _TYPE_FIELDS | None = None,
headers: typing.Mapping[str, str] | None = None,
preload_content: bool | None = True,
decode_content: bool | None = True,
redirect: bool | None = True,
retries: Retry | bool | int | None = None,
timeout: Timeout | float | int | None = 3,
json: typing.Any | None = None,
) -> BaseHTTPResponse:
"""
A convenience, top-level request method. It uses a module-global ``PoolManager`` instance.
Therefore, its side effects could be shared across dependencies relying on it.
To avoid side effects create a new ``PoolManager`` instance and use it instead.
The method does not accept low-level ``**urlopen_kw`` keyword arguments.
:param method:
HTTP request method (such as GET, POST, PUT, etc.)
:param url:
The URL to perform the request on.
:param body:
Data to send in the request body, either :class:`str`, :class:`bytes`,
an iterable of :class:`str`/:class:`bytes`, or a file-like object.
:param fields:
Data to encode and send in the request body.
:param headers:
Dictionary of custom headers to send, such as User-Agent,
If-None-Match, etc.
:param bool preload_content:
If True, the response's body will be preloaded into memory.
:param bool decode_content:
If True, will attempt to decode the body based on the
'content-encoding' header.
:param redirect:
If True, automatically handle redirects (status codes 301, 302,
303, 307, 308). Each redirect counts as a retry. Disabling retries
will disable redirect, too.
:param retries:
Configure the number of retries to allow before raising a
:class:`~urllib3.exceptions.MaxRetryError` exception.
If ``None`` (default) will retry 3 times, see ``Retry.DEFAULT``. Pass a
:class:`~urllib3.util.retry.Retry` object for fine-grained control
over different types of retries.
Pass an integer number to retry connection errors that many times,
but no other types of errors. Pass zero to never retry.
If ``False``, then retries are disabled and any exception is raised
immediately. Also, instead of raising a MaxRetryError on redirects,
the redirect response will be returned.
:type retries: :class:`~urllib3.util.retry.Retry`, False, or an int.
:param timeout:
If specified, overrides the default timeout for this one
request. It may be a float (in seconds) or an instance of
:class:`urllib3.util.Timeout`.
:param json:
Data to encode and send as JSON with UTF-encoded in the request body.
The ``"Content-Type"`` header will be set to ``"application/json"``
unless specified otherwise.
"""
return _DEFAULT_POOL.request(
method,
url,
body=body,
fields=fields,
headers=headers,
preload_content=preload_content,
decode_content=decode_content,
redirect=redirect,
retries=retries,
timeout=timeout,
json=json,
)
if sys.platform == "emscripten":
from .contrib.emscripten import inject_into_urllib3 # noqa: 401
inject_into_urllib3()
@@ -0,0 +1,167 @@
from __future__ import annotations
import typing
from .util.connection import _TYPE_SOCKET_OPTIONS
from .util.timeout import _DEFAULT_TIMEOUT, _TYPE_TIMEOUT
from .util.url import Url
_TYPE_BODY = typing.Union[
bytes, typing.IO[typing.Any], typing.Iterable[bytes | str], str
]
class ProxyConfig(typing.NamedTuple):
ssl_context: ssl.SSLContext | None
use_forwarding_for_https: bool
assert_hostname: None | str | typing.Literal[False]
assert_fingerprint: str | None
class _ResponseOptions(typing.NamedTuple):
# TODO: Remove this in favor of a better
# HTTP request/response lifecycle tracking.
request_method: str
request_url: str
preload_content: bool
decode_content: bool
enforce_content_length: bool
if typing.TYPE_CHECKING:
import ssl
from typing import Protocol
from .response import BaseHTTPResponse
class BaseHTTPConnection(Protocol):
default_port: typing.ClassVar[int]
default_socket_options: typing.ClassVar[_TYPE_SOCKET_OPTIONS]
host: str
port: int
timeout: None | (
float
) # Instance doesn't store _DEFAULT_TIMEOUT, must be resolved.
blocksize: int
source_address: tuple[str, int] | None
socket_options: _TYPE_SOCKET_OPTIONS | None
proxy: Url | None
proxy_config: ProxyConfig | None
is_verified: bool
proxy_is_verified: bool | None
def __init__(
self,
host: str,
port: int | None = None,
*,
timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
source_address: tuple[str, int] | None = None,
blocksize: int = 8192,
socket_options: _TYPE_SOCKET_OPTIONS | None = ...,
proxy: Url | None = None,
proxy_config: ProxyConfig | None = None,
) -> None: ...
def set_tunnel(
self,
host: str,
port: int | None = None,
headers: typing.Mapping[str, str] | None = None,
scheme: str = "http",
) -> None: ...
def connect(self) -> None: ...
def request(
self,
method: str,
url: str,
body: _TYPE_BODY | None = None,
headers: typing.Mapping[str, str] | None = None,
# We know *at least* botocore is depending on the order of the
# first 3 parameters so to be safe we only mark the later ones
# as keyword-only to ensure we have space to extend.
*,
chunked: bool = False,
preload_content: bool = True,
decode_content: bool = True,
enforce_content_length: bool = True,
) -> None: ...
def getresponse(self) -> BaseHTTPResponse: ...
def close(self) -> None: ...
@property
def is_closed(self) -> bool:
"""Whether the connection either is brand new or has been previously closed.
If this property is True then both ``is_connected`` and ``has_connected_to_proxy``
properties must be False.
"""
@property
def is_connected(self) -> bool:
"""Whether the connection is actively connected to any origin (proxy or target)"""
@property
def has_connected_to_proxy(self) -> bool:
"""Whether the connection has successfully connected to its proxy.
This returns False if no proxy is in use. Used to determine whether
errors are coming from the proxy layer or from tunnelling to the target origin.
"""
class BaseHTTPSConnection(BaseHTTPConnection, Protocol):
default_port: typing.ClassVar[int]
default_socket_options: typing.ClassVar[_TYPE_SOCKET_OPTIONS]
# Certificate verification methods
cert_reqs: int | str | None
assert_hostname: None | str | typing.Literal[False]
assert_fingerprint: str | None
ssl_context: ssl.SSLContext | None
# Trusted CAs
ca_certs: str | None
ca_cert_dir: str | None
ca_cert_data: None | str | bytes
# TLS version
ssl_minimum_version: int | None
ssl_maximum_version: int | None
ssl_version: int | str | None # Deprecated
# Client certificates
cert_file: str | None
key_file: str | None
key_password: str | None
def __init__(
self,
host: str,
port: int | None = None,
*,
timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
source_address: tuple[str, int] | None = None,
blocksize: int = 16384,
socket_options: _TYPE_SOCKET_OPTIONS | None = ...,
proxy: Url | None = None,
proxy_config: ProxyConfig | None = None,
cert_reqs: int | str | None = None,
assert_hostname: None | str | typing.Literal[False] = None,
assert_fingerprint: str | None = None,
server_hostname: str | None = None,
ssl_context: ssl.SSLContext | None = None,
ca_certs: str | None = None,
ca_cert_dir: str | None = None,
ca_cert_data: None | str | bytes = None,
ssl_minimum_version: int | None = None,
ssl_maximum_version: int | None = None,
ssl_version: int | str | None = None, # Deprecated
cert_file: str | None = None,
key_file: str | None = None,
key_password: str | None = None,
) -> None: ...
@@ -0,0 +1,486 @@
from __future__ import annotations
import typing
from collections import OrderedDict
from enum import Enum, auto
from threading import RLock
if typing.TYPE_CHECKING:
# We can only import Protocol if TYPE_CHECKING because it's a development
# dependency, and is not available at runtime.
from typing import Protocol
from typing_extensions import Self
class HasGettableStringKeys(Protocol):
def keys(self) -> typing.Iterator[str]: ...
def __getitem__(self, key: str) -> str: ...
__all__ = ["RecentlyUsedContainer", "HTTPHeaderDict"]
# Key type
_KT = typing.TypeVar("_KT")
# Value type
_VT = typing.TypeVar("_VT")
# Default type
_DT = typing.TypeVar("_DT")
ValidHTTPHeaderSource = typing.Union[
"HTTPHeaderDict",
typing.Mapping[str, str],
typing.Iterable[tuple[str, str]],
"HasGettableStringKeys",
]
class _Sentinel(Enum):
not_passed = auto()
def ensure_can_construct_http_header_dict(
potential: object,
) -> ValidHTTPHeaderSource | None:
if isinstance(potential, HTTPHeaderDict):
return potential
elif isinstance(potential, typing.Mapping):
# Full runtime checking of the contents of a Mapping is expensive, so for the
# purposes of typechecking, we assume that any Mapping is the right shape.
return typing.cast(typing.Mapping[str, str], potential)
elif isinstance(potential, typing.Iterable):
# Similarly to Mapping, full runtime checking of the contents of an Iterable is
# expensive, so for the purposes of typechecking, we assume that any Iterable
# is the right shape.
return typing.cast(typing.Iterable[tuple[str, str]], potential)
elif hasattr(potential, "keys") and hasattr(potential, "__getitem__"):
return typing.cast("HasGettableStringKeys", potential)
else:
return None
class RecentlyUsedContainer(typing.Generic[_KT, _VT], typing.MutableMapping[_KT, _VT]):
"""
Provides a thread-safe dict-like container which maintains up to
``maxsize`` keys while throwing away the least-recently-used keys beyond
``maxsize``.
:param maxsize:
Maximum number of recent elements to retain.
:param dispose_func:
Every time an item is evicted from the container,
``dispose_func(value)`` is called. Callback which will get called
"""
_container: typing.OrderedDict[_KT, _VT]
_maxsize: int
dispose_func: typing.Callable[[_VT], None] | None
lock: RLock
def __init__(
self,
maxsize: int = 10,
dispose_func: typing.Callable[[_VT], None] | None = None,
) -> None:
super().__init__()
self._maxsize = maxsize
self.dispose_func = dispose_func
self._container = OrderedDict()
self.lock = RLock()
def __getitem__(self, key: _KT) -> _VT:
# Re-insert the item, moving it to the end of the eviction line.
with self.lock:
item = self._container.pop(key)
self._container[key] = item
return item
def __setitem__(self, key: _KT, value: _VT) -> None:
evicted_item = None
with self.lock:
# Possibly evict the existing value of 'key'
try:
# If the key exists, we'll overwrite it, which won't change the
# size of the pool. Because accessing a key should move it to
# the end of the eviction line, we pop it out first.
evicted_item = key, self._container.pop(key)
self._container[key] = value
except KeyError:
# When the key does not exist, we insert the value first so that
# evicting works in all cases, including when self._maxsize is 0
self._container[key] = value
if len(self._container) > self._maxsize:
# If we didn't evict an existing value, and we've hit our maximum
# size, then we have to evict the least recently used item from
# the beginning of the container.
evicted_item = self._container.popitem(last=False)
# After releasing the lock on the pool, dispose of any evicted value.
if evicted_item is not None and self.dispose_func:
_, evicted_value = evicted_item
self.dispose_func(evicted_value)
def __delitem__(self, key: _KT) -> None:
with self.lock:
value = self._container.pop(key)
if self.dispose_func:
self.dispose_func(value)
def __len__(self) -> int:
with self.lock:
return len(self._container)
def __iter__(self) -> typing.NoReturn:
raise NotImplementedError(
"Iteration over this class is unlikely to be threadsafe."
)
def clear(self) -> None:
with self.lock:
# Copy pointers to all values, then wipe the mapping
values = list(self._container.values())
self._container.clear()
if self.dispose_func:
for value in values:
self.dispose_func(value)
def keys(self) -> set[_KT]: # type: ignore[override]
with self.lock:
return set(self._container.keys())
class HTTPHeaderDictItemView(set[tuple[str, str]]):
"""
HTTPHeaderDict is unusual for a Mapping[str, str] in that it has two modes of
address.
If we directly try to get an item with a particular name, we will get a string
back that is the concatenated version of all the values:
>>> d['X-Header-Name']
'Value1, Value2, Value3'
However, if we iterate over an HTTPHeaderDict's items, we will optionally combine
these values based on whether combine=True was called when building up the dictionary
>>> d = HTTPHeaderDict({"A": "1", "B": "foo"})
>>> d.add("A", "2", combine=True)
>>> d.add("B", "bar")
>>> list(d.items())
[
('A', '1, 2'),
('B', 'foo'),
('B', 'bar'),
]
This class conforms to the interface required by the MutableMapping ABC while
also giving us the nonstandard iteration behavior we want; items with duplicate
keys, ordered by time of first insertion.
"""
_headers: HTTPHeaderDict
def __init__(self, headers: HTTPHeaderDict) -> None:
self._headers = headers
def __len__(self) -> int:
return len(list(self._headers.iteritems()))
def __iter__(self) -> typing.Iterator[tuple[str, str]]:
return self._headers.iteritems()
def __contains__(self, item: object) -> bool:
if isinstance(item, tuple) and len(item) == 2:
passed_key, passed_val = item
if isinstance(passed_key, str) and isinstance(passed_val, str):
return self._headers._has_value_for_header(passed_key, passed_val)
return False
class HTTPHeaderDict(typing.MutableMapping[str, str]):
"""
:param headers:
An iterable of field-value pairs. Must not contain multiple field names
when compared case-insensitively.
:param kwargs:
Additional field-value pairs to pass in to ``dict.update``.
A ``dict`` like container for storing HTTP Headers.
Field names are stored and compared case-insensitively in compliance with
RFC 7230. Iteration provides the first case-sensitive key seen for each
case-insensitive pair.
Using ``__setitem__`` syntax overwrites fields that compare equal
case-insensitively in order to maintain ``dict``'s api. For fields that
compare equal, instead create a new ``HTTPHeaderDict`` and use ``.add``
in a loop.
If multiple fields that are equal case-insensitively are passed to the
constructor or ``.update``, the behavior is undefined and some will be
lost.
>>> headers = HTTPHeaderDict()
>>> headers.add('Set-Cookie', 'foo=bar')
>>> headers.add('set-cookie', 'baz=quxx')
>>> headers['content-length'] = '7'
>>> headers['SET-cookie']
'foo=bar, baz=quxx'
>>> headers['Content-Length']
'7'
"""
_container: typing.MutableMapping[str, list[str]]
def __init__(self, headers: ValidHTTPHeaderSource | None = None, **kwargs: str):
super().__init__()
self._container = {} # 'dict' is insert-ordered
if headers is not None:
if isinstance(headers, HTTPHeaderDict):
self._copy_from(headers)
else:
self.extend(headers)
if kwargs:
self.extend(kwargs)
def __setitem__(self, key: str, val: str) -> None:
# avoid a bytes/str comparison by decoding before httplib
if isinstance(key, bytes):
key = key.decode("latin-1")
self._container[key.lower()] = [key, val]
def __getitem__(self, key: str) -> str:
if isinstance(key, bytes):
key = key.decode("latin-1")
val = self._container[key.lower()]
return ", ".join(val[1:])
def __delitem__(self, key: str) -> None:
if isinstance(key, bytes):
key = key.decode("latin-1")
del self._container[key.lower()]
def __contains__(self, key: object) -> bool:
if isinstance(key, bytes):
key = key.decode("latin-1")
if isinstance(key, str):
return key.lower() in self._container
return False
def setdefault(self, key: str, default: str = "") -> str:
return super().setdefault(key, default)
def __eq__(self, other: object) -> bool:
maybe_constructable = ensure_can_construct_http_header_dict(other)
if maybe_constructable is None:
return False
else:
other_as_http_header_dict = type(self)(maybe_constructable)
return {k.lower(): v for k, v in self.itermerged()} == {
k.lower(): v for k, v in other_as_http_header_dict.itermerged()
}
def __ne__(self, other: object) -> bool:
return not self.__eq__(other)
def __len__(self) -> int:
return len(self._container)
def __iter__(self) -> typing.Iterator[str]:
# Only provide the originally cased names
for vals in self._container.values():
yield vals[0]
def discard(self, key: str) -> None:
try:
del self[key]
except KeyError:
pass
def add(self, key: str, val: str, *, combine: bool = False) -> None:
"""Adds a (name, value) pair, doesn't overwrite the value if it already
exists.
If this is called with combine=True, instead of adding a new header value
as a distinct item during iteration, this will instead append the value to
any existing header value with a comma. If no existing header value exists
for the key, then the value will simply be added, ignoring the combine parameter.
>>> headers = HTTPHeaderDict(foo='bar')
>>> headers.add('Foo', 'baz')
>>> headers['foo']
'bar, baz'
>>> list(headers.items())
[('foo', 'bar'), ('foo', 'baz')]
>>> headers.add('foo', 'quz', combine=True)
>>> list(headers.items())
[('foo', 'bar, baz, quz')]
"""
# avoid a bytes/str comparison by decoding before httplib
if isinstance(key, bytes):
key = key.decode("latin-1")
key_lower = key.lower()
new_vals = [key, val]
# Keep the common case aka no item present as fast as possible
vals = self._container.setdefault(key_lower, new_vals)
if new_vals is not vals:
# if there are values here, then there is at least the initial
# key/value pair
assert len(vals) >= 2
if combine:
vals[-1] = vals[-1] + ", " + val
else:
vals.append(val)
def extend(self, *args: ValidHTTPHeaderSource, **kwargs: str) -> None:
"""Generic import function for any type of header-like object.
Adapted version of MutableMapping.update in order to insert items
with self.add instead of self.__setitem__
"""
if len(args) > 1:
raise TypeError(
f"extend() takes at most 1 positional arguments ({len(args)} given)"
)
other = args[0] if len(args) >= 1 else ()
if isinstance(other, HTTPHeaderDict):
for key, val in other.iteritems():
self.add(key, val)
elif isinstance(other, typing.Mapping):
for key, val in other.items():
self.add(key, val)
elif isinstance(other, typing.Iterable):
for key, value in other:
self.add(key, value)
elif hasattr(other, "keys") and hasattr(other, "__getitem__"):
# THIS IS NOT A TYPESAFE BRANCH
# In this branch, the object has a `keys` attr but is not a Mapping or any of
# the other types indicated in the method signature. We do some stuff with
# it as though it partially implements the Mapping interface, but we're not
# doing that stuff safely AT ALL.
for key in other.keys():
self.add(key, other[key])
for key, value in kwargs.items():
self.add(key, value)
@typing.overload
def getlist(self, key: str) -> list[str]: ...
@typing.overload
def getlist(self, key: str, default: _DT) -> list[str] | _DT: ...
def getlist(
self, key: str, default: _Sentinel | _DT = _Sentinel.not_passed
) -> list[str] | _DT:
"""Returns a list of all the values for the named field. Returns an
empty list if the key doesn't exist."""
if isinstance(key, bytes):
key = key.decode("latin-1")
try:
vals = self._container[key.lower()]
except KeyError:
if default is _Sentinel.not_passed:
# _DT is unbound; empty list is instance of List[str]
return []
# _DT is bound; default is instance of _DT
return default
else:
# _DT may or may not be bound; vals[1:] is instance of List[str], which
# meets our external interface requirement of `Union[List[str], _DT]`.
return vals[1:]
def _prepare_for_method_change(self) -> Self:
"""
Remove content-specific header fields before changing the request
method to GET or HEAD according to RFC 9110, Section 15.4.
"""
content_specific_headers = [
"Content-Encoding",
"Content-Language",
"Content-Location",
"Content-Type",
"Content-Length",
"Digest",
"Last-Modified",
]
for header in content_specific_headers:
self.discard(header)
return self
# Backwards compatibility for httplib
getheaders = getlist
getallmatchingheaders = getlist
iget = getlist
# Backwards compatibility for http.cookiejar
get_all = getlist
def __repr__(self) -> str:
return f"{type(self).__name__}({dict(self.itermerged())})"
def _copy_from(self, other: HTTPHeaderDict) -> None:
for key in other:
val = other.getlist(key)
self._container[key.lower()] = [key, *val]
def copy(self) -> Self:
clone = type(self)()
clone._copy_from(self)
return clone
def iteritems(self) -> typing.Iterator[tuple[str, str]]:
"""Iterate over all header lines, including duplicate ones."""
for key in self:
vals = self._container[key.lower()]
for val in vals[1:]:
yield vals[0], val
def itermerged(self) -> typing.Iterator[tuple[str, str]]:
"""Iterate over all headers, merging duplicate ones together."""
for key in self:
val = self._container[key.lower()]
yield val[0], ", ".join(val[1:])
def items(self) -> HTTPHeaderDictItemView: # type: ignore[override]
return HTTPHeaderDictItemView(self)
def _has_value_for_header(self, header_name: str, potential_value: str) -> bool:
if header_name in self:
return potential_value in self._container[header_name.lower()][1:]
return False
def __ior__(self, other: object) -> HTTPHeaderDict:
# Supports extending a header dict in-place using operator |=
# combining items with add instead of __setitem__
maybe_constructable = ensure_can_construct_http_header_dict(other)
if maybe_constructable is None:
return NotImplemented
self.extend(maybe_constructable)
return self
def __or__(self, other: object) -> Self:
# Supports merging header dicts using operator |
# combining items with add instead of __setitem__
maybe_constructable = ensure_can_construct_http_header_dict(other)
if maybe_constructable is None:
return NotImplemented
result = self.copy()
result.extend(maybe_constructable)
return result
def __ror__(self, other: object) -> Self:
# Supports merging header dicts using operator | when other is on left side
# combining items with add instead of __setitem__
maybe_constructable = ensure_can_construct_http_header_dict(other)
if maybe_constructable is None:
return NotImplemented
result = type(self)(maybe_constructable)
result.extend(self)
return result
@@ -0,0 +1,278 @@
from __future__ import annotations
import json as _json
import typing
from urllib.parse import urlencode
from ._base_connection import _TYPE_BODY
from ._collections import HTTPHeaderDict
from .filepost import _TYPE_FIELDS, encode_multipart_formdata
from .response import BaseHTTPResponse
__all__ = ["RequestMethods"]
_TYPE_ENCODE_URL_FIELDS = typing.Union[
typing.Sequence[tuple[str, typing.Union[str, bytes]]],
typing.Mapping[str, typing.Union[str, bytes]],
]
class RequestMethods:
"""
Convenience mixin for classes who implement a :meth:`urlopen` method, such
as :class:`urllib3.HTTPConnectionPool` and
:class:`urllib3.PoolManager`.
Provides behavior for making common types of HTTP request methods and
decides which type of request field encoding to use.
Specifically,
:meth:`.request_encode_url` is for sending requests whose fields are
encoded in the URL (such as GET, HEAD, DELETE).
:meth:`.request_encode_body` is for sending requests whose fields are
encoded in the *body* of the request using multipart or www-form-urlencoded
(such as for POST, PUT, PATCH).
:meth:`.request` is for making any kind of request, it will look up the
appropriate encoding format and use one of the above two methods to make
the request.
Initializer parameters:
:param headers:
Headers to include with all requests, unless other headers are given
explicitly.
"""
_encode_url_methods = {"DELETE", "GET", "HEAD", "OPTIONS"}
def __init__(self, headers: typing.Mapping[str, str] | None = None) -> None:
self.headers = headers or {}
def urlopen(
self,
method: str,
url: str,
body: _TYPE_BODY | None = None,
headers: typing.Mapping[str, str] | None = None,
encode_multipart: bool = True,
multipart_boundary: str | None = None,
**kw: typing.Any,
) -> BaseHTTPResponse: # Abstract
raise NotImplementedError(
"Classes extending RequestMethods must implement "
"their own ``urlopen`` method."
)
def request(
self,
method: str,
url: str,
body: _TYPE_BODY | None = None,
fields: _TYPE_FIELDS | None = None,
headers: typing.Mapping[str, str] | None = None,
json: typing.Any | None = None,
**urlopen_kw: typing.Any,
) -> BaseHTTPResponse:
"""
Make a request using :meth:`urlopen` with the appropriate encoding of
``fields`` based on the ``method`` used.
This is a convenience method that requires the least amount of manual
effort. It can be used in most situations, while still having the
option to drop down to more specific methods when necessary, such as
:meth:`request_encode_url`, :meth:`request_encode_body`,
or even the lowest level :meth:`urlopen`.
:param method:
HTTP request method (such as GET, POST, PUT, etc.)
:param url:
The URL to perform the request on.
:param body:
Data to send in the request body, either :class:`str`, :class:`bytes`,
an iterable of :class:`str`/:class:`bytes`, or a file-like object.
:param fields:
Data to encode and send in the URL or request body, depending on ``method``.
:param headers:
Dictionary of custom headers to send, such as User-Agent,
If-None-Match, etc. If None, pool headers are used. If provided,
these headers completely replace any pool-specific headers.
:param json:
Data to encode and send as JSON with UTF-encoded in the request body.
The ``"Content-Type"`` header will be set to ``"application/json"``
unless specified otherwise.
"""
method = method.upper()
if json is not None and body is not None:
raise TypeError(
"request got values for both 'body' and 'json' parameters which are mutually exclusive"
)
if json is not None:
if headers is None:
headers = self.headers
if not ("content-type" in map(str.lower, headers.keys())):
headers = HTTPHeaderDict(headers)
headers["Content-Type"] = "application/json"
body = _json.dumps(json, separators=(",", ":"), ensure_ascii=False).encode(
"utf-8"
)
if body is not None:
urlopen_kw["body"] = body
if method in self._encode_url_methods:
return self.request_encode_url(
method,
url,
fields=fields, # type: ignore[arg-type]
headers=headers,
**urlopen_kw,
)
else:
return self.request_encode_body(
method, url, fields=fields, headers=headers, **urlopen_kw
)
def request_encode_url(
self,
method: str,
url: str,
fields: _TYPE_ENCODE_URL_FIELDS | None = None,
headers: typing.Mapping[str, str] | None = None,
**urlopen_kw: str,
) -> BaseHTTPResponse:
"""
Make a request using :meth:`urlopen` with the ``fields`` encoded in
the url. This is useful for request methods like GET, HEAD, DELETE, etc.
:param method:
HTTP request method (such as GET, POST, PUT, etc.)
:param url:
The URL to perform the request on.
:param fields:
Data to encode and send in the URL.
:param headers:
Dictionary of custom headers to send, such as User-Agent,
If-None-Match, etc. If None, pool headers are used. If provided,
these headers completely replace any pool-specific headers.
"""
if headers is None:
headers = self.headers
extra_kw: dict[str, typing.Any] = {"headers": headers}
extra_kw.update(urlopen_kw)
if fields:
url += "?" + urlencode(fields)
return self.urlopen(method, url, **extra_kw)
def request_encode_body(
self,
method: str,
url: str,
fields: _TYPE_FIELDS | None = None,
headers: typing.Mapping[str, str] | None = None,
encode_multipart: bool = True,
multipart_boundary: str | None = None,
**urlopen_kw: str,
) -> BaseHTTPResponse:
"""
Make a request using :meth:`urlopen` with the ``fields`` encoded in
the body. This is useful for request methods like POST, PUT, PATCH, etc.
When ``encode_multipart=True`` (default), then
:func:`urllib3.encode_multipart_formdata` is used to encode
the payload with the appropriate content type. Otherwise
:func:`urllib.parse.urlencode` is used with the
'application/x-www-form-urlencoded' content type.
Multipart encoding must be used when posting files, and it's reasonably
safe to use it in other times too. However, it may break request
signing, such as with OAuth.
Supports an optional ``fields`` parameter of key/value strings AND
key/filetuple. A filetuple is a (filename, data, MIME type) tuple where
the MIME type is optional. For example::
fields = {
'foo': 'bar',
'fakefile': ('foofile.txt', 'contents of foofile'),
'realfile': ('barfile.txt', open('realfile').read()),
'typedfile': ('bazfile.bin', open('bazfile').read(),
'image/jpeg'),
'nonamefile': 'contents of nonamefile field',
}
When uploading a file, providing a filename (the first parameter of the
tuple) is optional but recommended to best mimic behavior of browsers.
Note that if ``headers`` are supplied, the 'Content-Type' header will
be overwritten because it depends on the dynamic random boundary string
which is used to compose the body of the request. The random boundary
string can be explicitly set with the ``multipart_boundary`` parameter.
:param method:
HTTP request method (such as GET, POST, PUT, etc.)
:param url:
The URL to perform the request on.
:param fields:
Data to encode and send in the request body.
:param headers:
Dictionary of custom headers to send, such as User-Agent,
If-None-Match, etc. If None, pool headers are used. If provided,
these headers completely replace any pool-specific headers.
:param encode_multipart:
If True, encode the ``fields`` using the multipart/form-data MIME
format.
:param multipart_boundary:
If not specified, then a random boundary will be generated using
:func:`urllib3.filepost.choose_boundary`.
"""
if headers is None:
headers = self.headers
extra_kw: dict[str, typing.Any] = {"headers": HTTPHeaderDict(headers)}
body: bytes | str
if fields:
if "body" in urlopen_kw:
raise TypeError(
"request got values for both 'fields' and 'body', can only specify one."
)
if encode_multipart:
body, content_type = encode_multipart_formdata(
fields, boundary=multipart_boundary
)
else:
body, content_type = (
urlencode(fields), # type: ignore[arg-type]
"application/x-www-form-urlencoded",
)
extra_kw["body"] = body
extra_kw["headers"].setdefault("Content-Type", content_type)
extra_kw.update(urlopen_kw)
return self.urlopen(method, url, **extra_kw)
@@ -0,0 +1,24 @@
# file generated by vcs-versioning
# don't change, don't track in version control
from __future__ import annotations
__all__ = [
"__version__",
"__version_tuple__",
"version",
"version_tuple",
"__commit_id__",
"commit_id",
]
version: str
__version__: str
__version_tuple__: tuple[int | str, ...]
version_tuple: tuple[int | str, ...]
commit_id: str | None
__commit_id__: str | None
__version__ = version = '2.7.0'
__version_tuple__ = version_tuple = (2, 7, 0)
__commit_id__ = commit_id = None
@@ -0,0 +1,17 @@
from __future__ import annotations
import urllib3.connection
from ...connectionpool import HTTPConnectionPool, HTTPSConnectionPool
from .connection import EmscriptenHTTPConnection, EmscriptenHTTPSConnection
def inject_into_urllib3() -> None:
# override connection classes to use emscripten specific classes
# n.b. mypy complains about the overriding of classes below
# if it isn't ignored
HTTPConnectionPool.ConnectionCls = EmscriptenHTTPConnection
HTTPSConnectionPool.ConnectionCls = EmscriptenHTTPSConnection
urllib3.connection.HTTPConnection = EmscriptenHTTPConnection # type: ignore[misc,assignment]
urllib3.connection.HTTPSConnection = EmscriptenHTTPSConnection # type: ignore[misc,assignment]
urllib3.connection.VerifiedHTTPSConnection = EmscriptenHTTPSConnection # type: ignore[assignment]
@@ -0,0 +1,260 @@
from __future__ import annotations
import os
import typing
# use http.client.HTTPException for consistency with non-emscripten
from http.client import HTTPException as HTTPException # noqa: F401
from http.client import ResponseNotReady
from ..._base_connection import _TYPE_BODY
from ...connection import HTTPConnection, ProxyConfig, port_by_scheme
from ...exceptions import TimeoutError
from ...response import BaseHTTPResponse
from ...util.connection import _TYPE_SOCKET_OPTIONS
from ...util.timeout import _DEFAULT_TIMEOUT, _TYPE_TIMEOUT
from ...util.url import Url
from .fetch import _RequestError, _TimeoutError, send_request, send_streaming_request
from .request import EmscriptenRequest
from .response import EmscriptenHttpResponseWrapper, EmscriptenResponse
if typing.TYPE_CHECKING:
from ..._base_connection import BaseHTTPConnection, BaseHTTPSConnection
class EmscriptenHTTPConnection:
default_port: typing.ClassVar[int] = port_by_scheme["http"]
default_socket_options: typing.ClassVar[_TYPE_SOCKET_OPTIONS]
timeout: None | (float)
host: str
port: int
blocksize: int
source_address: tuple[str, int] | None
socket_options: _TYPE_SOCKET_OPTIONS | None
proxy: Url | None
proxy_config: ProxyConfig | None
is_verified: bool = False
proxy_is_verified: bool | None = None
response_class: type[BaseHTTPResponse] = EmscriptenHttpResponseWrapper
_response: EmscriptenResponse | None
def __init__(
self,
host: str,
port: int = 0,
*,
timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
source_address: tuple[str, int] | None = None,
blocksize: int = 8192,
socket_options: _TYPE_SOCKET_OPTIONS | None = None,
proxy: Url | None = None,
proxy_config: ProxyConfig | None = None,
) -> None:
self.host = host
self.port = port
self.timeout = timeout if isinstance(timeout, float) else 0.0
self.scheme = "http"
self._closed = True
self._response = None
# ignore these things because we don't
# have control over that stuff
self.proxy = None
self.proxy_config = None
self.blocksize = blocksize
self.source_address = None
self.socket_options = None
self.is_verified = False
def set_tunnel(
self,
host: str,
port: int | None = 0,
headers: typing.Mapping[str, str] | None = None,
scheme: str = "http",
) -> None:
pass
def connect(self) -> None:
pass
def request(
self,
method: str,
url: str,
body: _TYPE_BODY | None = None,
headers: typing.Mapping[str, str] | None = None,
# We know *at least* botocore is depending on the order of the
# first 3 parameters so to be safe we only mark the later ones
# as keyword-only to ensure we have space to extend.
*,
chunked: bool = False,
preload_content: bool = True,
decode_content: bool = True,
enforce_content_length: bool = True,
) -> None:
self._closed = False
if url.startswith("/"):
if self.port is not None:
port = f":{self.port}"
else:
port = ""
# no scheme / host / port included, make a full url
url = f"{self.scheme}://{self.host}{port}{url}"
request = EmscriptenRequest(
url=url,
method=method,
timeout=self.timeout if self.timeout else 0,
decode_content=decode_content,
)
request.set_body(body)
if headers:
for k, v in headers.items():
request.set_header(k, v)
self._response = None
try:
if not preload_content:
self._response = send_streaming_request(request)
if self._response is None:
self._response = send_request(request)
except _TimeoutError as e:
raise TimeoutError(e.message) from e
except _RequestError as e:
raise HTTPException(e.message) from e
def getresponse(self) -> BaseHTTPResponse:
if self._response is not None:
return EmscriptenHttpResponseWrapper(
internal_response=self._response,
url=self._response.request.url,
connection=self,
)
else:
raise ResponseNotReady()
def close(self) -> None:
self._closed = True
self._response = None
@property
def is_closed(self) -> bool:
"""Whether the connection either is brand new or has been previously closed.
If this property is True then both ``is_connected`` and ``has_connected_to_proxy``
properties must be False.
"""
return self._closed
@property
def is_connected(self) -> bool:
"""Whether the connection is actively connected to any origin (proxy or target)"""
return True
@property
def has_connected_to_proxy(self) -> bool:
"""Whether the connection has successfully connected to its proxy.
This returns False if no proxy is in use. Used to determine whether
errors are coming from the proxy layer or from tunnelling to the target origin.
"""
return False
class EmscriptenHTTPSConnection(EmscriptenHTTPConnection):
default_port = port_by_scheme["https"]
# all this is basically ignored, as browser handles https
cert_reqs: int | str | None = None
ca_certs: str | None = None
ca_cert_dir: str | None = None
ca_cert_data: None | str | bytes = None
cert_file: str | None
key_file: str | None
key_password: str | None
ssl_context: typing.Any | None
ssl_version: int | str | None = None
ssl_minimum_version: int | None = None
ssl_maximum_version: int | None = None
assert_hostname: None | str | typing.Literal[False]
assert_fingerprint: str | None = None
def __init__(
self,
host: str,
port: int = 0,
*,
timeout: _TYPE_TIMEOUT = _DEFAULT_TIMEOUT,
source_address: tuple[str, int] | None = None,
blocksize: int = 16384,
socket_options: (
None | _TYPE_SOCKET_OPTIONS
) = HTTPConnection.default_socket_options,
proxy: Url | None = None,
proxy_config: ProxyConfig | None = None,
cert_reqs: int | str | None = None,
assert_hostname: None | str | typing.Literal[False] = None,
assert_fingerprint: str | None = None,
server_hostname: str | None = None,
ssl_context: typing.Any | None = None,
ca_certs: str | None = None,
ca_cert_dir: str | None = None,
ca_cert_data: None | str | bytes = None,
ssl_minimum_version: int | None = None,
ssl_maximum_version: int | None = None,
ssl_version: int | str | None = None, # Deprecated
cert_file: str | None = None,
key_file: str | None = None,
key_password: str | None = None,
) -> None:
super().__init__(
host,
port=port,
timeout=timeout,
source_address=source_address,
blocksize=blocksize,
socket_options=socket_options,
proxy=proxy,
proxy_config=proxy_config,
)
self.scheme = "https"
self.key_file = key_file
self.cert_file = cert_file
self.key_password = key_password
self.ssl_context = ssl_context
self.server_hostname = server_hostname
self.assert_hostname = assert_hostname
self.assert_fingerprint = assert_fingerprint
self.ssl_version = ssl_version
self.ssl_minimum_version = ssl_minimum_version
self.ssl_maximum_version = ssl_maximum_version
self.ca_certs = ca_certs and os.path.expanduser(ca_certs)
self.ca_cert_dir = ca_cert_dir and os.path.expanduser(ca_cert_dir)
self.ca_cert_data = ca_cert_data
self.cert_reqs = None
# The browser will automatically verify all requests.
# We have no control over that setting.
self.is_verified = True
def set_cert(
self,
key_file: str | None = None,
cert_file: str | None = None,
cert_reqs: int | str | None = None,
key_password: str | None = None,
ca_certs: str | None = None,
assert_hostname: None | str | typing.Literal[False] = None,
assert_fingerprint: str | None = None,
ca_cert_dir: str | None = None,
ca_cert_data: None | str | bytes = None,
) -> None:
pass
# verify that this class implements BaseHTTP(s) connection correctly
if typing.TYPE_CHECKING:
_supports_http_protocol: BaseHTTPConnection = EmscriptenHTTPConnection("", 0)
_supports_https_protocol: BaseHTTPSConnection = EmscriptenHTTPSConnection("", 0)
@@ -0,0 +1,726 @@
"""
Support for streaming http requests in emscripten.
A few caveats -
If your browser (or Node.js) has WebAssembly JavaScript Promise Integration enabled
https://github.com/WebAssembly/js-promise-integration/blob/main/proposals/js-promise-integration/Overview.md
*and* you launch pyodide using `pyodide.runPythonAsync`, this will fetch data using the
JavaScript asynchronous fetch api (wrapped via `pyodide.ffi.call_sync`). In this case
timeouts and streaming should just work.
Otherwise, it uses a combination of XMLHttpRequest and a web-worker for streaming.
This approach has several caveats:
Firstly, you can't do streaming http in the main UI thread, because atomics.wait isn't allowed.
Streaming only works if you're running pyodide in a web worker.
Secondly, this uses an extra web worker and SharedArrayBuffer to do the asynchronous fetch
operation, so it requires that you have crossOriginIsolation enabled, by serving over https
(or from localhost) with the two headers below set:
Cross-Origin-Opener-Policy: same-origin
Cross-Origin-Embedder-Policy: require-corp
You can tell if cross origin isolation is successfully enabled by looking at the global crossOriginIsolated variable in
JavaScript console. If it isn't, streaming requests will fallback to XMLHttpRequest, i.e. getting the whole
request into a buffer and then returning it. it shows a warning in the JavaScript console in this case.
Finally, the webworker which does the streaming fetch is created on initial import, but will only be started once
control is returned to javascript. Call `await wait_for_streaming_ready()` to wait for streaming fetch.
NB: in this code, there are a lot of JavaScript objects. They are named js_*
to make it clear what type of object they are.
"""
from __future__ import annotations
import io
import json
from email.parser import Parser
from importlib.resources import files
from typing import TYPE_CHECKING, Any
import js # type: ignore[import-not-found]
from pyodide.ffi import ( # type: ignore[import-not-found]
JsArray,
JsException,
JsProxy,
to_js,
)
if TYPE_CHECKING:
from typing_extensions import Buffer
from .request import EmscriptenRequest
from .response import EmscriptenResponse
"""
There are some headers that trigger unintended CORS preflight requests.
See also https://github.com/koenvo/pyodide-http/issues/22
"""
HEADERS_TO_IGNORE = ("user-agent",)
SUCCESS_HEADER = -1
SUCCESS_EOF = -2
ERROR_TIMEOUT = -3
ERROR_EXCEPTION = -4
class _RequestError(Exception):
def __init__(
self,
message: str | None = None,
*,
request: EmscriptenRequest | None = None,
response: EmscriptenResponse | None = None,
):
self.request = request
self.response = response
self.message = message
super().__init__(self.message)
class _StreamingError(_RequestError):
pass
class _TimeoutError(_RequestError):
pass
def _obj_from_dict(dict_val: dict[str, Any]) -> JsProxy:
return to_js(dict_val, dict_converter=js.Object.fromEntries)
class _ReadStream(io.RawIOBase):
def __init__(
self,
int_buffer: JsArray,
byte_buffer: JsArray,
timeout: float,
worker: JsProxy,
connection_id: int,
request: EmscriptenRequest,
):
self.int_buffer = int_buffer
self.byte_buffer = byte_buffer
self.read_pos = 0
self.read_len = 0
self.connection_id = connection_id
self.worker = worker
self.timeout = int(1000 * timeout) if timeout > 0 else None
self.is_live = True
self._is_closed = False
self.request: EmscriptenRequest | None = request
def __del__(self) -> None:
self.close()
# this is compatible with _base_connection
def is_closed(self) -> bool:
return self._is_closed
# for compatibility with RawIOBase
@property
def closed(self) -> bool:
return self.is_closed()
def close(self) -> None:
if self.is_closed():
return
self.read_len = 0
self.read_pos = 0
self.int_buffer = None
self.byte_buffer = None
self._is_closed = True
self.request = None
if self.is_live:
self.worker.postMessage(_obj_from_dict({"close": self.connection_id}))
self.is_live = False
super().close()
def readable(self) -> bool:
return True
def writable(self) -> bool:
return False
def seekable(self) -> bool:
return False
def readinto(self, byte_obj: Buffer) -> int:
if not self.int_buffer:
raise _StreamingError(
"No buffer for stream in _ReadStream.readinto",
request=self.request,
response=None,
)
if self.read_len == 0:
# wait for the worker to send something
js.Atomics.store(self.int_buffer, 0, ERROR_TIMEOUT)
self.worker.postMessage(_obj_from_dict({"getMore": self.connection_id}))
if (
js.Atomics.wait(self.int_buffer, 0, ERROR_TIMEOUT, self.timeout)
== "timed-out"
):
raise _TimeoutError
data_len = self.int_buffer[0]
if data_len > 0:
self.read_len = data_len
self.read_pos = 0
elif data_len == ERROR_EXCEPTION:
string_len = self.int_buffer[1]
# decode the error string
js_decoder = js.TextDecoder.new()
json_str = js_decoder.decode(self.byte_buffer.slice(0, string_len))
raise _StreamingError(
f"Exception thrown in fetch: {json_str}",
request=self.request,
response=None,
)
else:
# EOF, free the buffers and return zero
# and free the request
self.is_live = False
self.close()
return 0
# copy from int32array to python bytes
ret_length = min(self.read_len, len(memoryview(byte_obj)))
subarray = self.byte_buffer.subarray(
self.read_pos, self.read_pos + ret_length
).to_py()
memoryview(byte_obj)[0:ret_length] = subarray
self.read_len -= ret_length
self.read_pos += ret_length
return ret_length
class _StreamingFetcher:
def __init__(self) -> None:
# make web-worker and data buffer on startup
self.streaming_ready = False
streaming_worker_code = (
files(__package__)
.joinpath("emscripten_fetch_worker.js")
.read_text(encoding="utf-8")
)
js_data_blob = js.Blob.new(
to_js([streaming_worker_code], create_pyproxies=False),
_obj_from_dict({"type": "application/javascript"}),
)
def promise_resolver(js_resolve_fn: JsProxy, js_reject_fn: JsProxy) -> None:
def onMsg(e: JsProxy) -> None:
self.streaming_ready = True
js_resolve_fn(e)
def onErr(e: JsProxy) -> None:
js_reject_fn(e) # Defensive: never happens in ci
self.js_worker.onmessage = onMsg
self.js_worker.onerror = onErr
js_data_url = js.URL.createObjectURL(js_data_blob)
self.js_worker = js.globalThis.Worker.new(js_data_url)
self.js_worker_ready_promise = js.globalThis.Promise.new(promise_resolver)
def send(self, request: EmscriptenRequest) -> EmscriptenResponse:
headers = {
k: v for k, v in request.headers.items() if k not in HEADERS_TO_IGNORE
}
body = request.body
fetch_data = {"headers": headers, "body": to_js(body), "method": request.method}
# start the request off in the worker
timeout = int(1000 * request.timeout) if request.timeout > 0 else None
js_shared_buffer = js.SharedArrayBuffer.new(1048576)
js_int_buffer = js.Int32Array.new(js_shared_buffer)
js_byte_buffer = js.Uint8Array.new(js_shared_buffer, 8)
js.Atomics.store(js_int_buffer, 0, ERROR_TIMEOUT)
js.Atomics.notify(js_int_buffer, 0)
js_absolute_url = js.URL.new(request.url, js.location).href
self.js_worker.postMessage(
_obj_from_dict(
{
"buffer": js_shared_buffer,
"url": js_absolute_url,
"fetchParams": fetch_data,
}
)
)
# wait for the worker to send something
js.Atomics.wait(js_int_buffer, 0, ERROR_TIMEOUT, timeout)
if js_int_buffer[0] == ERROR_TIMEOUT:
raise _TimeoutError(
"Timeout connecting to streaming request",
request=request,
response=None,
)
elif js_int_buffer[0] == SUCCESS_HEADER:
# got response
# header length is in second int of intBuffer
string_len = js_int_buffer[1]
# decode the rest to a JSON string
js_decoder = js.TextDecoder.new()
# this does a copy (the slice) because decode can't work on shared array
# for some silly reason
json_str = js_decoder.decode(js_byte_buffer.slice(0, string_len))
# get it as an object
response_obj = json.loads(json_str)
return EmscriptenResponse(
request=request,
status_code=response_obj["status"],
headers=response_obj["headers"],
body=_ReadStream(
js_int_buffer,
js_byte_buffer,
request.timeout,
self.js_worker,
response_obj["connectionID"],
request,
),
)
elif js_int_buffer[0] == ERROR_EXCEPTION:
string_len = js_int_buffer[1]
# decode the error string
js_decoder = js.TextDecoder.new()
json_str = js_decoder.decode(js_byte_buffer.slice(0, string_len))
raise _StreamingError(
f"Exception thrown in fetch: {json_str}", request=request, response=None
)
else:
raise _StreamingError(
f"Unknown status from worker in fetch: {js_int_buffer[0]}",
request=request,
response=None,
)
class _JSPIReadStream(io.RawIOBase):
"""
A read stream that uses pyodide.ffi.run_sync to read from a JavaScript fetch
response. This requires support for WebAssembly JavaScript Promise Integration
in the containing browser, and for pyodide to be launched via runPythonAsync.
:param js_read_stream:
The JavaScript stream reader
:param timeout:
Timeout in seconds
:param request:
The request we're handling
:param response:
The response this stream relates to
:param js_abort_controller:
A JavaScript AbortController object, used for timeouts
"""
def __init__(
self,
js_read_stream: Any,
timeout: float,
request: EmscriptenRequest,
response: EmscriptenResponse,
js_abort_controller: Any, # JavaScript AbortController for timeouts
):
self.js_read_stream = js_read_stream
self.timeout = timeout
self._is_closed = False
self._is_done = False
self.request: EmscriptenRequest | None = request
self.response: EmscriptenResponse | None = response
self.current_buffer = None
self.current_buffer_pos = 0
self.js_abort_controller = js_abort_controller
def __del__(self) -> None:
self.close()
# this is compatible with _base_connection
def is_closed(self) -> bool:
return self._is_closed
# for compatibility with RawIOBase
@property
def closed(self) -> bool:
return self.is_closed()
def close(self) -> None:
if self.is_closed():
return
self.read_len = 0
self.read_pos = 0
self.js_read_stream.cancel()
self.js_read_stream = None
self._is_closed = True
self._is_done = True
self.request = None
self.response = None
super().close()
def readable(self) -> bool:
return True
def writable(self) -> bool:
return False
def seekable(self) -> bool:
return False
def _get_next_buffer(self) -> bool:
result_js = _run_sync_with_timeout(
self.js_read_stream.read(),
self.timeout,
self.js_abort_controller,
request=self.request,
response=self.response,
)
if result_js.done:
self._is_done = True
return False
else:
self.current_buffer = result_js.value.to_py()
self.current_buffer_pos = 0
return True
def readinto(self, byte_obj: Buffer) -> int:
if self.current_buffer is None:
if not self._get_next_buffer() or self.current_buffer is None:
self.close()
return 0
ret_length = min(
len(byte_obj), len(self.current_buffer) - self.current_buffer_pos
)
byte_obj[0:ret_length] = self.current_buffer[
self.current_buffer_pos : self.current_buffer_pos + ret_length
]
self.current_buffer_pos += ret_length
if self.current_buffer_pos == len(self.current_buffer):
self.current_buffer = None
return ret_length
# check if we are in a worker or not
def is_in_browser_main_thread() -> bool:
return hasattr(js, "window") and hasattr(js, "self") and js.self == js.window
def is_cross_origin_isolated() -> bool:
return hasattr(js, "crossOriginIsolated") and js.crossOriginIsolated
def is_in_node() -> bool:
return (
hasattr(js, "process")
and hasattr(js.process, "release")
and hasattr(js.process.release, "name")
and js.process.release.name == "node"
)
def is_worker_available() -> bool:
return hasattr(js, "Worker") and hasattr(js, "Blob")
_fetcher: _StreamingFetcher | None = None
if is_worker_available() and (
(is_cross_origin_isolated() and not is_in_browser_main_thread())
and (not is_in_node())
):
_fetcher = _StreamingFetcher()
else:
_fetcher = None
NODE_JSPI_ERROR = (
"urllib3 only works in Node.js with pyodide.runPythonAsync"
" and requires the flag --experimental-wasm-stack-switching in "
" versions of node <24."
)
def send_streaming_request(request: EmscriptenRequest) -> EmscriptenResponse | None:
if has_jspi():
return send_jspi_request(request, True)
elif is_in_node():
raise _RequestError(
message=NODE_JSPI_ERROR,
request=request,
response=None,
)
if _fetcher and streaming_ready():
return _fetcher.send(request)
else:
_show_streaming_warning()
return None
_SHOWN_TIMEOUT_WARNING = False
def _show_timeout_warning() -> None:
global _SHOWN_TIMEOUT_WARNING
if not _SHOWN_TIMEOUT_WARNING:
_SHOWN_TIMEOUT_WARNING = True
message = "Warning: Timeout is not available on main browser thread"
js.console.warn(message)
_SHOWN_STREAMING_WARNING = False
def _show_streaming_warning() -> None:
global _SHOWN_STREAMING_WARNING
if not _SHOWN_STREAMING_WARNING:
_SHOWN_STREAMING_WARNING = True
message = "Can't stream HTTP requests because: \n"
if not is_cross_origin_isolated():
message += " Page is not cross-origin isolated\n"
if is_in_browser_main_thread():
message += " Python is running in main browser thread\n"
if not is_worker_available():
message += " Worker or Blob classes are not available in this environment." # Defensive: this is always False in browsers that we test in
if streaming_ready() is False:
message += """ Streaming fetch worker isn't ready. If you want to be sure that streaming fetch
is working, you need to call: 'await urllib3.contrib.emscripten.fetch.wait_for_streaming_ready()`"""
from js import console
console.warn(message)
def send_request(request: EmscriptenRequest) -> EmscriptenResponse:
if has_jspi():
return send_jspi_request(request, False)
elif is_in_node():
raise _RequestError(
message=NODE_JSPI_ERROR,
request=request,
response=None,
)
try:
js_xhr = js.XMLHttpRequest.new()
if not is_in_browser_main_thread():
js_xhr.responseType = "arraybuffer"
if request.timeout:
js_xhr.timeout = int(request.timeout * 1000)
else:
js_xhr.overrideMimeType("text/plain; charset=ISO-8859-15")
if request.timeout:
# timeout isn't available on the main thread - show a warning in console
# if it is set
_show_timeout_warning()
js_xhr.open(request.method, request.url, False)
for name, value in request.headers.items():
if name.lower() not in HEADERS_TO_IGNORE:
js_xhr.setRequestHeader(name, value)
js_xhr.send(to_js(request.body))
headers = dict(Parser().parsestr(js_xhr.getAllResponseHeaders()))
if not is_in_browser_main_thread():
body = js_xhr.response.to_py().tobytes()
else:
body = js_xhr.response.encode("ISO-8859-15")
return EmscriptenResponse(
status_code=js_xhr.status, headers=headers, body=body, request=request
)
except JsException as err:
if err.name == "TimeoutError":
raise _TimeoutError(err.message, request=request)
elif err.name == "NetworkError":
raise _RequestError(err.message, request=request)
else:
# general http error
raise _RequestError(err.message, request=request)
def send_jspi_request(
request: EmscriptenRequest, streaming: bool
) -> EmscriptenResponse:
"""
Send a request using WebAssembly JavaScript Promise Integration
to wrap the asynchronous JavaScript fetch api (experimental).
:param request:
Request to send
:param streaming:
Whether to stream the response
:return: The response object
:rtype: EmscriptenResponse
"""
timeout = request.timeout
js_abort_controller = js.AbortController.new()
headers = {k: v for k, v in request.headers.items() if k not in HEADERS_TO_IGNORE}
req_body = request.body
fetch_data = {
"headers": headers,
"body": to_js(req_body),
"method": request.method,
"signal": js_abort_controller.signal,
}
# Node.js returns the whole response (unlike opaqueredirect in browsers),
# so urllib3 can set `redirect: manual` to control redirects itself.
# https://stackoverflow.com/a/78524615
if _is_node_js():
fetch_data["redirect"] = "manual"
# Call JavaScript fetch (async api, returns a promise)
fetcher_promise_js = js.fetch(request.url, _obj_from_dict(fetch_data))
# Now suspend WebAssembly until we resolve that promise
# or time out.
response_js = _run_sync_with_timeout(
fetcher_promise_js,
timeout,
js_abort_controller,
request=request,
response=None,
)
headers = {}
header_iter = response_js.headers.entries()
while True:
iter_value_js = header_iter.next()
if getattr(iter_value_js, "done", False):
break
else:
headers[str(iter_value_js.value[0])] = str(iter_value_js.value[1])
status_code = response_js.status
body: bytes | io.RawIOBase = b""
response = EmscriptenResponse(
status_code=status_code, headers=headers, body=b"", request=request
)
if streaming:
# get via inputstream
if response_js.body is not None:
# get a reader from the fetch response
body_stream_js = response_js.body.getReader()
body = _JSPIReadStream(
body_stream_js, timeout, request, response, js_abort_controller
)
else:
# get directly via arraybuffer
# n.b. this is another async JavaScript call.
body = _run_sync_with_timeout(
response_js.arrayBuffer(),
timeout,
js_abort_controller,
request=request,
response=response,
).to_py()
response.body = body
return response
def _run_sync_with_timeout(
promise: Any,
timeout: float,
js_abort_controller: Any,
request: EmscriptenRequest | None,
response: EmscriptenResponse | None,
) -> Any:
"""
Await a JavaScript promise synchronously with a timeout which is implemented
via the AbortController
:param promise:
Javascript promise to await
:param timeout:
Timeout in seconds
:param js_abort_controller:
A JavaScript AbortController object, used on timeout
:param request:
The request being handled
:param response:
The response being handled (if it exists yet)
:raises _TimeoutError: If the request times out
:raises _RequestError: If the request raises a JavaScript exception
:return: The result of awaiting the promise.
"""
timer_id = None
if timeout > 0:
timer_id = js.setTimeout(
js_abort_controller.abort.bind(js_abort_controller), int(timeout * 1000)
)
try:
from pyodide.ffi import run_sync
# run_sync here uses WebAssembly JavaScript Promise Integration to
# suspend python until the JavaScript promise resolves.
return run_sync(promise)
except JsException as err:
if err.name == "AbortError":
raise _TimeoutError(
message="Request timed out", request=request, response=response
)
else:
raise _RequestError(message=err.message, request=request, response=response)
finally:
if timer_id is not None:
js.clearTimeout(timer_id)
def has_jspi() -> bool:
"""
Return true if jspi can be used.
This requires both browser support and also WebAssembly
to be in the correct state - i.e. that the javascript
call into python was async not sync.
:return: True if jspi can be used.
:rtype: bool
"""
try:
from pyodide.ffi import can_run_sync, run_sync # noqa: F401
return bool(can_run_sync())
except ImportError:
return False
def _is_node_js() -> bool:
"""
Check if we are in Node.js.
:return: True if we are in Node.js.
:rtype: bool
"""
return (
hasattr(js, "process")
and hasattr(js.process, "release")
# According to the Node.js documentation, the release name is always "node".
and js.process.release.name == "node"
)
def streaming_ready() -> bool | None:
if _fetcher:
return _fetcher.streaming_ready
else:
return None # no fetcher, return None to signify that
async def wait_for_streaming_ready() -> bool:
if _fetcher:
await _fetcher.js_worker_ready_promise
return True
else:
return False
@@ -0,0 +1,22 @@
from __future__ import annotations
from dataclasses import dataclass, field
from ..._base_connection import _TYPE_BODY
@dataclass
class EmscriptenRequest:
method: str
url: str
params: dict[str, str] | None = None
body: _TYPE_BODY | None = None
headers: dict[str, str] = field(default_factory=dict)
timeout: float = 0
decode_content: bool = True
def set_header(self, name: str, value: str) -> None:
self.headers[name.capitalize()] = value
def set_body(self, body: _TYPE_BODY | None) -> None:
self.body = body
@@ -0,0 +1,281 @@
from __future__ import annotations
import json as _json
import logging
import typing
from contextlib import contextmanager
from dataclasses import dataclass
from http.client import HTTPException as HTTPException
from io import BytesIO, IOBase
from ...exceptions import InvalidHeader, TimeoutError
from ...response import BaseHTTPResponse
from ...util.retry import Retry
from .request import EmscriptenRequest
if typing.TYPE_CHECKING:
from ..._base_connection import BaseHTTPConnection, BaseHTTPSConnection
log = logging.getLogger(__name__)
@dataclass
class EmscriptenResponse:
status_code: int
headers: dict[str, str]
body: IOBase | bytes
request: EmscriptenRequest
class EmscriptenHttpResponseWrapper(BaseHTTPResponse):
def __init__(
self,
internal_response: EmscriptenResponse,
url: str | None = None,
connection: BaseHTTPConnection | BaseHTTPSConnection | None = None,
):
self._pool = None # set by pool class
self._body = None
self._uncached_read_occurred = False
self._response = internal_response
self._url = url
self._connection = connection
self._closed = False
super().__init__(
headers=internal_response.headers,
status=internal_response.status_code,
request_url=url,
version=0,
version_string="HTTP/?",
reason="",
decode_content=True,
)
self.length_remaining = self._init_length(self._response.request.method)
self.length_is_certain = False
@property
def url(self) -> str | None:
return self._url
@url.setter
def url(self, url: str | None) -> None:
self._url = url
@property
def connection(self) -> BaseHTTPConnection | BaseHTTPSConnection | None:
return self._connection
@property
def retries(self) -> Retry | None:
return self._retries
@retries.setter
def retries(self, retries: Retry | None) -> None:
# Override the request_url if retries has a redirect location.
self._retries = retries
def stream(
self, amt: int | None = 2**16, decode_content: bool | None = None
) -> typing.Generator[bytes]:
"""
A generator wrapper for the read() method. A call will block until
``amt`` bytes have been read from the connection or until the
connection is closed.
:param amt:
How much of the content to read. The generator will return up to
much data per iteration, but may return less. This is particularly
likely when using compressed data. However, the empty string will
never be returned.
:param decode_content:
If True, will attempt to decode the body based on the
'content-encoding' header.
"""
while True:
data = self.read(amt=amt, decode_content=decode_content)
if data:
yield data
else:
break
def _init_length(self, request_method: str | None) -> int | None:
length: int | None
content_length: str | None = self.headers.get("content-length")
if content_length is not None:
try:
# RFC 7230 section 3.3.2 specifies multiple content lengths can
# be sent in a single Content-Length header
# (e.g. Content-Length: 42, 42). This line ensures the values
# are all valid ints and that as long as the `set` length is 1,
# all values are the same. Otherwise, the header is invalid.
lengths = {int(val) for val in content_length.split(",")}
if len(lengths) > 1:
raise InvalidHeader(
"Content-Length contained multiple "
"unmatching values (%s)" % content_length
)
length = lengths.pop()
except ValueError:
length = None
else:
if length < 0:
length = None
else: # if content_length is None
length = None
# Check for responses that shouldn't include a body
if (
self.status in (204, 304)
or 100 <= self.status < 200
or request_method == "HEAD"
):
length = 0
return length
def read(
self,
amt: int | None = None,
decode_content: bool | None = None, # ignored because browser decodes always
cache_content: bool = False,
) -> bytes:
if (
self._closed
or self._response is None
or (isinstance(self._response.body, IOBase) and self._response.body.closed)
):
return b""
with self._error_catcher():
# body has been preloaded as a string by XmlHttpRequest
if not isinstance(self._response.body, IOBase):
self.length_remaining = len(self._response.body)
self.length_is_certain = True
# wrap body in IOStream
self._response.body = BytesIO(self._response.body)
if amt is not None and amt >= 0:
# don't cache partial content
cache_content = False
data = self._response.body.read(amt)
self._uncached_read_occurred = True
else: # read all we can (and cache it)
data = self._response.body.read()
if cache_content and not self._uncached_read_occurred:
self._body = data
else:
self._uncached_read_occurred = True
if self.length_remaining is not None:
self.length_remaining = max(self.length_remaining - len(data), 0)
if len(data) == 0 or (
self.length_is_certain and self.length_remaining == 0
):
# definitely finished reading, close response stream
self._response.body.close()
return typing.cast(bytes, data)
def read_chunked(
self,
amt: int | None = None,
decode_content: bool | None = None,
) -> typing.Generator[bytes]:
# chunked is handled by browser
while True:
bytes = self.read(amt, decode_content)
if not bytes:
break
yield bytes
def release_conn(self) -> None:
if not self._pool or not self._connection:
return None
self._pool._put_conn(self._connection)
self._connection = None
def drain_conn(self) -> None:
self.close()
@property
def data(self) -> bytes:
if self._body:
return self._body
else:
return self.read(cache_content=True)
def json(self) -> typing.Any:
"""
Deserializes the body of the HTTP response as a Python object.
The body of the HTTP response must be encoded using UTF-8, as per
`RFC 8529 Section 8.1 <https://www.rfc-editor.org/rfc/rfc8259#section-8.1>`_.
To use a custom JSON decoder pass the result of :attr:`HTTPResponse.data` to
your custom decoder instead.
If the body of the HTTP response is not decodable to UTF-8, a
`UnicodeDecodeError` will be raised. If the body of the HTTP response is not a
valid JSON document, a `json.JSONDecodeError` will be raised.
Read more :ref:`here <json_content>`.
:returns: The body of the HTTP response as a Python object.
"""
data = self.data.decode("utf-8")
return _json.loads(data)
def close(self) -> None:
if not self._closed:
if isinstance(self._response.body, IOBase):
self._response.body.close()
if self._connection:
self._connection.close()
self._connection = None
self._closed = True
@contextmanager
def _error_catcher(self) -> typing.Generator[None]:
"""
Catch Emscripten specific exceptions thrown by fetch.py,
instead re-raising urllib3 variants, so that low-level exceptions
are not leaked in the high-level api.
On exit, release the connection back to the pool.
"""
from .fetch import _RequestError, _TimeoutError # avoid circular import
clean_exit = False
try:
yield
# If no exception is thrown, we should avoid cleaning up
# unnecessarily.
clean_exit = True
except _TimeoutError as e:
raise TimeoutError(str(e))
except _RequestError as e:
raise HTTPException(str(e))
finally:
# If we didn't terminate cleanly, we need to throw away our
# connection.
if not clean_exit:
# The response may not be closed but we're not going to use it
# anymore so close it now
if (
isinstance(self._response.body, IOBase)
and not self._response.body.closed
):
self._response.body.close()
# release the connection back to the pool
self.release_conn()
else:
# If we have read everything from the response stream,
# return the connection back to the pool.
if (
isinstance(self._response.body, IOBase)
and self._response.body.closed
):
self.release_conn()
@@ -0,0 +1,563 @@
"""
Module for using pyOpenSSL as a TLS backend. This module was relevant before
the standard library ``ssl`` module supported SNI, but now that we've dropped
support for Python 2.7 all relevant Python versions support SNI so
**this module is no longer recommended**.
This needs the following packages installed:
* `pyOpenSSL`_ (tested with 19.0.0)
* `cryptography`_ (minimum 2.3, from pyopenssl)
* `idna`_ (minimum 2.1, from cryptography)
However, pyOpenSSL depends on cryptography, so while we use all three directly here we
end up having relatively few packages required.
You can install them with the following command:
.. code-block:: bash
$ python -m pip install pyopenssl cryptography idna
To activate certificate checking, call
:func:`~urllib3.contrib.pyopenssl.inject_into_urllib3` from your Python code
before you begin making HTTP requests. This can be done in a ``sitecustomize``
module, or at any other time before your application begins using ``urllib3``,
like this:
.. code-block:: python
try:
import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()
except ImportError:
pass
.. _pyopenssl: https://www.pyopenssl.org
.. _cryptography: https://cryptography.io
.. _idna: https://github.com/kjd/idna
"""
from __future__ import annotations
import OpenSSL.SSL # type: ignore[import-not-found]
from cryptography import x509
try:
from cryptography.x509 import UnsupportedExtension # type: ignore[attr-defined]
except ImportError:
# UnsupportedExtension is gone in cryptography >= 2.1.0
class UnsupportedExtension(Exception): # type: ignore[no-redef]
pass
import logging
import ssl
import typing
from io import BytesIO
from socket import socket as socket_cls
from .. import util
if typing.TYPE_CHECKING:
from OpenSSL.crypto import X509 # type: ignore[import-not-found]
__all__ = ["inject_into_urllib3", "extract_from_urllib3"]
# Map from urllib3 to PyOpenSSL compatible parameter-values.
_openssl_versions: dict[int, int] = {
util.ssl_.PROTOCOL_TLS: OpenSSL.SSL.SSLv23_METHOD, # type: ignore[attr-defined]
util.ssl_.PROTOCOL_TLS_CLIENT: OpenSSL.SSL.SSLv23_METHOD, # type: ignore[attr-defined]
ssl.PROTOCOL_TLSv1: OpenSSL.SSL.TLSv1_METHOD,
}
if hasattr(ssl, "PROTOCOL_TLSv1_1") and hasattr(OpenSSL.SSL, "TLSv1_1_METHOD"):
_openssl_versions[ssl.PROTOCOL_TLSv1_1] = OpenSSL.SSL.TLSv1_1_METHOD
if hasattr(ssl, "PROTOCOL_TLSv1_2") and hasattr(OpenSSL.SSL, "TLSv1_2_METHOD"):
_openssl_versions[ssl.PROTOCOL_TLSv1_2] = OpenSSL.SSL.TLSv1_2_METHOD
_stdlib_to_openssl_verify = {
ssl.CERT_NONE: OpenSSL.SSL.VERIFY_NONE,
ssl.CERT_OPTIONAL: OpenSSL.SSL.VERIFY_PEER,
ssl.CERT_REQUIRED: OpenSSL.SSL.VERIFY_PEER
+ OpenSSL.SSL.VERIFY_FAIL_IF_NO_PEER_CERT,
}
_openssl_to_stdlib_verify = {v: k for k, v in _stdlib_to_openssl_verify.items()}
# The SSLvX values are the most likely to be missing in the future
# but we check them all just to be sure.
_OP_NO_SSLv2_OR_SSLv3: int = getattr(OpenSSL.SSL, "OP_NO_SSLv2", 0) | getattr(
OpenSSL.SSL, "OP_NO_SSLv3", 0
)
_OP_NO_TLSv1: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1", 0)
_OP_NO_TLSv1_1: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1_1", 0)
_OP_NO_TLSv1_2: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1_2", 0)
_OP_NO_TLSv1_3: int = getattr(OpenSSL.SSL, "OP_NO_TLSv1_3", 0)
_openssl_to_ssl_minimum_version: dict[int, int] = {
ssl.TLSVersion.MINIMUM_SUPPORTED: _OP_NO_SSLv2_OR_SSLv3,
ssl.TLSVersion.TLSv1: _OP_NO_SSLv2_OR_SSLv3,
ssl.TLSVersion.TLSv1_1: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1,
ssl.TLSVersion.TLSv1_2: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1 | _OP_NO_TLSv1_1,
ssl.TLSVersion.TLSv1_3: (
_OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1 | _OP_NO_TLSv1_1 | _OP_NO_TLSv1_2
),
ssl.TLSVersion.MAXIMUM_SUPPORTED: (
_OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1 | _OP_NO_TLSv1_1 | _OP_NO_TLSv1_2
),
}
_openssl_to_ssl_maximum_version: dict[int, int] = {
ssl.TLSVersion.MINIMUM_SUPPORTED: (
_OP_NO_SSLv2_OR_SSLv3
| _OP_NO_TLSv1
| _OP_NO_TLSv1_1
| _OP_NO_TLSv1_2
| _OP_NO_TLSv1_3
),
ssl.TLSVersion.TLSv1: (
_OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1_1 | _OP_NO_TLSv1_2 | _OP_NO_TLSv1_3
),
ssl.TLSVersion.TLSv1_1: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1_2 | _OP_NO_TLSv1_3,
ssl.TLSVersion.TLSv1_2: _OP_NO_SSLv2_OR_SSLv3 | _OP_NO_TLSv1_3,
ssl.TLSVersion.TLSv1_3: _OP_NO_SSLv2_OR_SSLv3,
ssl.TLSVersion.MAXIMUM_SUPPORTED: _OP_NO_SSLv2_OR_SSLv3,
}
# OpenSSL will only write 16K at a time
SSL_WRITE_BLOCKSIZE = 16384
orig_util_SSLContext = util.ssl_.SSLContext
log = logging.getLogger(__name__)
def inject_into_urllib3() -> None:
"Monkey-patch urllib3 with PyOpenSSL-backed SSL-support."
_validate_dependencies_met()
util.SSLContext = PyOpenSSLContext # type: ignore[assignment]
util.ssl_.SSLContext = PyOpenSSLContext # type: ignore[assignment]
util.IS_PYOPENSSL = True
util.ssl_.IS_PYOPENSSL = True
def extract_from_urllib3() -> None:
"Undo monkey-patching by :func:`inject_into_urllib3`."
util.SSLContext = orig_util_SSLContext
util.ssl_.SSLContext = orig_util_SSLContext
util.IS_PYOPENSSL = False
util.ssl_.IS_PYOPENSSL = False
def _validate_dependencies_met() -> None:
"""
Verifies that PyOpenSSL's package-level dependencies have been met.
Throws `ImportError` if they are not met.
"""
# Method added in `cryptography==1.1`; not available in older versions
from cryptography.x509.extensions import Extensions
if getattr(Extensions, "get_extension_for_class", None) is None:
raise ImportError(
"'cryptography' module missing required functionality. "
"Try upgrading to v1.3.4 or newer."
)
# pyOpenSSL 0.14 and above use cryptography for OpenSSL bindings. The _x509
# attribute is only present on those versions.
from OpenSSL.crypto import X509
x509 = X509()
if getattr(x509, "_x509", None) is None:
raise ImportError(
"'pyOpenSSL' module missing required functionality. "
"Try upgrading to v0.14 or newer."
)
def _dnsname_to_stdlib(name: str) -> str | None:
"""
Converts a dNSName SubjectAlternativeName field to the form used by the
standard library on the given Python version.
Cryptography produces a dNSName as a unicode string that was idna-decoded
from ASCII bytes. We need to idna-encode that string to get it back, and
then on Python 3 we also need to convert to unicode via UTF-8 (the stdlib
uses PyUnicode_FromStringAndSize on it, which decodes via UTF-8).
If the name cannot be idna-encoded then we return None signalling that
the name given should be skipped.
"""
def idna_encode(name: str) -> bytes | None:
"""
Borrowed wholesale from the Python Cryptography Project. It turns out
that we can't just safely call `idna.encode`: it can explode for
wildcard names. This avoids that problem.
"""
import idna
try:
for prefix in ["*.", "."]:
if name.startswith(prefix):
name = name[len(prefix) :]
return prefix.encode("ascii") + idna.encode(name)
return idna.encode(name)
except idna.core.IDNAError:
return None
# Don't send IPv6 addresses through the IDNA encoder.
if ":" in name:
return name
encoded_name = idna_encode(name)
if encoded_name is None:
return None
return encoded_name.decode("utf-8")
def get_subj_alt_name(peer_cert: X509) -> list[tuple[str, str]]:
"""
Given an PyOpenSSL certificate, provides all the subject alternative names.
"""
cert = peer_cert.to_cryptography()
# We want to find the SAN extension. Ask Cryptography to locate it (it's
# faster than looping in Python)
try:
ext = cert.extensions.get_extension_for_class(x509.SubjectAlternativeName).value
except x509.ExtensionNotFound:
# No such extension, return the empty list.
return []
except (
x509.DuplicateExtension,
UnsupportedExtension,
x509.UnsupportedGeneralNameType,
UnicodeError,
) as e:
# A problem has been found with the quality of the certificate. Assume
# no SAN field is present.
log.warning(
"A problem was encountered with the certificate that prevented "
"urllib3 from finding the SubjectAlternativeName field. This can "
"affect certificate validation. The error was %s",
e,
)
return []
# We want to return dNSName and iPAddress fields. We need to cast the IPs
# back to strings because the match_hostname function wants them as
# strings.
# Sadly the DNS names need to be idna encoded and then, on Python 3, UTF-8
# decoded. This is pretty frustrating, but that's what the standard library
# does with certificates, and so we need to attempt to do the same.
# We also want to skip over names which cannot be idna encoded.
names = [
("DNS", name)
for name in map(_dnsname_to_stdlib, ext.get_values_for_type(x509.DNSName))
if name is not None
]
names.extend(
("IP Address", str(name)) for name in ext.get_values_for_type(x509.IPAddress)
)
return names
class WrappedSocket:
"""API-compatibility wrapper for Python OpenSSL's Connection-class."""
def __init__(
self,
connection: OpenSSL.SSL.Connection,
socket: socket_cls,
suppress_ragged_eofs: bool = True,
) -> None:
self.connection = connection
self.socket = socket
self.suppress_ragged_eofs = suppress_ragged_eofs
self._io_refs = 0
self._closed = False
def fileno(self) -> int:
return self.socket.fileno()
# Copy-pasted from Python 3.5 source code
def _decref_socketios(self) -> None:
if self._io_refs > 0:
self._io_refs -= 1
if self._closed:
self.close()
def recv(self, *args: typing.Any, **kwargs: typing.Any) -> bytes:
try:
data = self.connection.recv(*args, **kwargs)
except OpenSSL.SSL.SysCallError as e:
if self.suppress_ragged_eofs and e.args == (-1, "Unexpected EOF"):
return b""
else:
raise OSError(e.args[0], str(e)) from e
except OpenSSL.SSL.ZeroReturnError:
if self.connection.get_shutdown() == OpenSSL.SSL.RECEIVED_SHUTDOWN:
return b""
else:
raise
except OpenSSL.SSL.WantReadError as e:
if not util.wait_for_read(self.socket, self.socket.gettimeout()):
raise TimeoutError("The read operation timed out") from e
else:
return self.recv(*args, **kwargs)
# TLS 1.3 post-handshake authentication
except OpenSSL.SSL.Error as e:
raise ssl.SSLError(f"read error: {e!r}") from e
else:
return data # type: ignore[no-any-return]
def recv_into(self, *args: typing.Any, **kwargs: typing.Any) -> int:
try:
return self.connection.recv_into(*args, **kwargs) # type: ignore[no-any-return]
except OpenSSL.SSL.SysCallError as e:
if self.suppress_ragged_eofs and e.args == (-1, "Unexpected EOF"):
return 0
else:
raise OSError(e.args[0], str(e)) from e
except OpenSSL.SSL.ZeroReturnError:
if self.connection.get_shutdown() == OpenSSL.SSL.RECEIVED_SHUTDOWN:
return 0
else:
raise
except OpenSSL.SSL.WantReadError as e:
if not util.wait_for_read(self.socket, self.socket.gettimeout()):
raise TimeoutError("The read operation timed out") from e
else:
return self.recv_into(*args, **kwargs)
# TLS 1.3 post-handshake authentication
except OpenSSL.SSL.Error as e:
raise ssl.SSLError(f"read error: {e!r}") from e
def settimeout(self, timeout: float) -> None:
return self.socket.settimeout(timeout)
def _send_until_done(self, data: bytes) -> int:
while True:
try:
return self.connection.send(data) # type: ignore[no-any-return]
except OpenSSL.SSL.WantWriteError as e:
if not util.wait_for_write(self.socket, self.socket.gettimeout()):
raise TimeoutError() from e
continue
except OpenSSL.SSL.SysCallError as e:
raise OSError(e.args[0], str(e)) from e
def sendall(self, data: bytes) -> None:
total_sent = 0
while total_sent < len(data):
sent = self._send_until_done(
data[total_sent : total_sent + SSL_WRITE_BLOCKSIZE]
)
total_sent += sent
def shutdown(self, how: int) -> None:
try:
self.connection.shutdown()
except OpenSSL.SSL.Error as e:
raise ssl.SSLError(f"shutdown error: {e!r}") from e
def close(self) -> None:
self._closed = True
if self._io_refs <= 0:
self._real_close()
def _real_close(self) -> None:
try:
return self.connection.close() # type: ignore[no-any-return]
except OpenSSL.SSL.Error:
return
def getpeercert(
self, binary_form: bool = False
) -> dict[str, list[typing.Any]] | None:
x509 = self.connection.get_peer_certificate()
if not x509:
return x509 # type: ignore[no-any-return]
if binary_form:
return OpenSSL.crypto.dump_certificate(OpenSSL.crypto.FILETYPE_ASN1, x509) # type: ignore[no-any-return]
return {
"subject": ((("commonName", x509.get_subject().CN),),), # type: ignore[dict-item]
"subjectAltName": get_subj_alt_name(x509),
}
def version(self) -> str:
return self.connection.get_protocol_version_name() # type: ignore[no-any-return]
def selected_alpn_protocol(self) -> str | None:
alpn_proto = self.connection.get_alpn_proto_negotiated()
return alpn_proto.decode() if alpn_proto else None
WrappedSocket.makefile = socket_cls.makefile # type: ignore[attr-defined]
class PyOpenSSLContext:
"""
I am a wrapper class for the PyOpenSSL ``Context`` object. I am responsible
for translating the interface of the standard library ``SSLContext`` object
to calls into PyOpenSSL.
"""
def __init__(self, protocol: int) -> None:
self.protocol = _openssl_versions[protocol]
self._ctx = OpenSSL.SSL.Context(self.protocol)
self._options = 0
self.check_hostname = False
self._minimum_version: int = ssl.TLSVersion.MINIMUM_SUPPORTED
self._maximum_version: int = ssl.TLSVersion.MAXIMUM_SUPPORTED
self._verify_flags: int = ssl.VERIFY_X509_TRUSTED_FIRST
@property
def options(self) -> int:
return self._options
@options.setter
def options(self, value: int) -> None:
self._options = value
self._set_ctx_options()
@property
def verify_flags(self) -> int:
return self._verify_flags
@verify_flags.setter
def verify_flags(self, value: int) -> None:
self._verify_flags = value
self._ctx.get_cert_store().set_flags(self._verify_flags)
@property
def verify_mode(self) -> int:
return _openssl_to_stdlib_verify[self._ctx.get_verify_mode()]
@verify_mode.setter
def verify_mode(self, value: ssl.VerifyMode) -> None:
self._ctx.set_verify(_stdlib_to_openssl_verify[value], _verify_callback)
def set_default_verify_paths(self) -> None:
self._ctx.set_default_verify_paths()
def set_ciphers(self, ciphers: bytes | str) -> None:
if isinstance(ciphers, str):
ciphers = ciphers.encode("utf-8")
self._ctx.set_cipher_list(ciphers)
def load_verify_locations(
self,
cafile: str | None = None,
capath: str | None = None,
cadata: bytes | None = None,
) -> None:
if cafile is not None:
cafile = cafile.encode("utf-8") # type: ignore[assignment]
if capath is not None:
capath = capath.encode("utf-8") # type: ignore[assignment]
try:
self._ctx.load_verify_locations(cafile, capath)
if cadata is not None:
self._ctx.load_verify_locations(BytesIO(cadata))
except OpenSSL.SSL.Error as e:
raise ssl.SSLError(f"unable to load trusted certificates: {e!r}") from e
def load_cert_chain(
self,
certfile: str,
keyfile: str | None = None,
password: str | None = None,
) -> None:
try:
self._ctx.use_certificate_chain_file(certfile)
if password is not None:
if not isinstance(password, bytes):
password = password.encode("utf-8") # type: ignore[assignment]
self._ctx.set_passwd_cb(lambda *_: password)
self._ctx.use_privatekey_file(keyfile or certfile)
except OpenSSL.SSL.Error as e:
raise ssl.SSLError(f"Unable to load certificate chain: {e!r}") from e
def set_alpn_protocols(self, protocols: list[bytes | str]) -> None:
protocols = [util.util.to_bytes(p, "ascii") for p in protocols]
return self._ctx.set_alpn_protos(protocols) # type: ignore[no-any-return]
def wrap_socket(
self,
sock: socket_cls,
server_side: bool = False,
do_handshake_on_connect: bool = True,
suppress_ragged_eofs: bool = True,
server_hostname: bytes | str | None = None,
) -> WrappedSocket:
cnx = OpenSSL.SSL.Connection(self._ctx, sock)
# If server_hostname is an IP, don't use it for SNI, per RFC6066 Section 3
if server_hostname and not util.ssl_.is_ipaddress(server_hostname):
if isinstance(server_hostname, str):
server_hostname = server_hostname.encode("utf-8")
cnx.set_tlsext_host_name(server_hostname)
cnx.set_connect_state()
while True:
try:
cnx.do_handshake()
except OpenSSL.SSL.WantReadError as e:
if not util.wait_for_read(sock, sock.gettimeout()):
raise TimeoutError("select timed out") from e
continue
except OpenSSL.SSL.Error as e:
raise ssl.SSLError(f"bad handshake: {e!r}") from e
break
return WrappedSocket(cnx, sock)
def _set_ctx_options(self) -> None:
self._ctx.set_options(
self._options
| _openssl_to_ssl_minimum_version[self._minimum_version]
| _openssl_to_ssl_maximum_version[self._maximum_version]
)
@property
def minimum_version(self) -> int:
return self._minimum_version
@minimum_version.setter
def minimum_version(self, minimum_version: int) -> None:
self._minimum_version = minimum_version
self._set_ctx_options()
@property
def maximum_version(self) -> int:
return self._maximum_version
@maximum_version.setter
def maximum_version(self, maximum_version: int) -> None:
self._maximum_version = maximum_version
self._set_ctx_options()
def _verify_callback(
cnx: OpenSSL.SSL.Connection,
x509: X509,
err_no: int,
err_depth: int,
return_code: int,
) -> bool:
return err_no == 0
@@ -0,0 +1,228 @@
"""
This module contains provisional support for SOCKS proxies from within
urllib3. This module supports SOCKS4, SOCKS4A (an extension of SOCKS4), and
SOCKS5. To enable its functionality, either install PySocks or install this
module with the ``socks`` extra.
The SOCKS implementation supports the full range of urllib3 features. It also
supports the following SOCKS features:
- SOCKS4A (``proxy_url='socks4a://...``)
- SOCKS4 (``proxy_url='socks4://...``)
- SOCKS5 with remote DNS (``proxy_url='socks5h://...``)
- SOCKS5 with local DNS (``proxy_url='socks5://...``)
- Usernames and passwords for the SOCKS proxy
.. note::
It is recommended to use ``socks5h://`` or ``socks4a://`` schemes in
your ``proxy_url`` to ensure that DNS resolution is done from the remote
server instead of client-side when connecting to a domain name.
SOCKS4 supports IPv4 and domain names with the SOCKS4A extension. SOCKS5
supports IPv4, IPv6, and domain names.
When connecting to a SOCKS4 proxy the ``username`` portion of the ``proxy_url``
will be sent as the ``userid`` section of the SOCKS request:
.. code-block:: python
proxy_url="socks4a://<userid>@proxy-host"
When connecting to a SOCKS5 proxy the ``username`` and ``password`` portion
of the ``proxy_url`` will be sent as the username/password to authenticate
with the proxy:
.. code-block:: python
proxy_url="socks5h://<username>:<password>@proxy-host"
"""
from __future__ import annotations
try:
import socks # type: ignore[import-untyped]
except ImportError:
import warnings
from ..exceptions import DependencyWarning
warnings.warn(
(
"SOCKS support in urllib3 requires the installation of optional "
"dependencies: specifically, PySocks. For more information, see "
"https://urllib3.readthedocs.io/en/latest/advanced-usage.html#socks-proxies"
),
DependencyWarning,
)
raise
import typing
from socket import timeout as SocketTimeout
from ..connection import HTTPConnection, HTTPSConnection
from ..connectionpool import HTTPConnectionPool, HTTPSConnectionPool
from ..exceptions import ConnectTimeoutError, NewConnectionError
from ..poolmanager import PoolManager
from ..util.url import parse_url
try:
import ssl
except ImportError:
ssl = None # type: ignore[assignment]
class _TYPE_SOCKS_OPTIONS(typing.TypedDict):
socks_version: int
proxy_host: str | None
proxy_port: str | None
username: str | None
password: str | None
rdns: bool
class SOCKSConnection(HTTPConnection):
"""
A plain-text HTTP connection that connects via a SOCKS proxy.
"""
def __init__(
self,
_socks_options: _TYPE_SOCKS_OPTIONS,
*args: typing.Any,
**kwargs: typing.Any,
) -> None:
self._socks_options = _socks_options
super().__init__(*args, **kwargs)
def _new_conn(self) -> socks.socksocket:
"""
Establish a new connection via the SOCKS proxy.
"""
extra_kw: dict[str, typing.Any] = {}
if self.source_address:
extra_kw["source_address"] = self.source_address
if self.socket_options:
extra_kw["socket_options"] = self.socket_options
try:
conn = socks.create_connection(
(self.host, self.port),
proxy_type=self._socks_options["socks_version"],
proxy_addr=self._socks_options["proxy_host"],
proxy_port=self._socks_options["proxy_port"],
proxy_username=self._socks_options["username"],
proxy_password=self._socks_options["password"],
proxy_rdns=self._socks_options["rdns"],
timeout=self.timeout,
**extra_kw,
)
except SocketTimeout as e:
raise ConnectTimeoutError(
self,
f"Connection to {self.host} timed out. (connect timeout={self.timeout})",
) from e
except socks.ProxyError as e:
# This is fragile as hell, but it seems to be the only way to raise
# useful errors here.
if e.socket_err:
error = e.socket_err
if isinstance(error, SocketTimeout):
raise ConnectTimeoutError(
self,
f"Connection to {self.host} timed out. (connect timeout={self.timeout})",
) from e
else:
# Adding `from e` messes with coverage somehow, so it's omitted.
# See #2386.
raise NewConnectionError(
self, f"Failed to establish a new connection: {error}"
)
else: # Defensive: see https://github.com/urllib3/urllib3/pull/3728#pullrequestreview-3816302703
raise NewConnectionError(
self, f"Failed to establish a new connection: {e}"
) from e
except OSError as e: # Defensive: PySocks should catch all these.
raise NewConnectionError(
self, f"Failed to establish a new connection: {e}"
) from e
return conn
# We don't need to duplicate the Verified/Unverified distinction from
# urllib3/connection.py here because the HTTPSConnection will already have been
# correctly set to either the Verified or Unverified form by that module. This
# means the SOCKSHTTPSConnection will automatically be the correct type.
class SOCKSHTTPSConnection(SOCKSConnection, HTTPSConnection):
pass
class SOCKSHTTPConnectionPool(HTTPConnectionPool):
ConnectionCls = SOCKSConnection
class SOCKSHTTPSConnectionPool(HTTPSConnectionPool):
ConnectionCls = SOCKSHTTPSConnection
class SOCKSProxyManager(PoolManager):
"""
A version of the urllib3 ProxyManager that routes connections via the
defined SOCKS proxy.
"""
pool_classes_by_scheme = {
"http": SOCKSHTTPConnectionPool,
"https": SOCKSHTTPSConnectionPool,
}
def __init__(
self,
proxy_url: str,
username: str | None = None,
password: str | None = None,
num_pools: int = 10,
headers: typing.Mapping[str, str] | None = None,
**connection_pool_kw: typing.Any,
):
parsed = parse_url(proxy_url)
if username is None and password is None and parsed.auth is not None:
split = parsed.auth.split(":")
if len(split) == 2:
username, password = split
if parsed.scheme == "socks5":
socks_version = socks.PROXY_TYPE_SOCKS5
rdns = False
elif parsed.scheme == "socks5h":
socks_version = socks.PROXY_TYPE_SOCKS5
rdns = True
elif parsed.scheme == "socks4":
socks_version = socks.PROXY_TYPE_SOCKS4
rdns = False
elif parsed.scheme == "socks4a":
socks_version = socks.PROXY_TYPE_SOCKS4
rdns = True
else:
raise ValueError(f"Unable to determine SOCKS version from {proxy_url}")
self.proxy_url = proxy_url
socks_options = {
"socks_version": socks_version,
"proxy_host": parsed.host,
"proxy_port": parsed.port,
"username": username,
"password": password,
"rdns": rdns,
}
connection_pool_kw["_socks_options"] = socks_options
super().__init__(num_pools, headers, **connection_pool_kw)
self.pool_classes_by_scheme = SOCKSProxyManager.pool_classes_by_scheme

Some files were not shown because too many files have changed in this diff Show More