Docs: align with code; document json=, per-request callbacks, and the new settings

Reconcile the docs with the reviewed code.

Fixes: non-existent settings (RETRY_TIMES, RANDOMIZE_DOWNLOAD_DELAY, DOWNLOAD_DELAY,
the old query-param settings); the MiddlewareResult factory API; signal handler
kwargs and request_failed/headers_received semantics; ALLOWED_DOMAINS -> allowed_domains;
Scheduler.get(); the RequestQueue ABC surface; response.text() as a method; the
inverted crawl-priority direction; the XML exporter's stringified list output; the
camoufox_page_event_handlers meta key; env-var vs Settings nesting; and the
StatsCollector verb API.

Documents the now-real capabilities: json= bodies, per-request callbacks
(callback/cb_kwargs), RANDOMIZE_DELAY, the RETRY_* settings, COOKIES_ENABLED/
RETRY_ENABLED, IGNORE/KEEP_QUERY_PARAMS, and follow(headers=). Multi-step auth
examples now use callbacks + json=.

Author: vasiliyk

docs/advanced-topics/anti_bot_evasion.md

@@ -35,11 +35,8 @@ custom_settings = {
     # Delay between requests to same domain (seconds)
     "DELAY_PER_DOMAIN": 2.0,
 
-    # Random delay range (adds randomness to delays)
-    "RANDOMIZE_DOWNLOAD_DELAY": True,
-
-    # Download delay (global, across all domains)
-    "DOWNLOAD_DELAY": 1.0,
+    # Jitter the per-domain delay to 50-150% per request
+    "RANDOMIZE_DELAY": True,
 }
 ```
 
@@ -48,8 +45,7 @@ custom_settings = {
 - `CONCURRENCY`: Maximum concurrent requests across all domains
 - `CONCURRENCY_PER_DOMAIN`: Maximum concurrent requests per domain (most important for anti-bot)
 - `DELAY_PER_DOMAIN`: Minimum delay between requests to same domain
-- `RANDOMIZE_DOWNLOAD_DELAY`: Adds random variance to delays (0.5x to 1.5x)
-- `DOWNLOAD_DELAY`: Global minimum delay between any requests
+- `RANDOMIZE_DELAY`: Jitters `DELAY_PER_DOMAIN` to 50-150% per request
 
 **Example Spider:**
 
@@ -62,7 +58,7 @@ class PoliteSpider(Spider):
     custom_settings = {
         "CONCURRENCY_PER_DOMAIN": 1,  # One request at a time per domain
         "DELAY_PER_DOMAIN": 3.0,       # 3 seconds between requests
-        "RANDOMIZE_DOWNLOAD_DELAY": True,  # Vary timing naturally
+        "RANDOMIZE_DELAY": True,  # Vary timing naturally
     }
 ```
 
@@ -217,7 +213,7 @@ custom_settings = {
     },
 
     # Retry configuration
-    "RETRY_TIMES": 3,  # Maximum retry attempts
+    "MAX_RETRIES": 3,  # Maximum retry attempts
 
     # HTTP status codes that trigger retry
     "RETRY_HTTP_CODES": [500, 502, 503, 504, 408, 429],
@@ -232,7 +228,7 @@ custom_settings = {
 1. Request fails with retryable status code (e.g., 503 Service Unavailable)
 2. Request is re-queued with lower priority
 3. Natural delay occurs before retry (due to queue processing)
-4. Process repeats up to `RETRY_TIMES` attempts
+4. Process repeats up to `MAX_RETRIES` attempts
 
 **Custom Retry Logic:**
 
@@ -401,7 +397,7 @@ class StealthSpider(Spider):
         "CONCURRENCY": 2,
         "CONCURRENCY_PER_DOMAIN": 1,
         "DELAY_PER_DOMAIN": 3.0,
-        "RANDOMIZE_DOWNLOAD_DELAY": True,
+        "RANDOMIZE_DELAY": True,
 
         # Headers (qCrawl framework-level)
         "DEFAULT_REQUEST_HEADERS": {
@@ -412,7 +408,7 @@ class StealthSpider(Spider):
         },
 
         # Retry with backoff (qCrawl framework-level)
-        "RETRY_TIMES": 3,
+        "MAX_RETRIES": 3,
         "RETRY_HTTP_CODES": [500, 502, 503, 504, 408, 429],
 
         # Middlewares (qCrawl framework-level)
@@ -568,7 +564,7 @@ if should_scrape_now():
 
 ```python
 async def parse(self, response):
-    if "captcha" in response.text.lower():
+    if "captcha" in response.text().lower()():
         self.logger.warning(f"CAPTCHA detected at {response.url}")
         # Option 1: Stop scraping
         return
@@ -601,7 +597,7 @@ class AdaptiveSpider(Spider):
                 self.detection_count -= 1
 
     def is_detected(self, response):
-        return response.status_code in [429, 403] or "captcha" in response.text.lower()
+        return response.status_code in [429, 403] or "captcha" in response.text().lower()()
 ```
 
 ## Common Anti-Bot Patterns and Solutions
@@ -616,7 +612,7 @@ custom_settings = {
     "DELAY_PER_DOMAIN": 5.0,
     "CONCURRENCY_PER_DOMAIN": 1,
     "RETRY_HTTP_CODES": [429],
-    "RETRY_TIMES": 5,
+    "MAX_RETRIES": 5,
 }
 ```
 
@@ -689,8 +685,8 @@ async def start_requests(self):
     yield Request(
         url="https://example.com/login",
         method="POST",
-        body={"username": "...", "password": "..."},
-        callback=self.after_login
+        json={"username": "...", "password": "..."},
+        callback=self.after_login,
     )
 
 async def after_login(self, response):
@@ -744,7 +740,7 @@ qCrawl provides comprehensive anti-bot evasion capabilities:
 | User Agent Rotation | Avoid fingerprinting    | Per-request headers                          |
 | Cookie Management   | Session handling        | `CookiesMiddleware` (automatic)              |
 | Proxy Support       | IP rotation             | `HttpProxyMiddleware`, per-request meta      |
-| Retry Strategy      | Handle transient errors | `RETRY_TIMES`, `RETRY_HTTP_CODES`            |
+| Retry Strategy      | Handle transient errors | `MAX_RETRIES`, `RETRY_HTTP_CODES`            |
 | Custom Headers      | Appear legitimate       | `DEFAULT_REQUEST_HEADERS`, per-request       |
 | Navigation Patterns | Human-like behavior     | PageMethod with Camoufox                     |
 

docs/advanced-topics/authentication.md

@@ -18,28 +18,30 @@ class AuthSpider(Spider):
     }
 
     async def parse(self, response):
+        # The login page: extract the CSRF token and submit the form.
         rv = self.response_view(response)
-
-        # Extract CSRF token
         csrf_token = rv.doc.cssselect("input[name=csrf_token]")[0].get("value")
 
-        # Submit login form
+        # Form login uses x-www-form-urlencoded, so encode the body (bytes).
+        from urllib.parse import urlencode
+
         yield Request(
             url="https://example.com/login",
             method="POST",
-            body={
-                "username": "user@example.com",
-                "password": "password123",
-                "csrf_token": csrf_token
-            },
-            meta={"next_action": "start_crawl"}
+            body=urlencode(
+                {
+                    "username": "user@example.com",
+                    "password": "password123",
+                    "csrf_token": csrf_token,
+                }
+            ).encode(),
+            headers={"Content-Type": "application/x-www-form-urlencoded"},
+            callback=self.after_login,
         )
 
-    async def parse_logged_in(self, response):
-        # Check if login succeeded
-        if response.request.meta.get("next_action") == "start_crawl":
-            # Start crawling protected pages
-            yield Request(url="https://example.com/dashboard")
+    async def after_login(self, response):
+        # Cookies from the login are stored automatically; crawl protected pages.
+        yield Request(url="https://example.com/dashboard")
 ```
 
 **Verify login success:**
@@ -221,40 +223,32 @@ class CustomAuthSpider(Spider):
     }
 
     async def parse(self, response):
-        """Step 1: Get initial token."""
+        """Step 1: get the initial token, then submit it (routed to step 2)."""
         rv = self.response_view(response)
-
         initial_token = rv.doc.cssselect("input[name=token]")[0].get("value")
 
-        # Step 2: Submit token
         yield Request(
             url="https://example.com/step2",
             method="POST",
-            body={"token": initial_token},
-            meta={"step": 2}
+            json={"token": initial_token},
+            callback=self.submit_credentials,
         )
 
-    async def parse_step2(self, response):
-        """Step 2: Submit credentials."""
-        step = response.request.meta.get("step")
+    async def submit_credentials(self, response):
+        """Step 2: submit credentials (routed to step 3)."""
+        rv = self.response_view(response)
+        csrf = rv.doc.cssselect("input[name=csrf]")[0].get("value")
 
-        if step == 2:
-            rv = self.response_view(response)
-            csrf = rv.doc.cssselect("input[name=csrf]")[0].get("value")
+        yield Request(
+            url="https://example.com/login",
+            method="POST",
+            json={"username": "user", "password": "pass", "csrf": csrf},
+            callback=self.start_crawl,
+        )
 
-            yield Request(
-                url="https://example.com/login",
-                method="POST",
-                body={
-                    "username": "user",
-                    "password": "pass",
-                    "csrf": csrf
-                },
-                meta={"step": 3}
-            )
-        elif step == 3:
-            # Authentication complete, start crawling
-            yield Request(url="https://example.com/protected")
+    async def start_crawl(self, response):
+        """Step 3: authentication complete, start crawling."""
+        yield Request(url="https://example.com/protected")
 ```
 
 

docs/advanced-topics/browser_automation.md

@@ -417,7 +417,7 @@ yield Request(
     url="https://example.com",
     meta={
         "use_handler": "camoufox",
-        "camoufox_event_handlers": {
+        "camoufox_page_event_handlers": {
             "console": handle_console,
             "dialog": handle_dialog
         }

docs/advanced-topics/crawl_ordering.md

@@ -1,5 +1,5 @@
 
-The scheduler processes requests based on priority values. Higher priority requests are processed first. By adjusting priorities, you can control whether your crawler explores pages breadth-first (level by level), depth-first (following paths deeply), or with custom focus on specific content.
+The scheduler processes requests based on priority values. Requests with a lower priority value are processed first (priority `0` is the default). By adjusting priorities, you can control whether your crawler explores pages breadth-first (level by level), depth-first (following paths deeply), or with custom focus on specific content.
 
 ## Breadth-first crawl (default)
 
@@ -39,15 +39,17 @@ async def parse(self, response):
     rv = self.response_view(response)
 
     current_depth = response.request.meta.get("depth", 0)
-    next_priority = current_depth  # Higher depth = higher priority
+    next_depth = current_depth + 1
 
     for link in rv.doc.cssselect("a"):
         href = link.get("href")
         if href:
+            # Lower priority value = processed sooner, so each request gets
+            # priority -depth: the deepest queued link is always followed first.
             yield rv.follow(
                 href,
-                priority=next_priority,
-                meta={"depth": current_depth + 1}
+                priority=-next_depth,
+                meta={"depth": next_depth},
             )
 ```
 
@@ -66,15 +68,15 @@ Prioritize specific content types or URL patterns:
 async def parse(self, response):
     rv = self.response_view(response)
 
-    # High priority for target content
+    # High priority for target content (lower value = processed first)
     if "product" in response.url:
         for link in rv.doc.cssselect("a.product"):
-            yield rv.follow(link.get("href"), priority=100)
+            yield rv.follow(link.get("href"), priority=-100)
 
-    # Low priority for other pages
+    # Lower priority for other pages
     else:
         for link in rv.doc.cssselect("a"):
-            yield rv.follow(link.get("href"), priority=1)
+            yield rv.follow(link.get("href"), priority=10)
 ```
 
 **Use cases:**

docs/advanced-topics/data_extraction.md

@@ -219,11 +219,7 @@ async def start_requests(self):
     yield Request(
         url="https://api.example.com/graphql",
         method="POST",
-        body={
-            "query": query,
-            "variables": {"page": 1}
-        },
-        headers={"Content-Type": "application/json"}
+        json={"query": query, "variables": {"page": 1}},
     )
 
 async def parse(self, response):
@@ -254,7 +250,7 @@ class SitemapSpider(Spider):
 
     async def parse(self, response):
         # Parse sitemap XML
-        root = ET.fromstring(response.text)
+        root = ET.fromstring(response.text())
         namespace = {"ns": "http://www.sitemaps.org/schemas/sitemap/0.9"}
 
         # Extract URLs
@@ -273,7 +269,7 @@ class SitemapSpider(Spider):
 ### Extract sitemap metadata
 ```python
 async def parse(self, response):
-    root = ET.fromstring(response.text)
+    root = ET.fromstring(response.text())
     namespace = {"ns": "http://www.sitemaps.org/schemas/sitemap/0.9"}
 
     for url_elem in root.findall(".//ns:url", namespace):

docs/advanced-topics/error_recovery.md

@@ -19,23 +19,20 @@ No configuration needed - it works out of the box!
 
 ### Configure retry settings
 
-Customize retry behavior in spider settings:
+All retry behaviour is configurable via settings:
 
 ```python
 class MySpider(Spider):
     name = "my_spider"
     start_urls = ["https://example.com"]
 
     custom_settings = {
-        "RETRY_ENABLED": True,  # Default: True
-        "RETRY_TIMES": 5,  # Max retries (default: 3)
-        "RETRY_HTTP_CODES": [429, 500, 502, 503, 504, 408],  # Status codes to retry
-        "RETRY_PRIORITY_ADJUST": -1,  # Lower priority for retries (default: -1)
-
-        # Exponential backoff parameters
-        "RETRY_BACKOFF_BASE": 2.0,  # Base delay in seconds (default: 1.0)
-        "RETRY_BACKOFF_MAX": 120.0,  # Max delay in seconds (default: 60.0)
-        "RETRY_BACKOFF_JITTER": 0.3,  # Jitter fraction (default: 0.3)
+        "MAX_RETRIES": 5,                                    # default: 3
+        "RETRY_HTTP_CODES": [429, 500, 502, 503, 504, 408],  # default: [429, 500, 502, 503, 504]
+        "RETRY_PRIORITY_ADJUST": -1,                         # default: -1 (lower priority for retries)
+        "RETRY_BACKOFF_BASE": 2.0,                           # default: 1.0
+        "RETRY_BACKOFF_MAX": 120.0,                          # default: 60.0
+        "RETRY_BACKOFF_JITTER": 0.3,                         # default: 0.3
     }
 ```
 
@@ -61,22 +58,28 @@ With jitter applied: `delay ± (delay * jitter)`
 Check retry statistics:
 
 ```python
-stats = crawler.stats.get_stats()
+stats = crawler.stats.snapshot()
 
 print(f"Total retries: {stats.get('downloader/retry/total', 0)}")
 print(f"Network errors: {stats.get('downloader/retry/network_error', 0)}")
 print(f"HTTP errors: {stats.get('downloader/retry/http_error', 0)}")
 print(f"Max retries reached: {stats.get('downloader/retry/max_reached', 0)}")
 ```
 
-### Disable retry for specific requests
+### Skip retries for specific requests
+
+Control retries per request through `meta`: `dont_retry` skips retrying a
+request entirely, and `max_retry_times` caps its retries independently of the
+`MAX_RETRIES` setting.
 
 ```python
-# Disable retry for a specific request
-yield Request(
-    url="https://example.com/optional",
-    meta={"dont_retry": True}
-)
+from qcrawl.core.request import Request
+
+# Never retry this request
+yield Request(url="https://example.com/optional", meta={"dont_retry": True})
+
+# Allow at most one retry for this request
+yield Request(url="https://example.com/flaky", meta={"max_retry_times": 1})
 ```
 
 
@@ -232,7 +235,7 @@ async def parse(self, response):
         self.logger.warning(
             f"No items found on {response.url} "
             f"(status: {response.status_code}, "
-            f"content-length: {len(response.text)})"
+            f"content-length: {len(response.text())})"
         )
 
         # Log page structure for debugging
@@ -252,7 +255,7 @@ async def parse(self, response):
 ## Best practices
 
 - **Use built-in RetryMiddleware**: It's enabled by default and handles network errors and HTTP errors automatically
-- **Configure retry settings**: Adjust `RETRY_TIMES`, `RETRY_HTTP_CODES`, and backoff parameters as needed
+- **Configure retry settings**: Adjust `MAX_RETRIES`, `RETRY_HTTP_CODES`, `RETRY_PRIORITY_ADJUST`, and `RETRY_BACKOFF_*` as needed
 - **Monitor retry stats**: Track retry metrics to detect problematic sources
 - **Handle missing data gracefully**: Use default values and defensive checks for optional fields
 - **Log errors appropriately**: Use different log levels (warning, error, debug)