Add GitBook-synced docs under docs/

Mirror the wiki content into a docs/ tree so GitBook Git Sync can connect to
this repo. Adds SUMMARY.md for the table of contents, .gitbook.yaml pointing
GitBook at docs/, and converts wiki page links to relative .md paths.

.gitbook.yaml

@@ -0,0 +1,5 @@
+root: ./docs/
+
+structure:
+  readme: README.md
+  summary: SUMMARY.md

docs/Adding-a-CDP-method-or-Web-API.md

@@ -0,0 +1,154 @@
+Two recipes for the two most common extensions: a new CDP method, and a new JS Web API.
+
+## Adding a CDP method
+
+Worked example: `MyDomain.doThing` that takes `{ name }` and returns `{ ok }`.
+
+### 1. Add the handler
+
+Create or edit a file under `crates/obscura-cdp/src/domains/`:
+
+```rust
+// crates/obscura-cdp/src/domains/my_domain.rs
+use serde_json::{json, Value};
+use crate::dispatch::CdpContext;
+
+pub async fn do_thing(
+    params: &Value,
+    _ctx: &mut CdpContext,
+    _session_id: &Option<String>,
+) -> Result<Value, String> {
+    let name = params.get("name")
+        .and_then(|v| v.as_str())
+        .ok_or("missing name")?;
+
+    // do the work
+
+    Ok(json!({ "ok": true, "name": name }))
+}
+```
+
+### 2. Register in the dispatcher
+
+In `crates/obscura-cdp/src/dispatch.rs`, add a match arm:
+
+```rust
+"MyDomain.doThing" => domains::my_domain::do_thing(&req.params, ctx, &req.session_id).await,
+```
+
+### 3. Test it
+
+`crates/obscura-cdp/tests/cdp_my_domain.rs`:
+
+```rust
+use obscura_cdp::dispatch::{dispatch, CdpContext};
+use obscura_cdp::types::CdpRequest;
+use serde_json::json;
+
+#[tokio::test(flavor = "current_thread")]
+async fn my_domain_do_thing_returns_ok() {
+    let mut ctx = CdpContext::new();
+    let resp = dispatch(&CdpRequest {
+        id: 1,
+        method: "MyDomain.doThing".into(),
+        params: json!({ "name": "test" }),
+        session_id: None,
+    }, &mut ctx).await;
+
+    assert!(resp.error.is_none());
+    assert_eq!(resp.result.unwrap()["ok"], true);
+}
+```
+
+Run:
+
+```bash
+cargo test -p obscura-cdp my_domain
+```
+
+## Adding a Web API
+
+Worked example: `crypto.subtle.digest`, real implementation backed by a Rust hash op.
+
+### 1. Add the Rust op
+
+In `crates/obscura-js/src/ops.rs`:
+
+```rust
+#[op2]
+#[buffer]
+fn op_subtle_digest(#[string] algorithm: &str, #[buffer] data: &[u8]) -> Vec<u8> {
+    use sha1::Digest as _;
+    match algorithm.to_ascii_uppercase().as_str() {
+        "SHA-1"   => sha1::Sha1::digest(data).to_vec(),
+        "SHA-256" => sha2::Sha256::digest(data).to_vec(),
+        "SHA-384" => sha2::Sha384::digest(data).to_vec(),
+        "SHA-512" => sha2::Sha512::digest(data).to_vec(),
+        _         => sha2::Sha256::digest(data).to_vec(),
+    }
+}
+```
+
+### 2. Register the op
+
+In the same file, `build_extension()`:
+
+```rust
+ops: std::borrow::Cow::Owned(vec![
+    op_dom(),
+    op_console_msg(),
+    // ...
+    op_subtle_digest(),
+]),
+```
+
+### 3. Add the JS shim
+
+In `crates/obscura-js/js/bootstrap.js`:
+
+```js
+globalThis.crypto = globalThis.crypto || {};
+globalThis.crypto.subtle = globalThis.crypto.subtle || {};
+globalThis.crypto.subtle.digest = function digest(algorithm, data) {
+  const algName = typeof algorithm === 'string' ? algorithm : algorithm.name;
+  const bytes = data instanceof ArrayBuffer
+    ? new Uint8Array(data)
+    : new Uint8Array(data.buffer, data.byteOffset, data.byteLength);
+  const out = Deno.core.ops.op_subtle_digest(algName, bytes);
+  return Promise.resolve(out.buffer);
+};
+```
+
+### 4. Add a dependency if needed
+
+`crates/obscura-js/Cargo.toml`:
+
+```toml
+sha1 = "0.10"
+sha2 = "0.10"
+```
+
+### 5. Smoke test
+
+```bash
+cargo build --release
+./target/release/obscura fetch https://example.com --eval "
+  crypto.subtle.digest('SHA-256', new TextEncoder().encode('hi'))
+    .then(buf => Array.from(new Uint8Array(buf)).map(b => b.toString(16).padStart(2, '0')).join(''))
+"
+```
+
+## Tips
+
+- Keep the JS shim thin. All side effects go through ops.
+- Use `Promise.resolve` to keep async-shaped APIs callable from sync ops.
+- Match the spec: Web API names and shapes are checked by Puppeteer / Playwright wrappers.
+- DOM mutations go through `op_dom`, not new ops.
+- For events that need to fire across handlers, use the existing `_makeListenerBox` helper in `bootstrap.js`.
+
+## Worked examples in the tree
+
+- CDP method with intercept: `crates/obscura-cdp/src/domains/page.rs` `do_navigate`.
+- Web API with op + JS shim: `crypto.subtle.digest` (above).
+- Web API in pure JS (no op): `DOMParser` in `bootstrap.js`.
+- Web API with async event firing: `WebSocket`, `IntersectionObserver` in `bootstrap.js`.

docs/Architecture-overview.md

@@ -0,0 +1,110 @@
+Obscura is a workspace of eight crates.
+
+```
+obscura-cli       CLI entry point. fetch, serve, scrape, mcp.
+obscura-cdp       Chrome DevTools Protocol server. WebSocket, dispatch, domain handlers.
+obscura-browser   Page type, navigation, lifecycle events.
+obscura-js        V8 runtime via deno_core. bootstrap.js + Rust ops.
+obscura-dom       DOM tree implementation.
+obscura-net       HTTP client, stealth client, cookie jar, robots cache, tracker blocklist.
+obscura-mcp       Model Context Protocol server.
+obscura           Embeddable Rust library API (Browser, Page, Element, CookieStore).
+```
+
+## Request flow
+
+A `Page.navigate` from a CDP client:
+
+```
+CDP client (Puppeteer)
+        │ WebSocket frame
+        ▼
+obscura-cdp/server.rs           accept, route by sessionId
+        │
+        ▼
+obscura-cdp/dispatch.rs         method router, acquires v8_lock
+        │
+        ▼
+obscura-cdp/domains/page.rs     Page.navigate handler
+        │
+        ▼
+obscura-browser/page.rs         navigate_with_wait
+        │
+        ├──► obscura-net/client.rs        HTTP fetch
+        │
+        ├──► obscura-dom/tree.rs          parse HTML into the tree
+        │
+        └──► obscura-js/runtime.rs        run inline scripts
+                  │
+                  └──► bootstrap.js + ops.rs    DOM bindings
+```
+
+The dispatcher emits CDP events (`Network.requestWillBeSent`, `Page.frameNavigated`, `Page.lifecycleEvent`) back to the client through the same WebSocket.
+
+## Single V8 isolate
+
+All pages in a process share one V8 isolate. The isolate is single-threaded by design.
+
+`obscura_js::v8_lock::global()` is a `tokio::sync::Mutex` that serializes V8 work. A handler that wants to run JS must acquire the lock first:
+
+```rust
+let _guard = obscura_js::v8_lock::global().lock().await;
+page.evaluate(expr).await
+```
+
+The dispatcher routes long-running operations (navigation, eval) through `process_with_interception` in `server.rs`, which spawns the work onto the tokio `LocalSet` and releases the dispatcher to keep handling other CDP messages.
+
+This is why `Target.createTarget` from many concurrent clients works: each `newPage` returns immediately while the actual navigation runs in a spawned task.
+
+## Robustness
+
+One page cannot hang or crash the process. `obscura-js/runtime.rs` provides a V8 termination watchdog (`arm_watchdog`, `run_event_loop_bounded`) that terminates the isolate from a separate thread when synchronous work overruns a budget, because `tokio::time::timeout` cannot preempt synchronous V8. It bounds the post-load settle, the navigation event-loop pumps, and `--eval`. `obscura-js/cdp_watchdog.rs` is a single shared watchdog the dispatcher arms around every CDP command, so a runaway page cannot hold the V8 lock and wedge other sessions (tunable via `OBSCURA_CDP_COMMAND_TIMEOUT_MS`). `op_dom` is wrapped in `catch_unwind` so a DOM-op panic degrades to a null result instead of aborting the process through V8's FFI frame, and `obscura-dom/tree.rs` rejects cyclic reparenting that would make tree walks loop forever. Scripted `fetch()`/XHR and module loads are timeout-bounded (`OBSCURA_FETCH_TIMEOUT_MS`), and the one-shot `fetch` CLI has a process-level hard deadline as a final backstop.
+
+## JS bridge
+
+`obscura-js/js/bootstrap.js` provides the browser globals: `document`, `window`, `navigator`, `location`, observers, fetch, indexedDB, etc.
+
+`obscura-js/src/ops.rs` registers Rust ops that the bootstrap calls into:
+
+```js
+Deno.core.ops.op_dom('insert_before', parentNid, refNid, newNid);
+```
+
+Adding a Web API usually means:
+
+1. JS shim in `bootstrap.js` that exposes the API surface.
+2. Rust op in `ops.rs` that performs the side effect (DOM mutation, fetch, crypto).
+3. Register the op in `build_extension()`.
+
+Worked example: [Adding a CDP method or Web API](Adding-a-CDP-method-or-Web-API.md).
+
+## CDP session model
+
+Each CDP client connection gets attached to one or more targets.
+Session IDs are `"{targetId}-session"`. The dispatcher routes by `sessionId` in the incoming frame to the right `Page`.
+
+Targets are created by `Target.createTarget`. Closing the WebSocket detaches all sessions but leaves the pages running.
+
+## Lifecycle
+
+Lifecycle events are emitted by `obscura-browser/lifecycle.rs` as the page transitions:
+
+```
+init → commit → domcontentloaded → load → networkidle2 → networkidle0
+```
+
+`waitUntil` on `Page.navigate` blocks until the requested level is reached. The Puppeteer / Playwright `goto` resolves on the matching `Page.lifecycleEvent` client-side.
+
+## Storage
+
+`--storage-dir` persists cookies (`cookies.json`) and localStorage (`localStorage/<origin>.json`). Reads on process start, writes on every navigation and on graceful shutdown.
+
+## Stealth
+
+`--stealth` swaps the default `reqwest` client for `obscura-net/wreq_client.rs`, which randomizes TLS ClientHello and cipher order to match a real browser, and applies the bundled tracker blocklist before any request leaves the process.
+
+## Workspace conventions
+
+- One crate per layer. Cross-crate calls go through the layer above, not sideways.
+- All async is `tokio` with a `LocalSet` because V8 is `!Send`.
+- All DOM ops go through `op_dom` to keep the JS/Rust boundary narrow.

docs/Build-from-source.md

@@ -0,0 +1,60 @@
+## Requirements
+
+- Rust 1.75+ ([rustup.rs](https://rustup.rs))
+- C compiler (gcc or clang)
+- ~5 GB free disk space (V8 compiles from source on first build)
+
+First build takes about 5 minutes. Incremental builds are seconds.
+
+## Build
+
+```bash
+git clone https://github.com/h4ckf0r0day/obscura.git
+cd obscura
+cargo build --release
+```
+
+Binary is at `./target/release/obscura`.
+
+## With stealth
+
+```bash
+cargo build --release --features stealth
+```
+
+Adds TLS fingerprint randomization and the tracker blocklist. See [Configure stealth and proxies](Configure-stealth-and-proxies.md).
+
+## OpenSSL on older systems
+
+If the build fails on the vendored OpenSSL with an AVX-512 assembler error (common on older VPS hosts):
+
+```bash
+OPENSSL_NO_VENDOR=1 cargo build --release
+```
+
+Uses the system OpenSSL instead.
+
+## Run from the build
+
+```bash
+./target/release/obscura --version
+./target/release/obscura fetch https://example.com --eval "document.title"
+```
+
+Install system-wide:
+
+```bash
+cargo install --path crates/obscura-cli
+```
+
+## Tests
+
+```bash
+cargo test --release
+```
+
+Integration suite:
+
+```bash
+python3 tests/test_all.py
+```

docs/CLI-reference.md

@@ -0,0 +1,106 @@
+## `obscura`
+
+Top-level flags apply to every subcommand.
+
+```
+-v, --verbose                Enable info logging
+-p, --port <PORT>            CDP port (default 9222)
+    --proxy <URL>            HTTP or SOCKS5 proxy
+    --obey-robots            Respect robots.txt
+    --user-agent <UA>        Override the User-Agent
+    --storage-dir <DIR>      Persistent cookies and localStorage
+    --allow-private-network  Permit loopback / RFC1918 / link-local
+    --v8-flags <FLAGS>       Raw V8 flags, applied at startup
+-h, --help                   Help
+-V, --version                Version
+```
+
+## `obscura fetch <URL>`
+
+Load a URL and print its content or an evaluated expression.
+
+```
+    --dump <FORMAT>          html | text | links | markdown | original | assets
+                             (default html)
+    --selector <CSS>         Narrow output to a CSS selector
+    --wait <SECONDS>         Extra wait after settle (default 5)
+    --timeout <SECONDS>      Navigation timeout (default 30)
+    --wait-until <LEVEL>     domcontentloaded | load | networkidle2 | networkidle0
+                             (default load)
+    --user-agent <UA>        Override the User-Agent
+    --proxy <URL>            HTTP or SOCKS5 proxy
+    --stealth                TLS fingerprint randomization + tracker blocking
+-e, --eval <JS>              Evaluate JS, print the result as JSON
+-o, --output <FILE>          Write to a file instead of stdout
+-q, --quiet                  Suppress info logging
+-v, --verbose                Enable verbose logging
+```
+
+`--dump` values:
+
+| Value      | Output                                                    |
+| ---------- | --------------------------------------------------------- |
+| `html`     | Rendered HTML (default)                                   |
+| `text`     | Plain text                                                |
+| `markdown` | Markdown conversion                                       |
+| `links`    | Every `<a href>`, one URL per line                        |
+| `assets`   | Every external resource, one JSON object per line         |
+| `original` | Raw HTTP response body (binary-safe, bypasses the engine) |
+
+## `obscura serve`
+
+Run the CDP server. Puppeteer and Playwright connect over WebSocket.
+
+```
+-p, --port <PORT>            CDP port (default 9222)
+    --host <HOST>            Bind host (default 127.0.0.1)
+    --proxy <URL>            HTTP or SOCKS5 proxy
+    --user-agent <UA>        Override the User-Agent
+    --stealth                TLS fingerprint randomization + tracker blocking
+    --workers <N>            Worker processes (default 1)
+    --allow-file-access      Permit CDP clients to navigate to file:// URLs
+    --storage-dir <DIR>      Persistent cookies and localStorage
+    --allow-private-network  Permit loopback / RFC1918 / link-local
+-v, --verbose                Enable info logging
+```
+
+Default endpoint is `ws://127.0.0.1:9222`.
+
+## `obscura scrape [URLS]...`
+
+Run a JS expression across many URLs in parallel.
+
+```
+-e, --eval <JS>              JS to run on each page
+    --concurrency <N>        Parallel pages (default 10)
+    --format <FORMAT>        Output format (default json)
+    --timeout <SECONDS>      Per-URL timeout (default 60)
+    --proxy <URL>            HTTP or SOCKS5 proxy
+    --allow-private-network  Permit loopback / RFC1918 / link-local
+-q, --quiet                  Suppress info logging
+-v, --verbose                Enable verbose logging
+```
+
+Read URLs from stdin with `-`:
+
+```bash
+cat urls.txt | obscura scrape - --eval "document.title" --concurrency 20
+```
+
+Requires `obscura-worker` next to `obscura` in `PATH`.
+
+## `obscura mcp`
+
+Run obscura as an MCP server.
+
+```
+    --http                   HTTP transport instead of stdio
+    --port <PORT>            HTTP port (default 3000)
+    --proxy <URL>            HTTP or SOCKS5 proxy
+    --user-agent <UA>        Override the User-Agent
+    --stealth                TLS fingerprint randomization + tracker blocking
+    --allow-private-network  Permit loopback / RFC1918 / link-local
+-v, --verbose                Enable info logging
+```
+
+Default transport is stdio. See [Use the MCP server](Use-the-MCP-server.md).

docs/Configure-stealth-and-proxies.md

@@ -0,0 +1,70 @@
+## Stealth mode
+
+```bash
+obscura fetch https://example.com --stealth
+obscura serve --stealth
+obscura mcp --stealth
+```
+
+What `--stealth` changes:
+
+- Uses the wreq HTTP client with browser-matching TLS fingerprints (ClientHello, ALPN, cipher order).
+- Loads a tracker blocklist that drops requests to known analytics and fingerprinting endpoints.
+- Bundles webpki roots instead of relying on the system store.
+
+Requires a build that includes the stealth feature. Release binaries on the Releases page include it. To build it yourself:
+
+```bash
+cargo build --release --features stealth
+```
+
+## What stealth handles
+
+- Basic bot detection that checks TLS fingerprint or User-Agent.
+- Sites that rely on third-party analytics being reachable.
+
+## What stealth does not handle
+
+- Cloudflare interactive challenges.
+- Datadome and Akamai bot manager active challenges.
+- CAPTCHAs.
+- IP-based rate limiting (use proxies).
+
+## Proxies
+
+HTTP proxy:
+
+```bash
+obscura fetch https://example.com --proxy http://proxy.example.com:8080
+obscura serve --proxy http://proxy.example.com:8080
+```
+
+With auth:
+
+```bash
+obscura fetch https://example.com --proxy http://user:pass@proxy.example.com:8080
+```
+
+SOCKS5:
+
+```bash
+obscura fetch https://example.com --proxy socks5://proxy.example.com:1080
+```
+
+## Custom User-Agent
+
+```bash
+obscura fetch https://example.com --user-agent "Mozilla/5.0 (...) ..."
+obscura serve --user-agent "Mozilla/5.0 (...) ..."
+```
+
+Default UA matches a recent Chrome on the build platform.
+
+## Combine
+
+```bash
+obscura serve \
+  --stealth \
+  --proxy http://user:pass@proxy.example.com:8080 \
+  --user-agent "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 ..."
+```

docs/Connect-Puppeteer-or-Playwright.md

@@ -0,0 +1,86 @@
+Obscura speaks the Chrome DevTools Protocol over WebSocket. Puppeteer and Playwright connect to it like remote Chrome.
+
+## Start the server
+
+```bash
+obscura serve --port 9222
+```
+
+```
+obscura listening on ws://127.0.0.1:9222
+```
+
+## Puppeteer
+
+```bash
+npm install puppeteer-core
+```
+
+```js
+const puppeteer = require('puppeteer-core');
+
+const browser = await puppeteer.connect({
+  browserWSEndpoint: 'ws://127.0.0.1:9222',
+});
+
+const page = await browser.newPage();
+await page.goto('https://example.com');
+console.log(await page.title()); // "Example Domain"
+
+await browser.disconnect();
+```
+
+Use `puppeteer-core`, not `puppeteer`. The `puppeteer` package bundles a Chrome download.
+
+## Playwright
+
+```bash
+npm install playwright
+```
+
+```js
+const { chromium } = require('playwright');
+
+const browser = await chromium.connectOverCDP('ws://127.0.0.1:9222');
+const context = browser.contexts()[0] || await browser.newContext();
+const page = await context.newPage();
+
+await page.goto('https://example.com');
+console.log(await page.title());
+
+await browser.close();
+```
+
+Use `connectOverCDP`, not `connect`. Playwright's `connect` speaks Playwright's own protocol, which obscura does not implement.
+
+## `waitUntil`
+
+Default is `domcontentloaded`. For full subresource load:
+
+```js
+await page.goto('https://example.com', { waitUntil: 'load' });
+```
+
+| Value              | Returns when                            |
+| ------------------ | --------------------------------------- |
+| `domcontentloaded` | HTML parsed, scripts ran (default)      |
+| `load`             | All subresources finished               |
+| `networkidle2`     | ≤2 network connections active for 500ms |
+| `networkidle0`     | 0 network connections active for 500ms  |
+
+## Supported
+
+- `page.goto`, `page.reload`, `page.goBack`, `page.goForward`
+- `page.evaluate`, `page.evaluateHandle`
+- `page.click`, `page.type`, `page.fill`
+- `page.waitForSelector`, `page.waitForFunction`, `page.waitForNavigation`
+- `page.cookies`, `page.setCookie`, `context.cookies`
+- `page.setRequestInterception`, block / modify
+- `page.exposeFunction`
+- `page.content`, `page.title`, `page.url`
+
+## Not supported
+
+- `page.screenshot`: obscura doesn't render pixels.
+- Per-page V8 isolation: pages share one V8 isolate. A heavy script on one page can stall others.
+- Cloudflare / Datadome / Akamai bypass: see [Configure stealth and proxies](Configure-stealth-and-proxies.md).

docs/Environment-variables.md

@@ -0,0 +1,83 @@
+## Runtime
+
+### `OBSCURA_ALLOW_PRIVATE_NETWORK`
+
+Allow fetches to loopback (`127.0.0.0/8`), RFC1918 (`10.0.0.0/8`, `172.16.0.0/12`, `192.168.0.0/16`), and link-local (`169.254.0.0/16`) addresses. Off by default to block SSRF.
+
+Truthy values: `1`, `true`, `yes`, `on`.
+
+```bash
+OBSCURA_ALLOW_PRIVATE_NETWORK=1 obscura fetch http://localhost:8080
+```
+
+Per-process equivalent: `--allow-private-network` on any subcommand.
+
+### `OBSCURA_NAV_TIMEOUT_MS`
+
+Hard ceiling on a single navigation. Default 30000 (30 seconds). Applies to `Page.navigate` and the CLI `fetch` command.
+
+```bash
+OBSCURA_NAV_TIMEOUT_MS=60000 obscura serve
+```
+
+### `OBSCURA_CDP_COMMAND_TIMEOUT_MS`
+
+Per-command deadline for the CDP server. A hung page (a runaway `Runtime.evaluate`, a synchronous DOM op) is terminated after this budget so one bad session cannot hold the shared V8 lock and stall the others. Default 60000 (60 seconds); `0` disables it. Navigation self-bounds via `OBSCURA_NAV_TIMEOUT_MS` well under this.
+
+```bash
+OBSCURA_CDP_COMMAND_TIMEOUT_MS=30000 obscura serve
+```
+
+### `OBSCURA_FETCH_TIMEOUT_MS`
+
+Request timeout for scripted `fetch()`, `XMLHttpRequest`, and ES-module loads. Without it a request to a server that accepts the connection but never responds (including a CORS preflight) hangs forever and the XHR is stuck with no completion event. Default 30000 (30 seconds).
+
+```bash
+OBSCURA_FETCH_TIMEOUT_MS=15000 obscura serve
+```
+
+### `OBSCURA_PROXY`
+
+Default proxy URL used by `obscura-worker` for the parallel `scrape` command when no `--proxy` flag is set.
+
+```bash
+OBSCURA_PROXY=http://proxy.example.com:8080 obscura scrape - < urls.txt
+```
+
+## Logging
+
+### `RUST_LOG`
+
+Standard `tracing` filter. Common settings:
+
+```bash
+RUST_LOG=obscura=info obscura serve
+RUST_LOG=obscura=debug obscura serve
+RUST_LOG=obscura_cdp=trace,obscura_browser=debug obscura serve
+```
+
+`--verbose` on the CLI is equivalent to `RUST_LOG=obscura=info`.
+
+## Build
+
+### `OPENSSL_NO_VENDOR`
+
+Forces `cargo build` to use the system OpenSSL instead of compiling the vendored copy. Set to `1` on hosts where the vendored OpenSSL fails (older VPS with AVX-512 issues).
+
+```bash
+OPENSSL_NO_VENDOR=1 cargo build --release
+```
+
+## V8
+
+V8 flags are passed via `--v8-flags`, not environment variables:
+
+```bash
+obscura serve --v8-flags "--max-old-space-size=2048 --expose-gc"
+```
+
+Default heap is 4 GB on 64-bit systems.
+
+## HTTP proxy environment
+
+Obscura does not honor `HTTP_PROXY` / `HTTPS_PROXY` / `NO_PROXY`. Use `--proxy` or `OBSCURA_PROXY`.

docs/Extract-data.md

@@ -0,0 +1,75 @@
+`--dump` formats the page output without writing JavaScript.
+
+```bash
+obscura fetch https://example.com --dump html
+obscura fetch https://example.com --dump text
+obscura fetch https://example.com --dump markdown
+obscura fetch https://example.com --dump links
+obscura fetch https://example.com --dump assets
+obscura fetch https://example.com --dump original
+```
+
+## `html`
+
+Rendered HTML after JavaScript runs. Default.
+
+```bash
+obscura fetch https://news.ycombinator.com --dump html > hn.html
+```
+
+## `text`
+
+Plain text. No markup.
+
+```bash
+obscura fetch https://en.wikipedia.org/wiki/Rust_(programming_language) --dump text
+```
+
+## `markdown`
+
+Markdown conversion: headings, lists, links, code blocks, images.
+
+```bash
+obscura fetch https://docs.example.com/page --dump markdown > page.md
+```
+
+## `links`
+
+Every `<a href>` on the page, one per line.
+
+```bash
+obscura fetch https://example.com --dump links
+```
+
+## `assets`
+
+Every external resource (stylesheets, scripts, images, fonts, iframes), one per line.
+
+```bash
+obscura fetch https://example.com --dump assets
+```
+
+## `original`
+
+The raw HTML the server sent, before JavaScript ran.
+
+```bash
+obscura fetch https://my-spa.example --dump original > before.html
+obscura fetch https://my-spa.example --dump html     > after.html
+diff before.html after.html
+```
+
+## With `--wait-until`
+
+`--dump` runs after the wait condition:
+
+```bash
+obscura fetch https://my-spa.example --wait-until load --dump markdown
+```
+
+## Pipe and redirect
+
+```bash
+obscura fetch https://example.com --dump markdown > example.md
+obscura fetch https://example.com --dump text --quiet | wc -w
+```

docs/Installation.md

@@ -0,0 +1,76 @@
+## Linux x86_64
+
+```bash
+curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-x86_64-linux.tar.gz
+tar xzf obscura-x86_64-linux.tar.gz
+./obscura --version
+```
+
+## Linux ARM64
+
+```bash
+curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-aarch64-linux.tar.gz
+tar xzf obscura-aarch64-linux.tar.gz
+./obscura --version
+```
+
+Linux builds target Ubuntu 22.04 and require glibc 2.35+.
+
+## macOS Apple Silicon
+
+```bash
+curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-aarch64-macos.tar.gz
+tar xzf obscura-aarch64-macos.tar.gz
+./obscura --version
+```
+
+## macOS Intel
+
+```bash
+curl -LO https://github.com/h4ckf0r0day/obscura/releases/latest/download/obscura-x86_64-macos.tar.gz
+tar xzf obscura-x86_64-macos.tar.gz
+./obscura --version
+```
+
+## Windows
+
+Download the `.zip` from [Releases](https://github.com/h4ckf0r0day/obscura/releases), extract, run `obscura.exe --version`.
+
+## Arch Linux (AUR)
+
+```bash
+yay -S obscura-browser
+```
+
+## Docker
+
+```bash
+docker run -d --name obscura -p 127.0.0.1:9222:9222 h4ckf0r0day/obscura
+```
+
+Image: [h4ckf0r0day/obscura](https://hub.docker.com/r/h4ckf0r0day/obscura). Built on `distroless/cc`, ~57 MB compressed.
+
+## From source
+
+See [Build from source](Build-from-source.md).
+
+## What's in the archive
+
+- `obscura`: CLI and CDP server.
+- `obscura-worker`: helper for the parallel `scrape` command. Keep both in the same directory.
+
+## Smoke test
+
+```bash
+./obscura fetch https://example.com --eval "document.title"
+```
+
+Expected output: `"Example Domain"`.
+
+## Troubleshooting
+
+`cannot execute binary file`: wrong arch. Check `uname -m`.
+
+`GLIBC_2.35 not found`: distro is older than Ubuntu 22.04. Use Docker or build from source.
+
+macOS Gatekeeper warning: `xattr -d com.apple.quarantine ./obscura`.

docs/Intercept-and-modify-requests.md

@@ -0,0 +1,120 @@
+CDP `Fetch.enable` lets you inspect, block, or modify every request the page makes.
+
+## Block by resource type
+
+Puppeteer:
+
+```js
+await page.setRequestInterception(true);
+
+page.on('request', req => {
+  if (['image', 'media', 'font'].includes(req.resourceType())) {
+    req.abort();
+  } else {
+    req.continue();
+  }
+});
+```
+
+Playwright:
+
+```js
+await page.route('**/*', route => {
+  if (['image', 'media', 'font'].includes(route.request().resourceType())) {
+    route.abort();
+  } else {
+    route.continue();
+  }
+});
+```
+
+## Block by URL pattern
+
+```js
+// Puppeteer
+page.on('request', req => {
+  const url = req.url();
+  if (url.includes('google-analytics.com') || url.includes('doubleclick.net')) {
+    req.abort();
+  } else {
+    req.continue();
+  }
+});
+```
+
+```js
+// Playwright
+await page.route(/google-analytics\.com|doubleclick\.net/, route => route.abort());
+```
+
+## Modify headers
+
+```js
+// Puppeteer
+page.on('request', req => {
+  req.continue({
+    headers: { ...req.headers(), 'X-Custom': 'value' },
+  });
+});
+```
+
+```js
+// Playwright
+await page.route('**/*', route => {
+  route.continue({
+    headers: { ...route.request().headers(), 'X-Custom': 'value' },
+  });
+});
+```
+
+## Return a fake response
+
+```js
+// Puppeteer
+page.on('request', req => {
+  if (req.url().endsWith('/api/feature-flags')) {
+    req.respond({
+      status: 200,
+      contentType: 'application/json',
+      body: JSON.stringify({ newDashboard: true }),
+    });
+  } else {
+    req.continue();
+  }
+});
+```
+
+```js
+// Playwright
+await page.route('**/api/feature-flags', route => {
+  route.fulfill({
+    status: 200,
+    contentType: 'application/json',
+    body: JSON.stringify({ newDashboard: true }),
+  });
+});
+```
+
+## Strip analytics in production scrapes
+
+```js
+const BLOCK = [
+  'google-analytics.com',
+  'googletagmanager.com',
+  'doubleclick.net',
+  'facebook.net',
+  'segment.io',
+  'mixpanel.com',
+  'hotjar.com',
+];
+
+page.on('request', req => {
+  if (BLOCK.some(host => req.url().includes(host))) {
+    req.abort();
+  } else {
+    req.continue();
+  }
+});
+```
+
+Built-in: `--stealth` ships with a tracker blocklist that handles most of these without per-script setup. See [Configure stealth and proxies](Configure-stealth-and-proxies.md).

docs/Markdown-extraction.md

@@ -0,0 +1,57 @@
+`--dump markdown` converts the rendered page to markdown.
+
+```bash
+obscura fetch https://example.com --dump markdown
+```
+
+## What gets converted
+
+- Headings (`<h1>` through `<h6>`)
+- Paragraphs, line breaks
+- Bold, italic, code spans
+- Links (with `href`)
+- Images (with `src` and `alt`)
+- Ordered and unordered lists
+- Block quotes
+- Code blocks (`<pre>`, `<code>`)
+- Tables
+
+## What gets stripped
+
+- `<script>`, `<style>`, `<noscript>`
+- Inline styles
+- ARIA attributes
+- Tracking pixels and beacons
+
+## Save to file
+
+```bash
+obscura fetch https://docs.example.com/page --dump markdown -o page.md
+```
+
+## For RAG / LLM context
+
+```bash
+obscura fetch https://docs.example.com/page --dump markdown --quiet
+```
+
+`--quiet` strips info logging so the output is just markdown.
+
+## Wait for SPA content
+
+For pages that render content client-side:
+
+```bash
+obscura fetch https://my-spa.example --wait-until load --dump markdown
+```
+
+## Narrow to a region
+
+`--selector` restricts the conversion to a CSS selector:
+
+```bash
+obscura fetch https://example.com --selector "main" --dump markdown
+obscura fetch https://example.com --selector "article.post" --dump markdown
+```
+
+Useful for skipping nav, sidebars, and footers.

docs/Persist-cookies-and-storage.md

@@ -0,0 +1,58 @@
+`--storage-dir` persists cookies and localStorage to disk so they survive across runs.
+
+## CLI
+
+```bash
+obscura fetch https://example.com --storage-dir ./obscura-data
+obscura fetch https://example.com --storage-dir ./obscura-data
+```
+
+The second invocation starts with the cookies and localStorage left by the first.
+
+## Server
+
+```bash
+obscura serve --storage-dir ./obscura-data
+```
+
+All CDP sessions read and write to the same directory. Run separate `obscura serve` processes with different `--storage-dir` paths for isolated profiles.
+
+## Layout
+
+Inside `./obscura-data`:
+
+- `cookies.json`: cookie jar in a stable format with `same_site`, `expires`, `http_only`, `secure`.
+- `localStorage/<origin>.json`: one file per origin.
+
+The format is stable. Inspect with `jq`:
+
+```bash
+jq '.[] | select(.domain == "example.com")' ./obscura-data/cookies.json
+```
+
+## When state is written
+
+- On clean process exit (Ctrl-C, SIGTERM).
+- After every navigation completes (CDP `Page.navigate`).
+- Manually via CDP `Network.setCookie` and `Network.deleteCookies`.
+
+## Login once, scrape many
+
+```bash
+obscura serve --storage-dir ./session-1
+```
+
+Drive a login flow once via Puppeteer or Playwright. Stop the server. Subsequent runs against the same `--storage-dir` start logged in.
+
+## Multiple identities
+
+```bash
+obscura serve --port 9222 --storage-dir ./identity-a
+obscura serve --port 9223 --storage-dir ./identity-b
+```
+
+## Clear state
+
+```bash
+rm -rf ./obscura-data
+```

docs/README.md

@@ -0,0 +1,52 @@
+Obscura is an open-source headless browser engine written in Rust. It runs JavaScript via V8, speaks the Chrome DevTools Protocol, and works as a drop-in replacement for headless Chrome with Puppeteer and Playwright.
+
+## Versus headless Chrome
+
+| Metric      | Obscura  | Headless Chrome |
+| ----------- | -------- | --------------- |
+| Memory      | 30 MB    | 200+ MB         |
+| Binary size | 70 MB    | 300+ MB         |
+| Startup     | Instant  | ~2s             |
+| Page load   | 85 ms    | ~500 ms         |
+| Anti-detect | Built-in | None            |
+| Puppeteer   | Yes      | Yes             |
+| Playwright  | Yes      | Yes             |
+
+## Quickstart
+
+- [Installation](Installation.md)
+- [Your first fetch](Your-first-fetch.md)
+- [Extract data](Extract-data.md)
+- [Connect Puppeteer or Playwright](Connect-Puppeteer-or-Playwright.md)
+
+## Guides
+
+- [Build from source](Build-from-source.md)
+- [Configure stealth and proxies](Configure-stealth-and-proxies.md)
+- [Markdown extraction](Markdown-extraction.md)
+- [Use with Puppeteer](Use-with-Puppeteer.md)
+- [Use with Playwright](Use-with-Playwright.md)
+- [Use the MCP server](Use-the-MCP-server.md)
+- [Use as a Rust library](Use-as-a-Rust-library.md)
+- [Persist cookies and storage](Persist-cookies-and-storage.md)
+- [Intercept and modify requests](Intercept-and-modify-requests.md)
+- [Run in production at scale](Run-in-production-at-scale.md)
+
+## Reference
+
+- [CLI reference](CLI-reference.md)
+- [Environment variables](Environment-variables.md)
+
+## Contributing
+
+- [Architecture overview](Architecture-overview.md)
+- [Adding a CDP method or Web API](Adding-a-CDP-method-or-Web-API.md)
+- [Testing and debugging](Testing-and-debugging.md)
+
+## Links
+
+- Source: https://github.com/h4ckf0r0day/obscura
+- Releases: https://github.com/h4ckf0r0day/obscura/releases
+- Issues: https://github.com/h4ckf0r0day/obscura/issues
+
+License: Apache-2.0.

docs/Run-in-production-at-scale.md

@@ -0,0 +1,145 @@
+## Docker
+
+```bash
+docker run -d \
+  --name obscura \
+  --restart unless-stopped \
+  -p 127.0.0.1:9222:9222 \
+  -v /srv/obscura/data:/data \
+  h4ckf0r0day/obscura \
+  serve --host 0.0.0.0 --storage-dir /data --stealth
+```
+
+The image runs `obscura serve` by default. Override with arguments after the image name.
+
+## Systemd
+
+`/etc/systemd/system/obscura.service`:
+
+```ini
+[Unit]
+Description=Obscura headless browser
+After=network.target
+
+[Service]
+ExecStart=/usr/local/bin/obscura serve --port 9222 --stealth --storage-dir /var/lib/obscura
+Restart=always
+RestartSec=5
+User=obscura
+Group=obscura
+LimitNOFILE=65536
+
+[Install]
+WantedBy=multi-user.target
+```
+
+```bash
+systemctl enable --now obscura
+journalctl -fu obscura
+```
+
+## Workers
+
+`obscura serve --workers N` runs N CDP server workers behind the listener.
+
+```bash
+obscura serve --workers 4
+```
+
+Use one worker per CPU core. Each worker handles its own pool of pages. Sessions are sticky to a worker.
+
+## V8 heap
+
+Default V8 heap is 4 GB on 64-bit systems. Override:
+
+```bash
+obscura serve --v8-flags "--max-old-space-size=2048"
+```
+
+Lower for memory-constrained hosts, raise for heavy SPAs.
+
+## Parallel scrape
+
+`obscura scrape` fans out a list of URLs across worker processes:
+
+```bash
+obscura scrape \
+  --concurrency 20 \
+  --format json \
+  --timeout 60 \
+  url1 url2 url3 ...
+```
+
+Reads URLs from stdin:
+
+```bash
+cat urls.txt | obscura scrape --concurrency 20 -
+```
+
+Requires `obscura-worker` next to `obscura` in `PATH`.
+
+## Resource limits
+
+Per-process memory cap with systemd:
+
+```ini
+[Service]
+MemoryMax=4G
+MemoryHigh=3G
+```
+
+Per-container with Docker:
+
+```bash
+docker run --memory=4g --cpus=2 ...
+```
+
+## Reverse proxy
+
+Expose obscura on TLS through nginx or caddy:
+
+```nginx
+location /obscura/ {
+  proxy_pass http://127.0.0.1:9222/;
+  proxy_http_version 1.1;
+  proxy_set_header Upgrade $http_upgrade;
+  proxy_set_header Connection "upgrade";
+  proxy_read_timeout 86400;
+}
+```
+
+CDP needs WebSocket upgrade and long read timeouts.
+
+## Authentication
+
+Obscura's CDP server has no built-in auth. Anyone who can reach the port can drive the browser. Options:
+
+- Bind to `127.0.0.1` and require SSH for access (default).
+- Put it behind a reverse proxy that enforces auth.
+- Use Docker network isolation.
+
+Never bind `0.0.0.0` on a public IP without one of the above.
+
+## Observability
+
+```bash
+obscura serve --verbose
+RUST_LOG=obscura=debug obscura serve
+```
+
+`--verbose` enables info-level logs. `RUST_LOG=obscura=debug` enables debug-level. Logs go to stderr.
+
+## Reliability and timeouts
+
+The engine is hardened so one page cannot hang, crash, or wedge a worker. A V8 watchdog terminates runaway scripts and microtask storms, DOM ops are panic-safe, cyclic DOM mutations are rejected, and the CDP server terminates any single command that overruns its budget so a hung session cannot stall the others. Scripted `fetch()`/XHR and navigation are timeout-bounded. You can point the server at arbitrary or heavy pages without a stuck worker.
+
+Tune the bounds with environment variables (see [Environment variables](Environment-variables.md)):
+
+```bash
+OBSCURA_NAV_TIMEOUT_MS=60000 \
+OBSCURA_CDP_COMMAND_TIMEOUT_MS=30000 \
+OBSCURA_FETCH_TIMEOUT_MS=20000 \
+  obscura serve
+```
+
+`OBSCURA_NAV_TIMEOUT_MS` is the per-navigation ceiling (default 30000). `OBSCURA_CDP_COMMAND_TIMEOUT_MS` is the per-CDP-command V8 deadline (default 60000, `0` disables). `OBSCURA_FETCH_TIMEOUT_MS` bounds scripted fetch/XHR and module loads (default 30000).

docs/SUMMARY.md

@@ -0,0 +1,34 @@
+# Table of contents
+
+* [Introduction](README.md)
+
+## Quickstart
+
+* [Installation](Installation.md)
+* [Your first fetch](Your-first-fetch.md)
+* [Extract data](Extract-data.md)
+* [Connect Puppeteer or Playwright](Connect-Puppeteer-or-Playwright.md)
+
+## Guides
+
+* [Build from source](Build-from-source.md)
+* [Configure stealth and proxies](Configure-stealth-and-proxies.md)
+* [Markdown extraction](Markdown-extraction.md)
+* [Use with Puppeteer](Use-with-Puppeteer.md)
+* [Use with Playwright](Use-with-Playwright.md)
+* [Use the MCP server](Use-the-MCP-server.md)
+* [Use as a Rust library](Use-as-a-Rust-library.md)
+* [Persist cookies and storage](Persist-cookies-and-storage.md)
+* [Intercept and modify requests](Intercept-and-modify-requests.md)
+* [Run in production at scale](Run-in-production-at-scale.md)
+
+## Reference
+
+* [CLI reference](CLI-reference.md)
+* [Environment variables](Environment-variables.md)
+
+## Contributing
+
+* [Architecture overview](Architecture-overview.md)
+* [Adding a CDP method or Web API](Adding-a-CDP-method-or-Web-API.md)
+* [Testing and debugging](Testing-and-debugging.md)

docs/Testing-and-debugging.md

@@ -0,0 +1,128 @@
+## Test suites
+
+### Rust unit and integration
+
+```bash
+cargo test --release
+```
+
+Crate-scoped:
+
+```bash
+cargo test -p obscura-cdp
+cargo test -p obscura-browser
+```
+
+By name:
+
+```bash
+cargo test runtime_click_submit_prevent_default
+```
+
+### CDP parity tests
+
+`crates/obscura-cdp/tests/cdp_*.rs` exercise CDP methods end-to-end with a real `dispatch` call and an in-process HTTP server.
+
+Pattern:
+
+```rust
+#[tokio::test(flavor = "current_thread")]
+async fn my_test() {
+    std::env::set_var("OBSCURA_ALLOW_PRIVATE_NETWORK", "1");
+    let url = serve_once().await;
+    let mut ctx = CdpContext::new();
+    let page_id = ctx.create_page();
+    let session_id = "session-1";
+    ctx.sessions.insert(session_id.to_string(), page_id.clone());
+
+    cdp(&mut ctx, 1, "Page.navigate", json!({"url": url}), session_id).await;
+    // assertions
+}
+```
+
+`serve_once` and `cdp` helpers are copied across the parity tests; reuse them.
+
+## Logging
+
+```bash
+RUST_LOG=obscura=info  obscura serve
+RUST_LOG=obscura=debug obscura serve
+RUST_LOG=obscura_cdp=trace,obscura_browser=debug obscura serve
+```
+
+Logs go to stderr.
+
+`--verbose` on any subcommand is equivalent to `RUST_LOG=obscura=info`.
+
+## Driving the CDP server manually
+
+```bash
+obscura serve --port 9222 --verbose
+```
+
+In another shell:
+
+```bash
+wscat -c ws://127.0.0.1:9222
+> {"id":1,"method":"Target.createTarget","params":{"url":"about:blank"}}
+> {"id":2,"method":"Target.attachToTarget","params":{"targetId":"...","flatten":true}}
+> {"id":3,"sessionId":"...-session","method":"Page.navigate","params":{"url":"https://example.com"}}
+> {"id":4,"sessionId":"...-session","method":"Runtime.evaluate","params":{"expression":"document.title"}}
+```
+
+Useful for reproducing what Puppeteer or Playwright is doing without their abstraction.
+
+## Common failure modes
+
+### `Target.createTarget timed out`
+
+Lock contention in the dispatcher. Should not happen on current main. If it does, run with `RUST_LOG=obscura_cdp=trace`, look for handlers that hold `v8_lock` across long awaits.
+
+### `page.goto()` returns `null` from Puppeteer
+
+Means `Network.requestWillBeSent` for the main document did not arrive with `requestId == loaderId`. Check `do_navigate` in `crates/obscura-cdp/src/domains/page.rs`.
+
+### `Cannot find context with specified id`
+
+Playwright's local context counter diverged from the server's `valid_context_ids`. Each navigation must allocate a fresh `executionContextId`. Check `ctx.next_isolated_context()` is called on every nav.
+
+### `V8_Fatal: heap->isolate() == Isolate::TryGetCurrent()`
+
+Two pages tried to use V8 concurrently. The `v8_lock` was bypassed, or a handler suspended a JS runtime while another isolate was entered. Search for direct `JsRuntime` access outside the lock.
+
+### Test hangs
+
+A handler is awaiting something that never resolves. Run with `RUST_LOG=obscura=trace` and check the last log line before the hang.
+
+## Reproducing user bug reports
+
+The integration suite in `tests/test_all.py` is the fastest path from a one-line repro to a regression test. Add the failing case as a new test function, get it failing, then fix.
+
+For Puppeteer / Playwright bug reports, the user's repro script usually drops straight in. Save it as `tests/repro_<issue>.js`, run with `node`, fix until it passes.
+
+## Profiling
+
+CPU with `perf` and a flamegraph:
+
+```bash
+cargo build --release
+perf record -F 99 -g -- ./target/release/obscura fetch https://heavy-spa.example
+perf script | flamegraph.pl > flame.svg
+```
+
+Memory with heaptrack:
+
+```bash
+heaptrack ./target/release/obscura serve
+```
+
+Tokio task inspection:
+
+```bash
+RUSTFLAGS="--cfg tokio_unstable" cargo build --release
+./target/release/obscura serve
+# in another shell
+tokio-console
+```
+
+Requires the workspace `tokio` dependency to be built with the `tracing` feature; not enabled by default, add it in the relevant `Cargo.toml` before profiling.

docs/Use-as-a-Rust-library.md

@@ -0,0 +1,68 @@
+The `obscura` crate embeds the engine in a Rust program with a `Browser` / `Page` / `Element` API plus a cookie store, no CDP round-trips. It builds V8 from source, so it is a git dependency rather than a crates.io release.
+
+## Add the dependency
+
+```toml
+[dependencies]
+obscura = { git = "https://github.com/h4ckf0r0day/obscura" }
+tokio = { version = "1", features = ["rt", "macros"] }
+anyhow = "1"
+```
+
+The first build compiles V8 from source, so it is slow and needs the same build tools as [Build from source](Build-from-source.md). Pin a tag for reproducible builds:
+
+```toml
+obscura = { git = "https://github.com/h4ckf0r0day/obscura", tag = "v0.1.7" }
+```
+
+## Quickstart
+
+```rust
+use obscura::Browser;
+use std::time::Duration;
+
+#[tokio::main]
+async fn main() -> anyhow::Result<()> {
+    let browser = Browser::builder()
+        .stealth(true)
+        .build()?;
+
+    let mut page = browser.new_page().await?;
+    page.goto("https://example.com").await?;
+
+    println!("URL: {}", page.url());
+    println!("HTML bytes: {}", page.content().len());
+
+    let el = page.wait_for_selector("h1", Duration::from_secs(5)).await?;
+    println!("Heading: {}", el.text());
+
+    let title = page.evaluate("document.title");
+    println!("Title: {}", title);
+
+    Ok(())
+}
+```
+
+## API surface
+
+`Browser::builder()` configures the engine: `.stealth(bool)`, `.proxy(url)`, `.user_agent(ua)`, `.storage_dir(dir)`, then `.build()`. `Browser::new()` uses defaults.
+
+`Page`:
+- `goto(url).await` navigate and wait for load
+- `content()` rendered HTML
+- `url()` current URL
+- `evaluate(js)` run JavaScript, returns a `serde_json::Value`
+- `query_selector(css)` first match as an `Element`, or `None`
+- `wait_for_selector(css, Duration).await` poll until present
+- `settle(max_ms).await` drive the event loop so async work (`fetch`, timers) completes
+
+`Element`: `text()`, `attribute(name)`, `click()`.
+
+`CookieStore`: `set`, `get_all`, `get_for_url`, `save_to_file`, `load_from_file`.
+
+## When to use which interface
+
+- Embedding the engine in a Rust service: this crate.
+- Driving from Node/Python with existing Puppeteer/Playwright code: the [CDP server](Connect-Puppeteer-or-Playwright.md).
+- Giving an AI agent browser tools: the [MCP server](Use-the-MCP-server.md).
+- One-off fetches and scraping from the shell: the [CLI](CLI-reference.md).

docs/Use-the-MCP-server.md

@@ -0,0 +1,92 @@
+`obscura mcp` exposes obscura as a Model Context Protocol server so MCP-capable clients (Claude Desktop, Claude Code, etc.) can drive it.
+
+## Run
+
+Stdio (default, for direct client integration):
+
+```bash
+obscura mcp
+```
+
+HTTP (for remote or shared use):
+
+```bash
+obscura mcp --http --port 3000
+```
+
+With stealth and proxy:
+
+```bash
+obscura mcp --stealth --proxy http://proxy.example.com:8080
+```
+
+## Tools exposed
+
+The server keeps a live browser session, so tools operate on the current page rather than taking a URL each call. Navigate first, then read or act.
+
+Navigation and lifecycle:
+
+- `browser_navigate`, `browser_back`, `browser_forward`, `browser_reload`, `browser_close`
+
+Read the page:
+
+- `browser_snapshot`: accessibility/DOM snapshot of the current page.
+- `browser_markdown`, `browser_links`, `browser_extract`: page as markdown, link list, or structured content.
+- `browser_interactive_elements`, `browser_detect_forms`: actionable elements and form fields.
+- `browser_get_attribute`, `browser_count`, `browser_search`: read an attribute, count matches, find text.
+
+Interact:
+
+- `browser_click`, `browser_fill`, `browser_fill_form`, `browser_type`, `browser_press_key`, `browser_select_option`, `browser_scroll`
+
+Wait and run JS:
+
+- `browser_wait_for`, `browser_wait_for_text`, `browser_evaluate`
+
+Diagnostics:
+
+- `browser_network_requests`, `browser_console_messages`
+
+Cookies and storage:
+
+- `browser_get_cookies`, `browser_set_cookie`, `browser_clear_cookies`, `browser_storage_state`, `browser_set_storage_state`
+
+Tabs:
+
+- `browser_tab_new`, `browser_tab_list`, `browser_tab_switch`, `browser_tab_close`
+
+## Claude Desktop
+
+Edit `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS) or `%APPDATA%\Claude\claude_desktop_config.json` (Windows):
+
+```json
+{
+  "mcpServers": {
+    "obscura": {
+      "command": "/path/to/obscura",
+      "args": ["mcp"]
+    }
+  }
+}
+```
+
+Restart Claude Desktop. The obscura tools appear in the tool list.
+
+## Claude Code
+
+```bash
+claude mcp add obscura /path/to/obscura mcp
+```
+
+## With stealth in config
+
+```json
+{
+  "mcpServers": {
+    "obscura": {
+      "command": "/path/to/obscura",
+      "args": ["mcp", "--stealth"]
+    }
+  }
+}
+```

docs/Use-with-Playwright.md

@@ -0,0 +1,110 @@
+## Setup
+
+```bash
+obscura serve --port 9222
+npm install playwright
+```
+
+## Connect
+
+```js
+const { chromium } = require('playwright');
+
+const browser = await chromium.connectOverCDP('ws://127.0.0.1:9222');
+const context = browser.contexts()[0] || await browser.newContext();
+const page = await context.newPage();
+```
+
+Use `connectOverCDP`, not `connect`. Playwright's `connect` speaks Playwright's own protocol.
+
+## Navigate
+
+```js
+await page.goto('https://example.com');
+await page.goto('https://example.com', { waitUntil: 'load' });
+await page.goto('https://example.com', { waitUntil: 'networkidle' });
+```
+
+Default is `domcontentloaded`. Other values: `load`, `networkidle`.
+
+## Evaluate
+
+```js
+const title = await page.evaluate(() => document.title);
+
+const items = await page.$$eval('.item', els => els.map(el => ({
+  text: el.textContent,
+  href: el.querySelector('a')?.href,
+})));
+```
+
+## Interact
+
+```js
+await page.click('#login-button');
+await page.fill('#username', 'alice');
+await page.fill('#password', 'secret');
+
+await page.waitForSelector('#dashboard');
+await page.waitForFunction(() => window.appReady === true);
+```
+
+## Locators
+
+```js
+await page.locator('button.submit').click();
+await page.getByRole('button', { name: 'Submit' }).click();
+await page.getByLabel('Email').fill('alice@example.com');
+```
+
+## Cookies
+
+```js
+await context.addCookies([{
+  name: 'session',
+  value: 'abc123',
+  domain: 'example.com',
+  path: '/',
+}]);
+
+const cookies = await context.cookies();
+```
+
+## Intercept requests
+
+```js
+await page.route('**/*', route => {
+  if (route.request().resourceType() === 'image') {
+    route.abort();
+  } else {
+    route.continue();
+  }
+});
+```
+
+## Multiple pages
+
+```js
+const page1 = await context.newPage();
+const page2 = await context.newPage();
+
+await Promise.all([
+  page1.goto('https://a.example.com'),
+  page2.goto('https://b.example.com'),
+]);
+```
+
+Pages share one V8 isolate. CPU-bound JS on one page blocks the others.
+
+## Disconnect
+
+```js
+await browser.close();  // closes the CDP connection, leaves obscura serve running
+```
+
+## Not supported
+
+- `page.screenshot()`, `page.pdf()`: no pixel rendering.
+- `page.video()`, tracing artifacts that need a real browser.
+- `BrowserContext` storage state save/restore: use `--storage-dir` on `obscura serve` instead, see [Persist cookies and storage](Persist-cookies-and-storage.md).
+- Service workers.

docs/Use-with-Puppeteer.md

@@ -0,0 +1,124 @@
+## Setup
+
+```bash
+obscura serve --port 9222
+npm install puppeteer-core
+```
+
+## Connect
+
+```js
+const puppeteer = require('puppeteer-core');
+
+const browser = await puppeteer.connect({
+  browserWSEndpoint: 'ws://127.0.0.1:9222',
+});
+```
+
+Use `puppeteer-core`, not `puppeteer`. The `puppeteer` package bundles a Chrome download.
+
+## Navigate
+
+```js
+const page = await browser.newPage();
+await page.goto('https://example.com');
+await page.goto('https://example.com', { waitUntil: 'load' });
+await page.goto('https://example.com', { waitUntil: 'networkidle0', timeout: 60000 });
+```
+
+Default `waitUntil` is `domcontentloaded`. Other values: `load`, `networkidle2`, `networkidle0`.
+
+## Evaluate
+
+```js
+const title = await page.evaluate(() => document.title);
+
+const items = await page.evaluate(() => {
+  return Array.from(document.querySelectorAll('.item')).map(el => ({
+    text: el.textContent,
+    href: el.querySelector('a')?.href,
+  }));
+});
+```
+
+## Interact
+
+```js
+await page.click('#login-button');
+await page.type('#username', 'alice');
+await page.fill('#password', 'secret');  // alias of .type for compat
+
+await page.waitForSelector('#dashboard');
+await page.waitForFunction(() => window.appReady === true);
+```
+
+## Cookies
+
+```js
+await page.setCookie({
+  name: 'session',
+  value: 'abc123',
+  domain: 'example.com',
+  path: '/',
+  httpOnly: true,
+  secure: true,
+});
+
+const cookies = await page.cookies();
+```
+
+For session persistence across runs see [Persist cookies and storage](Persist-cookies-and-storage.md).
+
+## Intercept requests
+
+```js
+await page.setRequestInterception(true);
+
+page.on('request', req => {
+  if (req.resourceType() === 'image') {
+    req.abort();
+  } else {
+    req.continue();
+  }
+});
+```
+
+See [Intercept and modify requests](Intercept-and-modify-requests.md).
+
+## Expose a Node callback
+
+```js
+await page.exposeFunction('logFromPage', (msg) => {
+  console.log('page:', msg);
+});
+
+await page.evaluate(() => {
+  window.logFromPage('hello from the browser');
+});
+```
+
+## Multiple pages
+
+```js
+const page1 = await browser.newPage();
+const page2 = await browser.newPage();
+
+await Promise.all([
+  page1.goto('https://a.example.com'),
+  page2.goto('https://b.example.com'),
+]);
+```
+
+Pages share one V8 isolate. Concurrent JS execution serializes through a lock. CPU-bound JS on one page blocks the others.
+
+## Disconnect
+
+```js
+await browser.disconnect();  // leaves obscura serve running
+```
+
+## Not supported
+
+- `page.screenshot()` and `page.pdf()`: no pixel rendering.
+- `page.emulate()` device emulation: viewport metadata only, no real layout.
+- Service workers: not implemented.

docs/Your-first-fetch.md

@@ -0,0 +1,71 @@
+`obscura fetch` loads a URL, runs its JavaScript, and prints the result.
+
+## Load a page
+
+```bash
+obscura fetch https://example.com
+```
+
+Prints the rendered HTML.
+
+## Run JavaScript with `--eval`
+
+```bash
+obscura fetch https://example.com --eval "document.title"
+```
+
+```
+"Example Domain"
+```
+
+Returns JSON:
+
+```bash
+obscura fetch https://news.ycombinator.com \
+  --eval "Array.from(document.querySelectorAll('.titleline a')).slice(0, 5).map(a => a.textContent)"
+```
+
+## Multi-statement eval
+
+`--eval` evaluates one expression. For multiple statements, wrap in an IIFE:
+
+```bash
+obscura fetch https://example.com --eval "(function(){
+  const links = document.querySelectorAll('a');
+  return Array.from(links).map(a => a.href);
+})()"
+```
+
+A bare block starting with `const` or `let` returns `null` because V8 gives top-level declarations an empty completion value.
+
+## Wait for the right moment
+
+CLI default is `load`. For faster returns on slow sites:
+
+```bash
+obscura fetch https://my-spa.example --wait-until domcontentloaded --eval "document.title"
+```
+
+| Level              | Returns when                                  |
+| ------------------ | --------------------------------------------- |
+| `domcontentloaded` | HTML parsed, scripts ran                      |
+| `load`             | All subresources finished (default)           |
+| `networkidle2`     | ≤2 network connections active for 500ms       |
+| `networkidle0`     | 0 network connections active for 500ms        |
+
+(When driving obscura via Puppeteer or Playwright the default is `domcontentloaded` to match client expectations.)
+
+## Common flags
+
+```
+--user-agent "..."        Override the User-Agent
+--timeout 30                Navigation timeout in seconds (default 30)
+--wait 5                    Extra wait after the page settles, in seconds (default 5)
+--selector ".main"          CSS selector to narrow output to
+--proxy http://host:port    Route through a proxy
+--stealth                   Stealth client (TLS fingerprint, tracker blocking)
+-o, --output file.html      Write output to a file
+-q, --quiet                 Suppress info logging
+```
+
+Full list: [CLI reference](CLI-reference.md).