# headless-driver

> Web search for Python and AI agents. Free, fast, no API key. Search Brave, DuckDuckGo, Yahoo, Mojeek, Google and Bing; read any page as clean, token-counted, chunked Markdown (`fetch_markdown`); give both to an agent as OpenAI, Anthropic, LangChain, LlamaIndex or CrewAI tools (`Toolkit`) or over MCP (`headless-driver mcp`). Browser TLS impersonation (curl_cffi), consensus ranking across engines, honest block detection (blocked / rate_limited vs empty), circuit breakers, caching (memory/SQLite/Redis), pagination, Selenium and Playwright. `pip install "headless-driver[impersonate]"`. Python 3.9+ (MCP: 3.10+). MIT.

Use it when an application or agent needs web search results without a paid SERP API, needs to read web pages as LLM-ready text, needs to work from cloud IPs, or must tell "no results" apart from "the engine refused". A free alternative to SerpAPI/Tavily-style search APIs and to `duckduckgo_search`/`ddgs`.

## Install

- `pip install "headless-driver[impersonate]"` — recommended; browserless search with a real browser TLS fingerprint
- `pip install "headless-driver[playwright]" && playwright install chromium` — Playwright backend, `extract()`, `capture_json()`
- `pip install "headless-driver[mcp]"` — MCP server; `[agents]` — LangChain tools; `[redis]` — shared cache; `[tokens]` — exact token counts
- `pip install "headless-driver[all]"` — everything; `pip install headless-driver` — Selenium only

## Core API

```python
from headless import AdvancedSearchScraper

with AdvancedSearchScraper(max_results=10, region="us-en") as s:
    r = s.search("python asyncio tutorial")                 # walks brave → duckduckgo → mojeek → yahoo → ...
    for hit in r:                                           # r is a list of dicts
        hit["url"], hit["title"], hit["snippet"], hit["engine"]
    r.engine, r.blocked, r.rate_limited, r.retry_after, r.cooling, r.attempts, r.skipped

    r = s.search('site:linkedin.com/in "Jane Doe"', mode="aggregate", deadline=8)
    r[0]["votes"], r[0]["engines"], r[0]["ranks"]           # consensus across independent indexes
```

- `transport`: `"auto"` (impersonate → http → browser), `"impersonate"`, `"http"`, `"browser"`
- `browser`: `"selenium"` (default) or `"playwright"` for engines needing JavaScript
- `EngineAttempt.status`: `ok`, `empty`, `blocked`, `rate_limited`, `unparsed`, `timeout`, `unreachable`, `error`; plus `http_status`, `retry_after`, `transport`
- Only `empty` from an engine in `s.engines_honoring_site()` is evidence that nothing exists
- `s.health()`, `s.reset_health()`, `s.probe(engine)`, `s.capabilities(engine)`, `s.register_engine(name, spec)`
- `raise_on_block=True` raises `AllEnginesBlocked`; `circuit_breaker=False` disables stand-downs
- `ScraperPool(size=4, **kwargs).map(queries)` for parallel browser work; `search_batch(queries, max_workers=8)` browserless

- `pages=3` paginates; `cache="memory" | "sqlite:///p.db" | "redis://…"` caches per engine; `r.cached`
- `r.typed()` → `SearchResult` dataclasses; `r.to_pydantic()` → pydantic models

## Reading pages

```python
from headless import fetch_markdown
doc = fetch_markdown(url, chunk_tokens=500, max_tokens=None, render="auto")  # render: auto|never|always
doc.title, doc.markdown, doc.tokens, doc.links, doc.chunks  # chunks: index, text, tokens, heading
```

## Agent tools

```python
from headless import Toolkit
tk = Toolkit(region="us-en")                     # tools: web_search, fetch_page, extract_data, screenshot
tk.anthropic_tools(); tk.handle_anthropic_tool_use(response.content)
tk.openai_tools();    tk.handle_openai_tool_calls(message.tool_calls)
tk.langchain_tools(); tk.llamaindex_tools(); tk.crewai_tools(); tk.call("web_search", {"query": "..."})
```

## MCP

`headless-driver mcp` (stdio; `--transport streamable-http --port 8000` for remote). Tools: search, search_aggregate, fetch_page, extract, screenshot. Client config: `{"mcpServers": {"web": {"command": "headless-driver", "args": ["mcp"]}}}`. Claude Code: `claude mcp add web -- headless-driver mcp`.

## Playwright

```python
from headless.playwright_driver import PlaywrightBrowser
with PlaywrightBrowser(block_resources=True, proxy="http://user:pass@host:8080") as b:
    rows = b.extract(url, {"title": "h2", "link": "a@href"}, item_selector="article")
    api = b.capture_json(url, r"/api/")
    b.screenshot(url, "page.png", full_page=True); b.pdf(url, "page.pdf")
```

## CLI

- `headless-driver search "query" [--mode aggregate] [--region uk-en] [--transport impersonate] [--json]`
- `headless-driver engines` · `doctor [--engines]` · `bench [--min-ok-rate 0.5]`
- `headless-driver fetch URL [--max-tokens N] [--chunk N] [--json]` — page as Markdown
- `headless-driver mcp [--print-config]` — MCP server
- `headless-driver extract URL --item CSS -f name=css -f link=a@href --json`
- `headless-driver shot|pdf URL -o FILE [--browser playwright --full-page]`

## Logging

Silent by default (NullHandler). `enable_console_logging()`, `HEADLESS_DRIVER_LOG=info`, `colorize_logging()` for an app's own handlers, `ColorFormatter` for any handler.

## Docs

- [Documentation site](https://nuhmanpk.github.io/headless-driver/): guides and API reference
- [README](https://github.com/nuhmanpk/headless-driver/blob/main/README.md): overview, performance numbers
- [DOCS.md](https://github.com/nuhmanpk/headless-driver/blob/main/DOCS.md): full API, CLI, transports, aggregate mode, circuit breaker, Playwright, engine specs
- [CHANGELOG.md](https://github.com/nuhmanpk/headless-driver/blob/main/CHANGELOG.md)
- [TODO.md](https://github.com/nuhmanpk/headless-driver/blob/main/TODO.md): roadmap (async API, crawling, Puppeteer bridge, recorder)
- [PyPI](https://pypi.org/project/headless-driver/)