headless-driver

headless-driver # headless-driver ### Web search for Python and AI agents **Search the web, read any page as clean Markdown, and give your AI agent both.**
Free, fast, no API key. PyPI version Python versions Downloads Package status License Tests Coverage Selenium Playwright MCP Docs Chrome headless Platforms Stars Forks Issues Last commit PRs welcome Sponsor Buy me a coffee
[**Documentation**](/headless-driver/DOCS.html) · [Quick start](#quick-start) · [Performance](#performance) · [CLI](#command-line) · [Engines](#search-engines) · [Roadmap](/headless-driver/TODO.html) · [PyPI](https://pypi.org/project/headless-driver/)

What you can do

How it keeps working

Features

Install

pip install "headless-driver[impersonate]"   # recommended: fast browserless search
pip install "headless-driver[playwright]"    # Playwright backend (then: playwright install chromium)
pip install "headless-driver[all]"           # everything
pip install headless-driver                  # Selenium only
Extra Adds Use it for
impersonate curl_cffi, beautifulsoup4 Browser TLS fingerprint; what transport="auto" prefers. The one to use from the cloud.
http requests, beautifulsoup4 Plain browserless fallback
playwright playwright browser="playwright", extract(), capture_json(), rendering JavaScript pages in fetch_markdown()
mcp mcp headless-driver mcp (Python 3.10+)
agents langchain-core, pydantic Toolkit.langchain_tools() and typed tool schemas
tokens tiktoken Exact token counts in fetch_markdown()
redis redis cache="redis://…"
fast lxml Faster parsing at volume, used automatically

Needs Python 3.9+. Search needs no browser at all; browser automation needs an installed Chrome or Chromium (a matching ChromeDriver is downloaded automatically), or Playwright’s own build via playwright install chromium.

On Debian/Ubuntu you can install a system browser and driver with:

sudo apt update && sudo apt install -y chromium chromium-driver

Quick start

from headless import Headless

with Headless() as driver:
    driver.get("https://example.com")
    print(driver.title)

Search the web — no browser needed:

from headless import AdvancedSearchScraper

with AdvancedSearchScraper(max_results=5) as scraper:
    response = scraper.search("python headless browser")

    for item in response:                 # iterates like a list
        print(item["title"], item["url"])

    print("answered by", response.engine)
    if response.blocked:                  # every engine refused, not "no results"
        print("blocked:", [str(a) for a in response.refused])

Read a page as Markdown, ready for an LLM:

from headless import fetch_markdown

doc = fetch_markdown("https://docs.python.org/3/library/asyncio.html", chunk_tokens=500)
print(doc.title, doc.tokens)
for chunk in doc.chunks:
    print(chunk.heading, chunk.tokens)

Give an agent web search and page reading:

from headless import Toolkit

toolkit = Toolkit()
response = client.messages.create(model="claude-sonnet-5", tools=toolkit.anthropic_tools(),
                                  messages=messages, max_tokens=2048)
messages.append({"role": "user", "content": toolkit.handle_anthropic_tool_use(response.content)})
# also: openai_tools() / handle_openai_tool_calls(), langchain_tools(),
#       llamaindex_tools(), crewai_tools()

Or connect Claude Desktop, Claude Code, Cursor or VS Code over MCP:

{"mcpServers": {"web": {"command": "headless-driver", "args": ["mcp"]}}}

Ask several engines at once and rank by agreement:

with AdvancedSearchScraper(region="uk-en") as scraper:
    response = scraper.search('site:linkedin.com/in "Jane Doe" Credo Capital',
                              mode="aggregate", deadline=8)
    for hit in response:
        print(hit["votes"], hit["engines"], hit["url"])
    # 3 ['brave', 'duckduckgo', 'mojeek'] https://www.linkedin.com/in/jane-doe

Being told you were blocked is the difference between backing off and recording a false negative. Engines CAPTCHA cloud address ranges, so the same code that works on a laptop can get nothing from ECS — response.blocked, response.rate_limited, response.retry_after and response.cooling say which happened.

Search in parallel:

from headless import ScraperPool

with ScraperPool(size=4) as pool:
    for query, response in pool.map(["python asyncio", "python typing"]):
        print(query, response.engine, len(response))

Playwright, with structured extraction:

from headless.playwright_driver import PlaywrightBrowser

with PlaywrightBrowser(block_resources=True) as browser:
    rows = browser.extract("https://news.ycombinator.com",
                           {"title": ".titleline > a", "link": ".titleline > a@href"},
                           item_selector="tr.athing")
    api = browser.capture_json("https://example.com/app", r"/api/")   # read the site's own JSON
    browser.screenshot("https://example.com", "page.png", full_page=True)

Screenshot and PDF with Selenium:

from headless import ExtendedHeadless

hl = ExtendedHeadless(stealth=True)
driver = hl.get_driver()
driver.get("https://example.com")

hl.screenshot("example.png")
hl.save_pdf("example.pdf")
hl.quit()

Performance

Measured 2026-09-29 from a laptop, three queries against Yahoo, then a five-engine aggregate search. Your numbers will differ; the ratios are the point.

How Time per query Answered Memory
transport="impersonate" (default with the extra) ~1.1 s 2/3 a few MB
transport="http" (plain requests) 0.2 s 0/3 — refused a few MB
Playwright (browser="playwright") 1.2 s warm, 2.9 s cold 3/3 ~300 MB
Selenium Chrome (transport="browser") 6–9 s 1/3 ~1 GB
mode="aggregate", 5 engines in parallel 1.07 s total 2 engines agreed on all 10 results a few MB

The plain-requests row is why impersonation is the default: a Python TLS handshake behind a Chrome User-Agent is refused on sight. Measure it from your own servers with headless-driver bench.

Command line

headless-driver search "python headless browser" -n 5
headless-driver search 'site:linkedin.com/in "jane doe"' --mode aggregate --region uk-en
headless-driver search "python web frameworks" --pages 3 --cache memory
headless-driver fetch https://example.com > page.md      # clean Markdown
headless-driver mcp                                      # serve tools to AI agents
headless-driver search "selenium stealth" --json | jq -r '.results[].url'
headless-driver search "python" --transport impersonate   # no browser at all
headless-driver engines                    # engines, capabilities, fallback order
headless-driver doctor                     # check chrome, driver, transports, connectivity
headless-driver doctor --engines           # check the engines still parse
headless-driver bench --min-ok-rate 0.5    # which engines answer from this address
headless-driver extract https://example.com -f title=h1 -f link=a@href --json
headless-driver shot https://example.com -o page.png --window 1280x720
headless-driver pdf  https://example.com -o page.pdf --browser playwright

doctor is the quickest way to explain a failing run:

── browser ──────────────────────────────────────────────
 ✓ chrome              Google Chrome 151.0.7922.140
 ✓ chromedriver        ChromeDriver 148.0.7778.179
 ! version match       chrome 151 vs driver 148 - a matching driver will be downloaded
── connectivity ─────────────────────────────────────────
 ✓ html.duckduckgo.com  reachable
 ✓ www.bing.com         reachable
── smoke test ───────────────────────────────────────────
 ✓ launch and navigate  page title 'ok'

 13 passed, 1 warning  +++++++++++++~

Colour switches off automatically when output is not a terminal, so piping gives plain text. NO_COLOR=1 or --no-color disables it; FORCE_COLOR=1 forces it on. Diagnostics go to stderr, keeping --json on stdout clean. Exit status is 0 on success, 1 when nothing could be produced, 2 for a usage error.

See DOCS.md for every command and flag.

Search engines

Searching starts at Brave, which answers from datacentre addresses and honours site: paths. If an engine refuses, has nothing, or stalls, the next is tried — browserless engines first, browser-only ones late, and Bing, which ignores site: paths, last:

brave → duckduckgo → mojeek → yahoo → google_basic → duckduckgo_lite
      → duckduckgo_js → startpage → google → yandex → bing

AdvancedSearchScraper(search_engine="yahoo")                 # start elsewhere
AdvancedSearchScraper(fallback=False)                        # single engine
AdvancedSearchScraper(fallback_engines=["mojeek", "yahoo"])  # custom order
AdvancedSearchScraper(mode="aggregate")                      # fan out by default
scraper.search("query", engine="duckduckgo")                 # force, one-off
scraper.engines_honoring_site()                              # trust these for absence

Each engine declares what it can do, so you do not have to discover it by observation:

scraper.capabilities("yahoo")
# {"js": False, "snippets": True, "honors_site": True, "provider": "bing", "method": "GET", ...}

Add your own engine with register_engine(): a spec can set the HTTP method, form or query builder, cookies, headers, a per-request URL builder, a pre-flight hook, a redirect unwrapper and “no results” markers. Details in DOCS.md.

More

Support

If this project saves you time, you can support its development:

Sponsor Buy me a coffee

Starring the repository helps too.

License

MIT — see LICENSE.

Happy coding 🚀

This site is open source. Improve this page.