Hermes Agent Web Scraping: 60x Faster With Free Backends

Hermes Agent now reads the web up to 60x faster and 49x cheaper with new free web search backends, including DuckDuckGo, Brave, and xAI Grok, plus per-capability provider mixing.

Hermes Agent Web Scraping: 60x Faster With Free Backends

Hermes Agent just got dramatically faster and cheaper at reading the web. I ran through the updated docs and the newly announced backends, and the headline number is real: up to 60x faster and 49x cheaper than before.

Here’s what changed and why it matters if you run Hermes for research, automation, or content work.

What Nous Actually Announced

Nous Research posted today that Hermes Agent’s web reading pipeline has been overhauled. The key change: scraping backends now pass clean content straight to the agent without redundant processing. Large pages get saved locally and paged on demand. The result is the same extraction quality at a fraction of the time and cost.

https://x.com/nousresearch/status/2071974594961977727

The updated web search docs spell out what that means in practice.

The New Free Backends

Before this update, you basically had two real options for running web search in Hermes Agent: Firecrawl (500 free credits/month, then paid) or self-hosting SearXNG (free but requires a Docker setup and a running server). This is the same pace of improvement as last week’s Mixture of Agents 2.0 update and the recent event loop stability fix.

Now there are three more free or near-free options:

DDGS (DuckDuckGo). Completely free, no API key, no signup. Hermes uses the ddgs Python package under the hood and lazy-installs it on first use. If you don’t want to manage another API key, this is your option.

Brave Search (free tier). 2,000 queries per month on the free plan. That is a lot of headroom for a typical power user. You need a BRAVE_SEARCH_API_KEY but the free tier is generous.

xAI (Grok). Available through a SuperGrok subscription or XAI_API_KEY. Results are LLM-generated rather than index-backed, so the output format is different (model-generated descriptions and URLs rather than raw search engine results). Worth noting: xAI is search-only.

These backends are all search-only — you pair any of them with a separate extract backend when you need full page content.

Per-Capability Provider Mixing

This is the sleeper feature. You can now split search and extract across different providers.

Before, your web backend was a single choice. Now you can run SearXNG or DDGS (free) for search queries and Firecrawl or Tavily for page extraction. The config looks like this:

web:
  search_backend: "searxng"
  extract_backend: "firecrawl"

This is where the 49x cheaper claim comes from. If you were paying Firecrawl for everything, switching search to a free backend and only using Firecrawl for extraction cuts the bill dramatically.

The auto-detection logic also got smarter. If you only set one API key — say SEARXNG_URL — Hermes auto-selects SearXNG. Multiple keys trigger backend selection from the config.

Smarter Page Extraction Pipeline

The web_extract tool now has a size-aware pipeline that avoids unnecessary LLM calls:

  • Under 5,000 characters: passed through as-is. No LLM call, no summarization, no cost.
  • 5,000 to 500,000 characters: single-pass summary via the optional auxiliary model, capped at about 5,000 chars of output.
  • 500,000 to 2,000,000 characters: chunked into 100K-char pieces, summarized in parallel, then synthesized.
  • Over 2,000,000 characters: refused with a hint to use a more focused source URL.

This is where the 60x faster number comes from. Small pages that used to hit an LLM now pass straight through. Medium pages get one efficient pass instead of a full context load. Large pages get parallel chunking instead of serial processing.

The summarization is a content compressor, not a paraphraser — quotes, code blocks, and key facts keep their original formatting. If the summarization fails or times out, Hermes falls back to the first 5,000 chars of raw content rather than returning a useless error.

You can also configure which model handles extraction summaries via auxiliary model settings, which is worth doing if your main chat model is an expensive reasoning model.

How to Switch Your Hermes Agent Web Backend

If you already run Hermes, switching to the new backends is straightforward:

  1. Run hermes tools and navigate to Web Search & Extract.
  2. Pick a provider. The wizard prompts for any required API key or URL and writes it to your config.
  3. For per-capability splitting, edit config.yaml directly:
web:
  search_backend: "searxng"
  extract_backend: "firecrawl"

DDGS (DuckDuckGo) needs no config at all beyond selecting it as your search backend. If pip install ddgs hasn’t run yet, Hermes lazy-installs it on the first web search call.

If you have a paid Nous Portal subscription, the Tool Gateway already includes managed web search and extract via Firecrawl — no extra API key needed. Run hermes setup --portal to turn it on.

What Is Still Missing

The federated search plugin that aggregates multiple search backends with LLM-based result ranking is still an open PR, not merged yet. That one will let you fan out queries to several backends simultaneously and rank the combined results. It is worth keeping an eye on, but don’t plan your workflow around it today.

xAI Grok search is search-only — no page extraction. If you need extract, pair it with Firecrawl, Tavily, Exa, or Parallel.

And the page size cap at 2 million characters for web_extract means genuinely massive pages (think War and Peace-length documentation sites) still need the browser tool or a targeted URL.

Bottom Line

This update is a big deal for anyone running Hermes Agent for web research, content monitoring, or data collection. You can now run web search entirely for free with DDGS or Brave, mix in a paid extract backend only when you need full page content, and get faster results because the pipeline skips unnecessary LLM processing on small pages.

For $0 a month for search and maybe $0 for extraction depending on volume, the economics of running Hermes as a research agent just got a lot better.

Tony Simons

Reviewed & Written By

Tony Simons

Independent tech reviewer and creator of Tony Reviews Things. 14 years of hands-on testing, software auditing, and workflow automation. I test the gear so you don't waste your money on junk.

Submit a Take

Your email address will not be published. Required fields are marked *