Proposal: add self-hosted Firecrawl to the homelab and wire Hermes web_extract to it #52

New issue

Closed

opened 2026-08-29 15:37:37 +02:00 by pazpi · 0 comments

pazpi commented

2026-08-29 15:37:37 +02:00

Owner

Context

The Hermes agent on the hermesagent host uses a self-hosted SearXNG instance (search.pasetto.me) as its web search backend. Search works, but the page-content extraction capability (web_extract) is currently broken: SearXNG is a search-only backend and its plugin reports "SearXNG is a search-only backend and cannot extract URL content." (see plugins/web/searxng -> supports_extract() returns False in tools/web_tools.py:881).

Because of this, the agent cannot extract/read the full text of web pages - only search snippets - which blocks many workflows (extracting product pages, docs, articles, etc.).

Per the Hermes docs and the error message itself (tools/web_tools.py:888), the fix is to set web.extract_backend to a real extract-capable backend: firecrawl, tavily, exa, or parallel.

We chose Firecrawl, and - consistent with the rest of the homelab - we want it self-hosted rather than depending on the Firecrawl cloud API.

What Hermes supports

The Firecrawl web plugin (plugins/web/firecrawl) supports a self-hosted endpoint via the FIRECRAWL_API_URL env var, alongside FIRECRAWL_API_KEY:

# hermes config
web:
  extract_backend: firecrawl

# ~/.hermes/.env
FIRECRAWL_API_KEY=***          # required even for self-hosted
FIRECRAWL_API_URL=https://firecrawl.<home-domain>   # self-hosted instance override

FIRECRAWL_API_URL overrides the default cloud endpoint (https://api.firecrawl.dev), so the agent talks to our own instance.
FIRECRAWL_API_KEY is still required by the plugin's availability check (check_firecrawl_api_key(), plugins/web/firecrawl/provider.py:170).

What we need from the NixOS homelab (this repo)

We need to deploy a self-hosted Firecrawl instance as a new container service. Firecrawl is normally run via Docker with several backing services (Redis, Postgres, etc.), so a docker-compose-style deployment inside our Docker LXC host (see containers/) is likely the cleanest fit, consistent with other self-hosted apps in this repo.

Acceptance criteria

Firecrawl is reachable from hermesagent on the tailnet (e.g. http://firecrawl...:3002/v1)
FIRECRAWL_API_URL + FIRECRAWL_API_KEY are set in the Hermes .env.
web.extract_backend: firecrawl is set in Hermes config.
web_extract("https://example.com") returns the page content (no more "search-only backend" error).

### Context The Hermes agent on the `hermesagent` host uses a self-hosted SearXNG instance (`search.pasetto.me`) as its web **search** backend. Search works, but the **page-content extraction** capability (`web_extract`) is currently broken: SearXNG is a *search-only* backend and its plugin reports `"SearXNG is a search-only backend and cannot extract URL content."` (see `plugins/web/searxng` -> `supports_extract()` returns False in `tools/web_tools.py:881`). Because of this, the agent cannot extract/read the full text of web pages - only search snippets - which blocks many workflows (extracting product pages, docs, articles, etc.). Per the Hermes docs and the error message itself (`tools/web_tools.py:888`), the fix is to set `web.extract_backend` to a real extract-capable backend: **firecrawl**, **tavily**, **exa**, or **parallel**. We chose **Firecrawl**, and - consistent with the rest of the homelab - we want it **self-hosted** rather than depending on the Firecrawl cloud API. ### What Hermes supports The Firecrawl web plugin (`plugins/web/firecrawl`) supports a **self-hosted endpoint** via the `FIRECRAWL_API_URL` env var, alongside `FIRECRAWL_API_KEY`: ```yaml # hermes config web: extract_backend: firecrawl ``` ```bash # ~/.hermes/.env FIRECRAWL_API_KEY=*** # required even for self-hosted FIRECRAWL_API_URL=https://firecrawl.<home-domain> # self-hosted instance override ``` - `FIRECRAWL_API_URL` overrides the default cloud endpoint (`https://api.firecrawl.dev`), so the agent talks to our own instance. - `FIRECRAWL_API_KEY` is still required by the plugin's availability check (`check_firecrawl_api_key()`, `plugins/web/firecrawl/provider.py:170`). ### What we need from the NixOS homelab (this repo) We need to **deploy a self-hosted Firecrawl instance** as a new container service. Firecrawl is normally run via Docker with several backing services (Redis, Postgres, etc.), so a `docker-compose`-style deployment inside our Docker LXC host (see `containers/`) is likely the cleanest fit, consistent with other self-hosted apps in this repo. ### Acceptance criteria - [x] Firecrawl is reachable from `hermesagent` on the tailnet (e.g. `http://firecrawl...:3002/v1`) - [x] `FIRECRAWL_API_URL` + `FIRECRAWL_API_KEY` are set in the Hermes `.env`. - [x] `web.extract_backend: firecrawl` is set in Hermes config. - [x] `web_extract("https://example.com")` returns the page content (no more "search-only backend" error).

pazpi added reference master

2026-08-30 09:30:25 +02:00

pazpi removed reference master

2026-08-30 09:30:30 +02:00

pazpi self-assigned this

2026-08-30 09:30:42 +02:00

pazpi added reference master

2026-08-30 09:31:14 +02:00

pazpi referenced this issue from a commit

2026-08-30 10:03:19 +02:00

Add self-hosted Firecrawl Compose stack for Hermes page extraction.

pazpi referenced this issue