The Web Ingester

What it does

Give it a seed URL. It crawls the same domain breadth-first — respecting robots.txt, rate-limited, bounded by depth and page count — strips each page down to its readable main text, and hands that text to the Context Graph's normal ingestion pipeline.

The important detail is the last one: each page's text is inserted with its URL as the file path. That URL becomes the provenance field of the RelationContext on every (h, r, t, rc) the graph extracts from that page. You don't just get facts — you get facts that know which page they came from.

Binary and structured files discovered along the way (PDFs, Office docs, JSON) are downloaded into the workspace's input directory and picked up by the Context Graph's standard document scan. The whole thing runs as an async background job you start and poll.

The polite crawler

The default path is a static breadth-first crawl (SiteCrawler). It stays on the seed's host, optionally seeds itself from /sitemap.xml, and de-duplicates URLs by normalising them (dropping the fragment, default port, and trailing slash while preserving the query string). Obvious static assets are filtered by extension before they're ever fetched.

Every response is classified by content-type into three buckets:

For JavaScript-heavy sites, set render_js=true and the crawler wraps its fetcher in a headless Chromium (PlaywrightFetcher) that renders each page and also captures the data payloads the page's own scripts request over the network — which is where the connector framework comes in.

The connector framework

Some platforms hide their documents behind a JavaScript widget or a private API. The files never appear as <a href> links, so ordinary crawling — and even JS rendering — walks right past them. A connector is a plugin that cracks a platform.

Each connector is generic across its platform — it keys off the platform's signature, never a specific site — and implements a two-method contract:

class Connector(ABC):
    name: str = "connector"
    description: str = ""   # one line shown to the LLM selector

    def detect(self, *, requests, responses, page_url, html) -> Optional[dict]:
        # Recognise the platform from render signals.
        # Return a small template dict for resolve(), or None.

    async def resolve(self, request_context, template, *, max_files=None) -> List[FetchResult]:
        # Replay the platform's API with the browser session,
        # enumerate and download the hidden files.

Connectors register in one list — DEFAULT_CONNECTORS. Three ship enabled today:

Adding a connector is deliberately mechanical: copy example.py (a ready template that ships disabled), implement detect/resolve for the platform, and append it to DEFAULT_CONNECTORS.

LLM-driven connector selection

When you crawl with render_js=true and analyze=true, the ingester doesn't blindly run every connector against every page. After a page renders, it builds a compact signals summary — the <meta generator> tag, platform markers, and up to 25 API-ish request URLs — and asks the workspace's own LLM which connector fits:

# LLMConnectorSelector.select(signals, connectors)
{
  "connectors": ["wordpress"],
  "reason": "api.w.org link tag and /wp-json/ requests present"
}

The model sees a manifest of each connector's one-line description and returns the matching names (or an empty list if none clearly apply — which it is allowed to do). Two safety properties make this robust:

The selector is only constructed when analyze=true and the workspace has an LLM configured. Otherwise the connectors run purely on their deterministic detect() signatures.

Ingestion & provenance

Here is the full path from a seed URL to graph edges, as orchestrated by WebIngestor.ingest_site:

The site analyst

A large site produces a lot of noise — tracking pixels, CDN junk, empty JSON. With analyze=true, the SiteAnalyst filters what actually gets ingested:

• Binary documents (PDF, Office) are kept automatically — no LLM call needed.
• Obvious noise (tracking/CDN URLs, sub-40-byte bodies) is dropped deterministically.
• Ambiguous text resources (JSON/CSV/XML, sampled up to ~2500 chars, capped at 40 resources) go to the LLM, which returns keep/drop decisions and any document URLs it found inside those payloads — e.g. a PDF link buried in an API response.

The policy is default-drop, consistently: a resource is ingested only if it is a binary document (auto-kept, never LLM-judged) or the LLM explicitly keeps it. If the LLM call fails or returns unparseable output, the unjudged text candidates are dropped too — the same as an unmentioned candidate on the success path — so a transient LLM hiccup can't leak noise into the graph. The run is flagged ok:false so it can be re-run; binary documents are still kept.

The /scrape API

Three endpoints, all workspace-scoped via the LIGHTRAG-WORKSPACE header.

Request body — ScrapeRequest

{
  "url": "https://example-district.org",   // required, http(s)
  "max_pages": 50,        // 1–500   HTML pages to ingest
  "max_documents": 200,   // 0–2000  data files to download
  "max_depth": 2,          // 0–6     link depth from the seed
  "same_domain": true,      // stay on the seed's host
  "respect_robots": true,   // honour robots.txt
  "render_js": false,       // headless browser for JS sites
  "analyze": false,         // LLM analyst + connector selection
  "use_sitemap": true,      // seed from sitemap.xml
  "min_interval": 1.0       // 0–30s between hits to a host
}

Response — JobResponse

The summary is populated as the job progresses: ingested (HTML pages inserted), urls, documents ({url, file}), skipped ({url, reason}), track_ids, and a nullable analyst block. Job state moves running → scanning → done, or error.

# Start a JS-rendered, LLM-filtered crawl of a district site
curl -X POST http://localhost:9621/scrape \
  -H "LIGHTRAG-WORKSPACE: district_acme" \
  -H "Content-Type: application/json" \
  -d '{"url":"https://acme.k12.org","render_js":true,"analyze":true,"max_depth":3}'

# -> {"job_id":"a1b2c3d4e5f6","state":"running", ...}
curl http://localhost:9621/scrape/a1b2c3d4e5f6 -H "LIGHTRAG-WORKSPACE: district_acme"

Limits & config

There is no dedicated environment variable to enable or tune the web ingester — everything is set per request via ScrapeRequest. A few operational facts worth knowing:

• The feature is wired at server startup inside a guarded block: if its dependencies are missing, the server logs "Web-ingest API unavailable" and starts anyway.
• Auth reuses the server's existing API key, exactly like every other router.
• render_js=true needs Playwright + Chromium installed (pip install playwright && playwright install chromium). It's imported lazily, so static crawls have no such dependency.
• analyze=true uses the workspace's own LLM function — provider-agnostic, no separate config. Without an LLM the analyst and connector selector are silently skipped.