> ## Documentation Index
> Fetch the complete documentation index at: https://docs.animam.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# How the crawl & corpus work

> The full content lifecycle: initial crawl, curation, automatic sync, and deterministic two-tier retrieval — including PDFs, JS-heavy sites, and contradictory content

Your agent answers from a **corpus** — a curated knowledge base — not from a live view of your website. This page describes the whole content lifecycle: how content gets in, how it stays fresh, and how it is retrieved at answer time. No magic claimed; every step below is how the platform actually behaves.

## The lifecycle at a glance

```
Initial read (crawl / import)  →  Curation (you edit)  →  Auto-sync (daily / on demand)
                                          ↓
                        Retrieval at chat time (two-tier, deterministic)
```

## 1. The initial read

### The instant demo crawl (`/try`)

When you paste a URL on [animam.ai/try](https://animam.ai/try), the crawl runs on **Cloudflare Browser Rendering** — a real headless browser, off our infrastructure:

* **JavaScript is rendered.** Single-page apps and JS-injected content are read like a visitor would see them.
* Up to **\~8 pages** are fetched per demo, priority-ordered: pricing, services, contact and about pages are preferred over blog archives.
* Each page is converted to markdown and becomes corpus entries, and a profile of the business (what it sells, to whom, in which tone) is synthesized for the agent's system prompt.
* Pages behind authentication, and content only reachable through interactions (open-a-modal pricing, configurators) are **not** captured — add those by hand or over the API.

The demo crawl is a seed, not a promise of completeness: its job is to produce a convincing agent in \~20 seconds that you then refine.

### Other ways in

| Source           | How                                          | Notes                                                                                       |
| ---------------- | -------------------------------------------- | ------------------------------------------------------------------------------------------- |
| Dashboard editor | **Knowledge Base → Add entry**               | Title, content, segment, priority, tags                                                     |
| REST API         | `POST /tenants/{slug}/corpus`                | See [API reference](/api-reference/endpoints/corpus)                                        |
| Bulk upsert      | `POST /tenants/{slug}/corpus/bulk`           | Idempotent by `externalId` — built for CMS integrations                                     |
| PDF upload       | Dashboard (or widget, per-tenant capability) | Stateless extraction: the PDF is converted to text, **no file is stored**                   |
| WordPress plugin | Automatic                                    | Every post/page save syncs to the corpus (`externalId: wp-{type}-{id}`), deletions included |
| Corpus sources   | Declared URLs, auto-synced                   | See below                                                                                   |

## 2. Curation — contradictions, old content, and who arbitrates

The honest answer to "what does the agent do with contradictory content?" is: **it doesn't arbitrate — you do.** The crawl seeds the corpus; the corpus is then yours:

* Every entry is **editable and deletable** in the dashboard or over the API.
* **Priority** and **segments** let you rank entries and scope them to parts of your site (different pages → different context).
* If two prices coexist because an old page still says the old price, the fix is curation: delete the stale entry, or fix the source page and resync.

This is a design choice, not a limitation: a model silently "resolving" contradictions in your business content is exactly the failure mode we avoid. The corpus is the single source of truth the agent answers from.

## 3. Staying fresh — auto-synced sources

You can declare up to **10 source URLs** per tenant (`CorpusSource`). Each source is re-fetched:

* **daily** (server-side job), or
* **on demand** — `POST /tenants/{slug}/corpus/sources/{id}/sync`, from the dashboard, or by asking your own agent over MCP.

The sync fetch is a plain server-side HTTP GET with HTML→markdown extraction, guarded against SSRF (DNS resolution checked). Two consequences to know:

* It is **cheap and unlimited** — freshness doesn't consume a browser-rendering quota.
* It does **not execute JavaScript**. For JS-only content, use the API, the bulk upsert, or the WordPress plugin instead — those push content rather than pull it.

WordPress sites don't need sources at all: the plugin pushes on every save, so the corpus follows your back office in near real time.

## 4. Retrieval at answer time — two-tier, deterministic

At chat time the agent uses a **two-tier corpus**:

1. **Tier 1 — summaries in the system prompt.** Every entry's summary is visible to the agent on every turn. It always knows *what it knows*.
2. **Tier 2 — full content on demand.** When a summary looks relevant, the agent calls the `EXPLORE_CORPUS` tool to fetch the entry's full text before answering.

Retrieval is **plain-text search over your entries — there is no embedding index**. This is deliberate:

* **Deterministic**: the same question finds the same entries. No vector-similarity surprises, no silent index drift after a resync.
* **Transparent**: when an answer is wrong, you look at the entry, fix the text, done. There is no opaque embedding space to debug.
* **Language-agnostic**: works identically in any language your corpus is written in.

The trade-off is honest too: exotic paraphrases match less magically than with embeddings. In practice the two-tier design compensates — the agent *reads the summaries of everything* on every turn, so it doesn't depend on lexical luck to know where to look.

## Testing it yourself

Before paying, measure rather than trust:

1. Build a demo on your real site (`/try`) and ask questions whose answers live on different pages.
2. Ask something **not** on your site — the agent should say it doesn't know or escalate, not improvise.
3. Add a corpus entry over the API and ask again — the change is live on the next message (summaries are rebuilt per conversation turn).
4. Declare a source URL, edit the target page, trigger `POST .../sources/{id}/sync`, and verify the answer changed.