> ## Documentation Index
> Fetch the complete documentation index at: https://docs.credibledata.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Performance & Cost
> Materialize sources and index dimensions for fast, searchable models
Your model is documented and secured — the last stage before publishing is **optimizing how your modeled data is stored and served, for performance and cost**. Credible maintains managed, derived copies of your data, and you opt parts of your model into them with a single annotation. The platform builds the copy, keeps it fresh, reuses it where it can, and wires it into serving — so your model gets faster, cheaper, and more searchable without you managing any of the machinery.
There are three kinds of derived copy, each opted into at its natural grain:
| Derived copy | Goal | What you opt in | Annotation |
| ----------------------------------- | ---------------------------------------------- | --------------- | -------------------------------- |
| **Materialized table** | Make a source fast and cheap to query | A source | `#@ persist` on the source |
| **Search index** | Make a dimension's values searchable | A dimension | `#(index)` on the dimension |
| **Pre-aggregation** *(coming soon)* | Make a measure fast and cheap at coarse grains | A measure | `#@ preaggregate` on the measure |
Each is derived from your published model, kept fresh by Credible, and reused automatically. You choose *what* to persist, *what* to make searchable, and *what* to roll up; Credible owns everything else.
`#(index)` appears in two stories. For **what to index and how it improves AI retrieval**, see [Discovery Metadata](/how-to/modeling/metadata-tags). This page covers the other side: how the index is built, kept fresh, and served.
## Serving Behavior
A query on a persisted source **serves from the derived copy when one exists, and otherwise runs live** against your warehouse. The result is always correct — if a derived copy isn't available yet, the query is simply slower, not wrong.
* A **materialized table** persists a source's data as a physical table and routes queries to it.
* A **search index** embeds a dimension's values so they are findable by value search, filter suggestions, and the AI agent.
## How Materialization and Indexing Compose
The two features work together automatically, with no extra configuration. If you index a dimension on a source you have also materialized, Credible builds the search index **from the materialized table** instead of re-scanning the warehouse — and reverts to the warehouse if you un-materialize the source.
You can rely on three properties:
* **Consistent** — the index reflects the materialized snapshot, so searchable values match what queries return.
* **Stable** — a table-backed index refreshes exactly when its source table refreshes. An index on a source you have *not* materialized refreshes on publish and on demand, and — if you declare a freshness window — on that window.
* **Cheap** — indexing reuses the table you already built instead of paying to re-scan the warehouse.
The recommended pattern for an expensive, frequently-searched source is **"materialize the source, then index its dimensions."** Credible sequences the work for you so the index is always built after the table it derives from.
## Deciding What to Persist
Queried often or expensive to compute? Add `#@ persist`.
Values that users or the agent search or filter by? Add `#(index)`.
Credible links them (index-from-table) and orders them automatically.
Neither annotation: queried straight from the warehouse, not searchable.
The annotation alone is the intended usage. Defaults are chosen so the platform can optimize on your behalf — deduplicating copies across model versions and scheduling refreshes to meet a freshness objective. At most, add a freshness window for data with a real staleness requirement.
Two guardrails are enforced when you publish an indexed dimension: it may be partitioned by **at most one required filter**, and it may **not** sit on a source that requires parameters. Both surface as publish-time errors rather than silent wrong answers.
## Configuration
### Annotations
Add the annotation to the source or dimension you want to persist. Options are optional `key="value"` pairs; omit them to accept the platform default.
```malloy theme={"languages":{"custom":["/languages/motly.tmGrammar.json","/languages/malloy.tmGrammar.json"]}}
#@ persist name="orders_fast" refresh="incremental" watermark="order_date" freshness.window="24h"
source: orders is conn.table('sales.orders') extend {
#(index)
dimension: status is order_status
}
```
* `name` — choose where the materialized table lands (optional; container-qualifiable).
* `refresh` — `"full"` (the default) rebuilds the whole copy; `"incremental"` applies only what changed and requires a `watermark` (see [Incremental Refresh](#incremental-refresh)).
* `watermark` / `merge_key` — how an incremental table finds and applies new rows (see [Incremental Refresh](#incremental-refresh)).
* `freshness.window` — the staleness objective the platform schedules against (see [Freshness](#freshness)).
### Package Manifest
Reuse scope and the refresh cadence are declared once for the whole package in `publisher.json` (the same manifest described in [Publishing](/how-to/modeling/publishing)):
```json theme={"languages":{"custom":["/languages/motly.tmGrammar.json","/languages/malloy.tmGrammar.json"]}}
{
"scope": "package",
"materialization": { "freshness": { "window": "24h", "fallback": "live" } }
}
```
* **`scope: package`** (the default) — a derived copy is reused across the package's versions whenever they define the same thing. Maximal reuse, lowest cost.
* **`scope: version`** — each version keeps its own copies, with no cross-version reuse. Choose this when you want to own an exact rebuild schedule for a version.
Declare **either** a `freshness` objective **or** an explicit `materialization.schedule`, never both. A fixed schedule is the power-tier option and is only valid under `scope: version`.
## Incremental Refresh
By default (`refresh="full"`) every refresh recomputes the whole table. For a large, append-mostly source — a fact table that grows daily — that means re-reading years of data to add one day. Declare `refresh="incremental"` instead, and each refresh reads only the rows that are new since the last build and applies them to the existing table:
```malloy theme={"languages":{"custom":["/languages/motly.tmGrammar.json","/languages/malloy.tmGrammar.json"]}}
#@ persist refresh="incremental" watermark="order_date"
source: daily_revenue is orders -> {
group_by: order_date
aggregate: revenue is amount.sum()
}
```
The whole declaration lives on the `#@ persist` tag — the source body is exactly what you would write with no persistence at all, queries against the source are unchanged, and search indexes are already incremental with no declaration needed.
| Key | Means |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `refresh` | Set to `"incremental"` to advance the table with a bounded delta instead of a full recompute. Requires `watermark`. |
| `watermark` | Names the one output dimension a refresh derives its range from — an event timestamp, an order date, an ingestion time. Its values must be **monotone**: a given row's watermark value never decreases. |
| `merge_key` | Declare this **only when a row's watermark value moves** (e.g. `watermark="updated_at"` on a mutable table). Names the row's stable identity — one or more output dimensions, comma-separated — so a changed row is merged in place of its stale copy instead of appended beside it. |
The three keys form a chain — `merge_key` requires `watermark`, and `watermark` requires `refresh="incremental"` — and publishing fails with a targeted error if any link is missing, if a named dimension doesn't resolve to an output column, or if it names a measure. You find out where you declared it, not by watching a table that never advances.
### Which shape is yours
**A rollup or an append-only fact — no `merge_key`.** A row's `order_date` or `ingested_at` never changes, so each refresh replaces its date range outright. This also picks up rows deleted upstream *within* the refreshed range:
```malloy theme={"languages":{"custom":["/languages/motly.tmGrammar.json","/languages/malloy.tmGrammar.json"]}}
#@ persist refresh="incremental" watermark="ingested_at"
source: events is conn.table('raw.events') -> {
select: ingested_at, event_id, user_id, payload
}
```
**A mutable table — `watermark` plus `merge_key`.** `updated_at` moves when a row changes, so the platform needs `id` to find and replace the stale copy:
```malloy theme={"languages":{"custom":["/languages/motly.tmGrammar.json","/languages/malloy.tmGrammar.json"]}}
#@ persist refresh="incremental" watermark="updated_at" merge_key="id"
source: accounts is conn.table('raw.accounts')
```
**No monotone dimension — leave `refresh` unset.** A small lookup table overwritten wholesale upstream has nothing to order rows by; full-copy is a supported, cheap answer.
### Limits and repairs
Incremental trades completeness for cost, and two gaps are disclosed at publish rather than solved:
* **Late data.** A row that arrives with a watermark value below the range already covered is never picked up automatically.
* **Hard deletes.** A row deleted upstream can't appear in any delta, so a `merge_key` source retains it. Prefer **soft deletes** — a tombstone flag arrives as an ordinary update — and keep the flag in the persisted source's output, filtering it in consuming views instead.
Both are repaired the same two ways: correct the row upstream and advance its watermark so the next refresh applies it, or force a full rebuild with a **Rerun** from the package page (or `forceFullRebuild` on the runs API). Changing the model always triggers a full rebuild automatically — a delta is never applied across a logic change.
Non-additive measures — an exact `count_distinct`, a median — are **safe** in incremental sources: each refresh recomputes affected output rows from the full input rather than merging stored partial aggregates. Window calculations that look *forward* along the watermark (`lead()`, whole-partition percentages) are rejected at publish, because rows already materialized would go silently stale; trailing windows are fine.
## Pre-Aggregations
**Coming soon** — pre-aggregation support is under active development.
A pre-aggregation makes a **measure** fast and cheap at coarse grains. You mark a hot measure and its rollup grain; Credible maintains a rolled-up copy and silently answers coarse queries from it. You never hand-write a rollup source, and no query ever names one — routing is a property of the platform, not a judgment the caller (or the AI agent) makes per query:
```malloy theme={"languages":{"custom":["/languages/motly.tmGrammar.json","/languages/malloy.tmGrammar.json"]}}
#@ preaggregate grain="order_time.day, category"
measure: total_revenue is amount.sum()
```
* `grain` — **required**: the dimensions the rollup stores. A query is served from the rollup when everything it groups by and filters on is covered by the grain — a coarser truncation of a stored time dimension (`order_time.month` over a `day` grain) counts. Anything else falls back to the base source and runs live: "unsupported" and "unaccelerated" are the same, correct outcome.
* `#@ -preaggregate` pins a measure to the base even when a covering rollup exists — the escape hatch for a consumer that can't tolerate the rollup's freshness.
Routing is correctness-aware — the platform never serves a silently wrong number from a rollup:
* **Additive measures** (`sum`, `count`, `min`, `max`) and `avg` are re-aggregated from the rollup at any covered grain.
* **Non-additive measures** (`count(distinct)`, `median`, percentiles) can't be correctly re-aggregated to a coarser grain, so a rollup answers them only at exactly its declared grain — other grains run live. You're told this once, as a publish-time warning on the measure.
Everything else on this page applies unchanged. Measures declared at the same grain pack into a single rollup table, built by the same runs — from the materialized table when the base source is also `#@ persist`-ed, straight from the warehouse when it isn't (often the right choice: a rollup is frequently worth maintaining when a full copy of the base is not). Rollups share their base's freshness window — a rollup is never fresher than the table it was built from, and a stale rollup is skipped in favor of the base, never served — and they're reused and garbage-collected like any other derived copy.
For an expensive source with hot measures, the recommended pattern extends to a three-part bundle: **persist the source, index its searchable dimensions, and pre-aggregate its hot measures** — three annotations, with Credible sequencing all of it.
## Freshness
`freshness.window` is an **objective**, not a fixed refresh time: it tells Credible how stale the derived copy is allowed to get, and the platform schedules refreshes to meet it. This lets Credible batch work, run off-peak, and skip a refresh any recent publish or on-demand run already covered.
The `fallback` setting controls what a query does when a materialized table is older than its window — `live` runs the query against the warehouse instead of serving stale data.
Search indexes surface their staleness on the version page and in search and retrieval responses, so the agent can tell when suggestions come from an older snapshot.
## Builds and Refreshes
* **On publish** — Credible builds every persisted source and search index for the new version automatically.
* **On demand** — trigger a **Rerun** from the package page (or the runs API) to force a rebuild. You can rerun a whole version, or a single source or dimension — optionally including its upstream persisted sources.
* **On a schedule** — the platform refreshes derived copies to meet their freshness objectives.
The version page shows **Materialized sources** and **Indexed dimensions** side by side, each with a simple status and its build and refresh history, so you can see at a glance whether a version is fully built.
### Storage Reclamation
Credible **garbage-collects every unused derived copy** — a materialized table or index is kept only while an unarchived package version references it. Archiving a version releases its references, and any copies no longer referenced by another version are reclaimed automatically. So the way to keep storage costs down is to **archive package versions you no longer use** — [auto-archive](/how-to/modeling/publishing#auto-archive) (on by default) does this for you on a retention window you control.
## Next Steps
Publish to build your materialized tables and indexes
How search indexes power AI discovery and analysis