Schema Migration & Maintenance (v2, Beta)

This guide covers how to evolve a v2 (rep2 / "collection") database in place — adding and altering columns, converging to a target schema declaratively, updating and backfilling data, and compacting storage — without a full reload. v2 is in beta.

Everything here works on collection (rep2) tables. Where an operation also works on legacy table (rep1) tables, it's called out; where it's collection-only, that's noted too. See Schema Design for how to model a schema in the first place.

Add or alter a column

PUT /api/v2/schema/{table}/{column} both adds a new column and alters an existing one.

Add a new column; existing rows are backfilled from the optional value default:

PUT /api/v2/schema/users/tier
{ "type": "String", "value": "free" }

A non-nullable column with no value is rejected on a non-empty table — either supply a value, or make the column "nullable": true to backfill with null.

Alter an existing column's type or analyzer in place — the engine re-indexes the data (durable across optimize, not just a read-time view):

PUT /api/v2/schema/orders/qty
{ "type": "Long" }

Only type-compatible changes are allowed: widening Int → Long → Decimal, and String ↔ Text (switching the analyzer). Lossy changes (e.g. Int → String) are rejected. In-place alter is a collection (v2) capability — on a legacy table (v1) you must recreate the column.

Plan and apply

For a whole-schema change, converge declaratively — like terraform plan / apply. Post the target schema (same shape as create) and the engine diffs it against the current schema.

Plan is a dry run — it changes nothing and returns a classified diff:

POST /api/v2/schema/customers/_plan
{
  "type": "collection",
  "columns": {
    "id":      { "type": "Int" },
    "qty":     { "type": "Long" },
    "note":    { "type": "Text", "analyzer": "English" },
    "segment": { "type": "String", "nullable": true }
  }
}

Each column comes back with an action — add, rebuild, drop, rejected, or noop — plus a summary telling you whether the change is additive-only, lossy, rejected, or needs confirmation:

{
  "table": "customers",
  "summary": {
    "changes": 3, "rebuildColumns": 1, "rows": 3,
    "dataLoss": true, "rejected": false,
    "applicable": true, "requiresConfirmation": true, "additiveOnly": false
  },
  "plan": [
    { "column": "qty",     "action": "rebuild", "from": "Int", "to": "Long", "rows": 3 },
    { "column": "segment", "action": "add",     "type": "String", "backfill": "null" },
    { "column": "legacy",  "action": "drop",    "from": "String", "dataLoss": true }
  ]
}

Apply runs the plan in one atomic commit:

POST /api/v2/schema/customers/_apply

_apply refuses rejected (lossy) changes outright, and refuses column drops (data loss) unless you pass ?confirm=true. It is idempotent — applying the current schema as the target is a no-op. Plan/apply is collection (v2) only.

Update and delete rows

Change data — not schema — with POST /api/v2/data/_modify. It applies one operation, or { "operations": [ ... ] } for several in a single commit, on both TableDb and CollectionDb:

// update (add "upsert": true to insert when nothing matches)
{ "update": "users", "set": { "tier": "gold" }, "where": { "id": 1 } }

// delete
{ "from": "users", "delete": { "id": 1 } }

// insert
{ "into": "users", "insert": { "id": 9, "name": "dana" } }

To delete by a query rather than a key, use POST /api/v2/data/_delete with a where clause.

Bulk backfill a column

After adding a column, the fast way to populate it across many existing rows is POST /api/v2/data/{table}/_backfill. The body is columnar — a key column name plus index-aligned keys and columns value arrays — so it stays compact for large jobs (no per-row field-name repetition):

POST /api/v2/data/users/_backfill
{
  "key": "id",
  "keys":    [1, 2, 3],
  "columns": { "tier": ["gold", "silver", "bronze"] }
}

The whole batch is applied in one commit and returns { "total": N }. A key that matches no row is a no-op; add "upsert": true to insert a new row for unmatched keys. Multiple columns can be set at once — one array per column, all aligned to keys.

The same capability is available as a backfill operation inside _modify ({ "backfill": "users", "key": "id", "keys": [...], "columns": { ... } }), which works on both v1 (/api/v1/data/_modify) and v2.

Optimize

Writes land as segments; POST /api/v2/data/{table}/optimize merges them into a compact, read-optimized form. It's safe to call any time and is worth running after a large migration or backfill to reclaim space and keep reads fast.

Related

• Schema Design — modelling types, links, and the quantity flag.
• CollectionDb — the v2 engine: JSON import, schema inference & evolution.
• Environments — branch the database into a sandbox, migrate there, then promote.