Data Engineering

Incremental Load SQL Generator

Writes a watermark-based incremental extract query from a CREATE TABLE statement.

Loading the tool…

Processing happens locally in your browser. What you paste or load is processed by this page and is not uploaded to a server. Nothing is stored unless you use a control that says it stores something, and you can clear anything this site has kept from the privacy page.

How to use this tool

  1. Paste the CREATE TABLE statement for the source table.
  2. Name the watermark column if the automatic guess picks the wrong one, and name the target table.
  3. Select Generate query, and review it before running it — nothing here has seen your database.
  4. Decide separately how deletes will be handled. A watermark query cannot see them.

What incremental load sql generator does

Reloading a whole table every night works until it does not, and the switch to incremental loading is nearly always the same query: find the highest timestamp already loaded, and pull everything above it. Simple enough to write from memory, and simple enough to get subtly wrong at three in the morning.

This writes it from the DDL, picking the watermark column automatically when a column looks like one and refusing rather than guessing when none does. It is worth being clear about what this approach cannot do: a watermark extract never sees a deleted row, and it misses updates that do not touch the watermark column. If either matters, you need change data capture rather than a query.

Frequently asked questions

Nothing sees them. A watermark query asks for rows changed since the last load, and a row that no longer exists cannot be returned. If deletes matter you need log-based change data capture, a soft-delete flag, or a periodic full reconciliation.

It looks for a column whose name contains updated, modified, changed or timestamp, or ends in _at. If none matches it refuses rather than picking something arbitrary, because an incremental load against the wrong column silently skips data.

Not on its own. A transaction that commits after your query reads the maximum but with an earlier timestamp will be missed permanently. The usual mitigation is a small overlap window — read from slightly before the watermark and rely on the target being idempotent.

The dialect option controls identifier quoting. The query body is standard enough to run on most platforms, but review it before use — nothing here has seen your database, your indexes or your data volumes.