Data Engineering

Table Partition Size Calculator

Compares every partition grain by size per partition and partition count at full retention.

Loading the tool…

Processing happens locally in your browser. What you paste or load is processed by this page and is not uploaded to a server. Nothing is stored unless you use a control that says it stores something, and you can clear anything this site has kept from the privacy page.

How to use this tool

  1. Set rows per day, average row size and the retention window.
  2. Pick the grain you were planning to use.
  3. Select Compare grains and read the summary — it says which failure mode, if either, your choice falls into.
  4. Compare the other rows before settling. The grain one step coarser is often materially better.

What partition size calculator does

Partitioning has two failure modes and the grain alone tells you nothing about which one you are in. Too fine and you get tens of thousands of small files, where listing the partitions costs more than the scan you avoided. Too coarse and every query still reads a year of data, which is the situation partitioning was supposed to fix.

This works out rows and bytes per partition at every grain from hourly to yearly, alongside how many partitions exist once the retention window is full, and says which of the two failure modes your chosen grain falls into. The thresholds are the conventional ones for columnar lake formats — a row store with a good index tolerates smaller partitions than a Parquet table does.

Frequently asked questions

For columnar lake formats, somewhere between about 128 MB and a few gigabytes is the usual advice: large enough that the file and metadata overhead is not the dominant cost, small enough that pruning to one partition actually saves you something. Those are the thresholds this uses.

Less strictly. A row store with a good index tolerates much smaller partitions than a Parquet table on object storage, because it is not paying per-file overhead in the same way. Treat the guidance as calibrated for lake formats and relax it for a database.

Because the catalogue has to list them. Query planning against tens of thousands of partitions gets slow independently of how much data each one holds, which is how an hourly partitioning scheme on a small table ends up slower than no partitioning at all.

Often, yes — but only if your queries filter on it. Partitioning on a column nobody filters by costs you the small-files problem and buys nothing. This tool assumes a time grain because that is the common case; the arithmetic for any other key is the same once you know the rows per partition.