Data Engineering

Pipeline Throughput Calculator

Daily volume in, sustained rate, peak rate, bandwidth and worker count out.

Loading the tool…

Processing happens locally in your browser. What you paste or load is processed by this page and is not uploaded to a server. Nothing is stored unless you use a control that says it stores something, and you can clear anything this site has kept from the privacy page.

How to use this tool

  1. Set the daily row volume. Suffixes work, so 12M is twelve million.
  2. Add the processing window if the job does not have all day, and an average row size if you care about bandwidth.
  3. Set the peak multiplier to whatever your busiest hour looks like against the average — 2 to 3 is typical for a batch source.
  4. Select Calculate throughput, and size the fleet against the peak row rather than the average.

What throughput calculator does

A volume stated per day tells you almost nothing about whether a pipeline will cope, because nothing arrives evenly. Twelve million rows a day is eight hundred and thirty a second if they trickle in and thirty thousand a second if they arrive in the hour after the source system closes its books, and only one of those two numbers is worth designing against.

This converts the daily figure into the sustained rate, applies a peak multiplier and reports both, along with the bandwidth each implies once an average row size is known. Where a per-worker rate is given it also sizes the fleet twice — once for the average and once for the peak with parallel efficiency applied — because the gap between those two numbers is the capacity argument you will end up having.

Frequently asked questions

If you have the arrival data, use the busiest hour divided by the average hour and stop guessing. If you do not, 2 to 3 covers most batch sources and 5 or more is realistic where the source is a business process with a deadline, such as end-of-day trading or month-end close.

Two things compound: the peak multiplier itself, and parallel efficiency, which means each additional worker adds less than a full worker of capacity. If the two numbers are more than about three times apart, the summary flags it — that usually points at either a buffer between the source and the processing, or a source you can smooth.

No. It reports the rate and bandwidth you need; whether the network, the sink or a rate limit can supply it is a separate question this cannot answer. The bandwidth figures are there to be compared against those limits by hand.

Row counts accept k, M and bn suffixes. Sizes accept bytes, KB, MB, GB and TB as powers of 1000 and KiB, MiB, GiB and TiB as powers of 1024. Durations accept seconds, minutes, hours, days, weeks, months and years.