Skip to content

Inter-Step Buffer

A Pipeline contains multiple vertices that ingest data from sources, process data, and forward processed data to sinks. Vertices are not connected directly, but through Inter-Step Buffers.

Inter-Step Buffer can be implemented by a variety of data buffering technologies. Those technologies should support:

  • Durability
  • Offsets
  • Transactions for Exactly-Once forwarding
  • Concurrent reading
  • Ability to explicitly acknowledge each data or offset
  • Claim pending messages (read but not acknowledge)
  • Ability to trim data (buffer size control)
  • Fast (high throughput low latency)
  • Ability to query buffer information

Currently, there are 2 Inter-Step Buffer implementations: