APIs

Packages:

numaflow.numaproj.io/v1alpha1

Resource Types:

AbstractPodTemplate

(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, NativeRedis, SideInputsManagerTemplate, VertexTemplate)

AbstractPodTemplate provides a template for pod customization in vertices, daemon deployments and so on.

Field Description
metadata
Metadata 		(Optional)

Metadata sets the pods’s metadata, i.e. annotations and labels

nodeSelector
map\[string\]string 		(Optional)

NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/

tolerations
\[\]Kubernetes core/v1.Toleration 		(Optional)

If specified, the pod’s tolerations.

securityContext
Kubernetes core/v1.PodSecurityContext 		(Optional)

SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field.

imagePullSecrets
\[\]Kubernetes core/v1.LocalObjectReference 		(Optional)

ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod

priorityClassName
string 		(Optional)

If specified, indicates the Redis pod’s priority. “system-node-critical” and “system-cluster-critical” are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/

priority
int32 		(Optional)

The priority value. Various system components use this field to find the priority of the Redis pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/

affinity
Kubernetes core/v1.Affinity 		(Optional)

The pod’s scheduling constraints More info: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/

serviceAccountName
string 		(Optional)

ServiceAccountName applied to the pod

runtimeClassName
string 		(Optional)

RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the “legacy” RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class

automountServiceAccountToken
bool 		(Optional)

AutomountServiceAccountToken indicates whether a service account token should be automatically mounted.

dnsPolicy
Kubernetes core/v1.DNSPolicy 		(Optional)

Set DNS policy for the pod. Defaults to “ClusterFirst”. Valid values are ‘ClusterFirstWithHostNet’, ‘ClusterFirst’, ‘Default’ or ‘None’. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to ‘ClusterFirstWithHostNet’.

dnsConfig
Kubernetes core/v1.PodDNSConfig 		(Optional)

Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy.

AbstractSink

(Appears on: Sink)

Field Description
log
Log 		(Optional)

Log sink is used to write the data to the log.

kafka
KafkaSink 		(Optional)

Kafka sink is used to write the data to the Kafka.

blackhole
Blackhole 		(Optional)

Blackhole sink is used to write the data to the blackhole sink, which is a sink that discards all the data written to it.

udsink
UDSink 		(Optional)

UDSink sink is used to write the data to the user-defined sink.

AbstractVertex

(Appears on: PipelineSpec, VertexSpec)

Field Description
name
string
source
Source 		(Optional)
sink
Sink 		(Optional)
udf
UDF 		(Optional)
containerTemplate
ContainerTemplate 		(Optional)

Container template for the main numa container.

initContainerTemplate
ContainerTemplate 		(Optional)

Container template for all the vertex pod init containers spawned by numaflow, excluding the ones specified by the user.

AbstractPodTemplate
AbstractPodTemplate

(Members of AbstractPodTemplate are embedded into this type.)

(Optional)
volumes
\[\]Kubernetes core/v1.Volume 		(Optional)
limits
VertexLimits 		(Optional)

Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, will override pipeline level settings

scale
Scale 		(Optional)

Settings for autoscaling

initContainers
\[\]Kubernetes core/v1.Container 		(Optional)

List of customized init containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/

sidecars
\[\]Kubernetes core/v1.Container 		(Optional)

List of customized sidecar containers belonging to the pod.

partitions
int32 		(Optional)

Number of partitions of the vertex owned buffers. It applies to udf and sink vertices only.

sideInputs
\[\]string 		(Optional)

Names of the side inputs used in this vertex.

sideInputsContainerTemplate
ContainerTemplate 		(Optional)

Container template for the side inputs watcher container.

Authorization

(Appears on: HTTPSource)

Field Description
token
Kubernetes core/v1.SecretKeySelector 		(Optional)

A secret selector which contains bearer token To use this, the client needs to add “Authorization: Bearer ” in the header

BasicAuth

(Appears on: NatsAuth)

BasicAuth represents the basic authentication approach which contains a user name and a password.

Field Description
user
Kubernetes core/v1.SecretKeySelector 		(Optional)

Secret for auth user

password
Kubernetes core/v1.SecretKeySelector 		(Optional)

Secret for auth password

Blackhole

(Appears on: AbstractSink)

Blackhole is a sink to emulate /dev/null

BufferFullWritingStrategy (string alias)

(Appears on: Edge)

BufferServiceConfig

(Appears on: InterStepBufferServiceStatus)

Field Description
redis
RedisConfig
jetstream
JetStreamConfig

CombinedEdge

(Appears on: VertexSpec)

CombinedEdge is a combination of Edge and some other properties such as vertex type, partitions, limits. It’s used to decorate the fromEdges and toEdges of the generated Vertex objects, so that in the vertex pod, it knows the properties of the connected vertices, for example, how many partitioned buffers I should write to, what is the write buffer length, etc.

Field Description
Edge
Edge

(Members of Edge are embedded into this type.)

fromVertexType
VertexType

From vertex type.

fromVertexPartitionCount
int32 		(Optional)

The number of partitions of the from vertex, if not provided, the default value is set to “1”.

fromVertexLimits
VertexLimits 		(Optional)
toVertexType
VertexType

To vertex type.

toVertexPartitionCount
int32 		(Optional)

The number of partitions of the to vertex, if not provided, the default value is set to “1”.

toVertexLimits
VertexLimits 		(Optional)

ConditionType (string alias)

ConditionType is a valid value of Condition.Type

Container

(Appears on: SideInput, UDF, UDSink, UDSource, UDTransformer)

Container is used to define the container properties for user-defined functions, sinks, etc.

Field Description
image
string 		(Optional)
command
\[\]string 		(Optional)
args
\[\]string 		(Optional)
env
\[\]Kubernetes core/v1.EnvVar 		(Optional)
envFrom
\[\]Kubernetes core/v1.EnvFromSource 		(Optional)
volumeMounts
\[\]Kubernetes core/v1.VolumeMount 		(Optional)
resources
Kubernetes core/v1.ResourceRequirements 		(Optional)
securityContext
Kubernetes core/v1.SecurityContext 		(Optional)
imagePullPolicy
Kubernetes core/v1.PullPolicy 		(Optional)

ContainerTemplate

(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, NativeRedis, SideInputsManagerTemplate, VertexTemplate)

ContainerTemplate defines customized spec for a container

Field Description
resources
Kubernetes core/v1.ResourceRequirements 		(Optional)
imagePullPolicy
Kubernetes core/v1.PullPolicy 		(Optional)
securityContext
Kubernetes core/v1.SecurityContext 		(Optional)
env
\[\]Kubernetes core/v1.EnvVar 		(Optional)
envFrom
\[\]Kubernetes core/v1.EnvFromSource 		(Optional)

DaemonTemplate

(Appears on: Templates)

Field Description
AbstractPodTemplate
AbstractPodTemplate

(Members of AbstractPodTemplate are embedded into this type.)

(Optional)
replicas
int32 		(Optional)

Replicas is the number of desired replicas of the Deployment. This is a pointer to distinguish between explicit zero and unspecified. Defaults to 1. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#what-is-a-replicationcontroller

containerTemplate
ContainerTemplate 		(Optional)
initContainerTemplate
ContainerTemplate 		(Optional)

Edge

(Appears on: CombinedEdge, PipelineSpec)

Field Description
from
string
to
string
conditions
ForwardConditions 		(Optional)

Conditional forwarding, only allowed when “From” is a Sink or UDF.

onFull
BufferFullWritingStrategy 		(Optional)

OnFull specifies the behaviour for the write actions when the inter step buffer is full. There are currently two options, retryUntilSuccess and discardLatest. if not provided, the default value is set to “retryUntilSuccess”

FixedWindow

(Appears on: Window)

FixedWindow describes a fixed window

Field Description
length
Kubernetes meta/v1.Duration

Length is the duration of the fixed window.

streaming
bool 		(Optional)

Streaming should be set to true if the reduce udf is streaming.

ForwardConditions

(Appears on: Edge)

Field Description
tags
TagConditions

Tags used to specify tags for conditional forwarding

Function

(Appears on: UDF)

Field Description
name
string
args
\[\]string 		(Optional)
kwargs
map\[string\]string 		(Optional)

GSSAPI

(Appears on: SASL)

GSSAPI represents a SASL GSSAPI config

Field Description
serviceName
string
realm
string
usernameSecret
Kubernetes core/v1.SecretKeySelector

UsernameSecret refers to the secret that contains the username

authType
KRB5AuthType

valid inputs - KRB5_USER_AUTH, KRB5_KEYTAB_AUTH

passwordSecret
Kubernetes core/v1.SecretKeySelector 		(Optional)

PasswordSecret refers to the secret that contains the password

keytabSecret
Kubernetes core/v1.SecretKeySelector 		(Optional)

KeytabSecret refers to the secret that contains the keytab

kerberosConfigSecret
Kubernetes core/v1.SecretKeySelector 		(Optional)

KerberosConfigSecret refers to the secret that contains the kerberos config

GeneratorSource

(Appears on: Source)

Field Description
rpu
int64 		(Optional)
duration
Kubernetes meta/v1.Duration 		(Optional)
msgSize
int32 		(Optional)

Size of each generated message

keyCount
int32 		(Optional)

KeyCount is the number of unique keys in the payload

value
uint64 		(Optional)

Value is an optional uint64 value to be written in to the payload

jitter
Kubernetes meta/v1.Duration 		(Optional)

Jitter is the jitter for the message generation, used to simulate out of order messages for example if the jitter is 10s, then the message’s event time will be delayed by a random time between 0 and 10s which will result in the message being out of order by 0 to 10s

GetDaemonDeploymentReq

Field Description
ISBSvcType
ISBSvcType
Image
string
PullPolicy
Kubernetes core/v1.PullPolicy
Env
\[\]Kubernetes core/v1.EnvVar
DefaultResources
Kubernetes core/v1.ResourceRequirements

GetJetStreamServiceSpecReq

Field Description
Labels
map\[string\]string
ClusterPort
int32
ClientPort
int32
MonitorPort
int32
MetricsPort
int32

GetJetStreamStatefulSetSpecReq

Field Description
ServiceName
string
Labels
map\[string\]string
NatsImage
string
MetricsExporterImage
string
ConfigReloaderImage
string
ClusterPort
int32
ClientPort
int32
MonitorPort
int32
MetricsPort
int32
ServerAuthSecretName
string
ServerEncryptionSecretName
string
ConfigMapName
string
PvcNameIfNeeded
string
StartCommand
string
DefaultResources
Kubernetes core/v1.ResourceRequirements

GetRedisServiceSpecReq

Field Description
Labels
map\[string\]string
RedisContainerPort
int32
SentinelContainerPort
int32

GetRedisStatefulSetSpecReq

Field Description
ServiceName
string
Labels
map\[string\]string
RedisImage
string
SentinelImage
string
MetricsExporterImage
string
InitContainerImage
string
RedisContainerPort
int32
SentinelContainerPort
int32
RedisMetricsContainerPort
int32
CredentialSecretName
string
TLSEnabled
bool
PvcNameIfNeeded
string
ConfConfigMapName
string
ScriptsConfigMapName
string
HealthConfigMapName
string
DefaultResources
Kubernetes core/v1.ResourceRequirements

GetSideInputDeploymentReq

Field Description
ISBSvcType
ISBSvcType
Image
string
PullPolicy
Kubernetes core/v1.PullPolicy
Env
\[\]Kubernetes core/v1.EnvVar
DefaultResources
Kubernetes core/v1.ResourceRequirements

GetVertexPodSpecReq

Field Description
ISBSvcType
ISBSvcType
Image
string
PullPolicy
Kubernetes core/v1.PullPolicy
Env
\[\]Kubernetes core/v1.EnvVar
SideInputsStoreName
string
DefaultResources
Kubernetes core/v1.ResourceRequirements

GroupBy

(Appears on: UDF)

GroupBy indicates it is a reducer UDF

Field Description
window
Window

Window describes the windowing strategy.

keyed
bool 		(Optional)
allowedLateness
Kubernetes meta/v1.Duration 		(Optional)

AllowedLateness allows late data to be included for the Reduce operation as long as the late data is not later than (Watermark - AllowedLateness).

storage
PBQStorage

Storage is used to define the PBQ storage for a reduce vertex.

HTTPSource

(Appears on: Source)

Field Description
auth
Authorization 		(Optional)
service
bool 		(Optional)

Whether to create a ClusterIP Service

ISBSvcPhase (string alias)

(Appears on: InterStepBufferServiceStatus)

ISBSvcType (string alias)

(Appears on: GetDaemonDeploymentReq, GetSideInputDeploymentReq, GetVertexPodSpecReq, InterStepBufferServiceStatus)

IdleSource

(Appears on: Watermark)

Field Description
threshold
Kubernetes meta/v1.Duration

Threshold is the duration after which a source is marked as Idle due to lack of data. Ex: If watermark found to be idle after the Threshold duration then the watermark is progressed by IncrementBy.

stepInterval
Kubernetes meta/v1.Duration 		(Optional)

StepInterval is the duration between the subsequent increment of the watermark as long the source remains Idle. The default value is 0s which means that once we detect idle source, we will be incrementing the watermark by IncrementBy for time we detect that we source is empty (in other words, this will be a very frequent update).

incrementBy
Kubernetes meta/v1.Duration

IncrementBy is the duration to be added to the current watermark to progress the watermark when source is idling.

InterStepBufferService

Field Description
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
InterStepBufferServiceSpec

redis
RedisBufferService
jetstream
JetStreamBufferService
status
InterStepBufferServiceStatus 		(Optional)

InterStepBufferServiceSpec

(Appears on: InterStepBufferService)

Field Description
redis
RedisBufferService
jetstream
JetStreamBufferService

InterStepBufferServiceStatus

(Appears on: InterStepBufferService)

Field Description
Status
Status

(Members of Status are embedded into this type.)

phase
ISBSvcPhase
message
string
config
BufferServiceConfig
type
ISBSvcType

JetStreamBufferService

(Appears on: InterStepBufferServiceSpec)

Field Description
version
string

JetStream version, such as “2.7.1”

replicas
int32

Redis StatefulSet size

containerTemplate
ContainerTemplate 		(Optional)

ContainerTemplate contains customized spec for NATS container

reloaderContainerTemplate
ContainerTemplate 		(Optional)

ReloaderContainerTemplate contains customized spec for config reloader container

metricsContainerTemplate
ContainerTemplate 		(Optional)

MetricsContainerTemplate contains customized spec for metrics container

persistence
PersistenceStrategy 		(Optional)
AbstractPodTemplate
AbstractPodTemplate

(Members of AbstractPodTemplate are embedded into this type.)

(Optional)
settings
string 		(Optional)

Nats/JetStream configuration, if not specified, global settings in numaflow-controller-config will be used. See https://docs.nats.io/running-a-nats-service/configuration#limits and https://docs.nats.io/running-a-nats-service/configuration#jetstream. For limits, only “max_payload” is supported for configuration, defaults to 1048576 (1MB), not recommended to use values over 8388608 (8MB) but max_payload can be set up to 67108864 (64MB). For jetstream, only “max_memory_store” and “max_file_store” are supported for configuration, do not set “store_dir” as it has been hardcoded.

startArgs
\[\]string 		(Optional)

Optional arguments to start nats-server. For example, “-D” to enable debugging output, “-DV” to enable debugging and tracing. Check https://docs.nats.io/ for all the available arguments.

bufferConfig
string 		(Optional)

Optional configuration for the streams, consumers and buckets to be created in this JetStream service, if specified, it will be merged with the default configuration in numaflow-controller-config. It accepts a YAML format configuration, it may include 4 sections, “stream”, “consumer”, “otBucket” and “procBucket”. Available fields under “stream” include “retention” (e.g. interest, limits, workerQueue), “maxMsgs”, “maxAge” (e.g. 72h), “replicas” (1, 3, 5), “duplicates” (e.g. 5m). Available fields under “consumer” include “ackWait” (e.g. 60s) Available fields under “otBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5). Available fields under “procBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5).

encryption
bool 		(Optional)

Whether encrypt the data at rest, defaults to false Enabling encryption might impact the performance, see https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/encryption_at_rest for the detail Toggling the value will impact encrypting/decrypting existing messages.

tls
bool 		(Optional)

Whether enable TLS, defaults to false Enabling TLS might impact the performance

JetStreamConfig

(Appears on: BufferServiceConfig)

Field Description
url
string

JetStream (NATS) URL

auth
NatsAuth
streamConfig
string 		(Optional)
tlsEnabled
bool

TLS enabled or not

JobTemplate

(Appears on: Templates)

Field Description
AbstractPodTemplate
AbstractPodTemplate

(Members of AbstractPodTemplate are embedded into this type.)

(Optional)
containerTemplate
ContainerTemplate 		(Optional)
ttlSecondsAfterFinished
int32 		(Optional)

ttlSecondsAfterFinished limits the lifetime of a Job that has finished execution (either Complete or Failed). If this field is set, ttlSecondsAfterFinished after the Job finishes, it is eligible to be automatically deleted. When the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the Job won’t be automatically deleted. If this field is set to zero, the Job becomes eligible to be deleted immediately after it finishes. Numaflow defaults to 30

backoffLimit
int32 		(Optional)

Specifies the number of retries before marking this job failed. More info: https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy Numaflow defaults to 20

KRB5AuthType (string alias)

(Appears on: GSSAPI)

KRB5AuthType describes the kerberos auth type

KafkaSink

(Appears on: AbstractSink)

Field Description
brokers
\[\]string
topic
string
tls
TLS 		(Optional)

TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS.

config
string 		(Optional)
sasl
SASL 		(Optional)

SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL.

KafkaSource

(Appears on: Source)

Field Description
brokers
\[\]string
topic
string
consumerGroup
string
tls
TLS 		(Optional)

TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS.

config
string 		(Optional)
sasl
SASL 		(Optional)

SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL.

Lifecycle

(Appears on: PipelineSpec)

Field Description
deleteGracePeriodSeconds
int32 		(Optional)

DeleteGracePeriodSeconds used to delete pipeline gracefully

desiredPhase
PipelinePhase 		(Optional)

DesiredPhase used to bring the pipeline from current phase to desired phase

pauseGracePeriodSeconds
int32 		(Optional)

PauseGracePeriodSeconds used to pause pipeline gracefully

Log

(Appears on: AbstractSink)

LogicOperator (string alias)

(Appears on: TagConditions)

Metadata

(Appears on: AbstractPodTemplate)

Field Description
annotations
map\[string\]string
labels
map\[string\]string

NativeRedis

(Appears on: RedisBufferService)

Field Description
version
string

Redis version, such as “6.0.16”

replicas
int32

Redis StatefulSet size

redisContainerTemplate
ContainerTemplate 		(Optional)

RedisContainerTemplate contains customized spec for Redis container

sentinelContainerTemplate
ContainerTemplate 		(Optional)

SentinelContainerTemplate contains customized spec for Redis container

metricsContainerTemplate
ContainerTemplate 		(Optional)

MetricsContainerTemplate contains customized spec for metrics container

initContainerTemplate
ContainerTemplate 		(Optional)
persistence
PersistenceStrategy 		(Optional)
AbstractPodTemplate
AbstractPodTemplate

(Members of AbstractPodTemplate are embedded into this type.)

(Optional)
settings
RedisSettings 		(Optional)

Redis configuration, if not specified, global settings in numaflow-controller-config will be used.

NatsAuth

(Appears on: JetStreamConfig, NatsSource)

NatsAuth defines how to authenticate the nats access

Field Description
basic
BasicAuth 		(Optional)

Basic auth which contains a user name and a password

token
Kubernetes core/v1.SecretKeySelector 		(Optional)

Token auth

nkey
Kubernetes core/v1.SecretKeySelector 		(Optional)

NKey auth

NatsSource

(Appears on: Source)

Field Description
url
string

URL to connect to NATS cluster, multiple urls could be separated by comma.

subject
string

Subject holds the name of the subject onto which messages are published.

queue
string

Queue is used for queue subscription.

tls
TLS 		(Optional)

TLS configuration for the nats client.

auth
NatsAuth 		(Optional)

Auth information

NoStore

(Appears on: PBQStorage)

NoStore means there will be no persistence storage and there will be data loss during pod restarts. Use this option only if you do not care about correctness (e.g., approx statistics pipeline like sampling rate, etc.).

PBQStorage

(Appears on: GroupBy)

PBQStorage defines the persistence configuration for a vertex.

Field Description
persistentVolumeClaim
PersistenceStrategy 		(Optional)
emptyDir
Kubernetes core/v1.EmptyDirVolumeSource 		(Optional)
no_store
NoStore 		(Optional)

PersistenceStrategy

(Appears on: JetStreamBufferService, NativeRedis, PBQStorage)

PersistenceStrategy defines the strategy of persistence

Field Description
storageClassName
string 		(Optional)

Name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1

accessMode
Kubernetes core/v1.PersistentVolumeAccessMode 		(Optional)

Available access modes such as ReadWriteOnce, ReadWriteMany https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes

volumeSize
k8s.io/apimachinery/pkg/api/resource.Quantity

Volume size, e.g. 50Gi

Pipeline

Field Description
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
PipelineSpec

interStepBufferServiceName
string 		(Optional)
vertices
\[\]AbstractVertex
edges
\[\]Edge

Edges define the relationships between vertices

lifecycle
Lifecycle 		(Optional)

Lifecycle define the Lifecycle properties

limits
PipelineLimits 		(Optional)

Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, they could be overridden by each vertex’s settings

watermark
Watermark 		(Optional)

Watermark enables watermark progression across the entire pipeline.

templates
Templates 		(Optional)

Templates are used to customize additional kubernetes resources required for the Pipeline

sideInputs
\[\]SideInput 		(Optional)

SideInputs defines the Side Inputs of a pipeline.

status
PipelineStatus 		(Optional)

PipelineLimits

(Appears on: PipelineSpec)

Field Description
readBatchSize
uint64 		(Optional)

Read batch size for all the vertices in the pipeline, can be overridden by the vertex’s limit settings.

bufferMaxLength
uint64 		(Optional)

BufferMaxLength is used to define the max length of a buffer. Only applies to UDF and Source vertices as only they do buffer write. It can be overridden by the settings in vertex limits.

bufferUsageLimit
uint32 		(Optional)

BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. Only applies to UDF and Source vertices as only they do buffer write. It will be overridden by the settings in vertex limits.

readTimeout
Kubernetes meta/v1.Duration 		(Optional)

Read timeout for all the vertices in the pipeline, can be overridden by the vertex’s limit settings

PipelinePhase (string alias)

(Appears on: Lifecycle, PipelineStatus)

PipelineSpec

(Appears on: Pipeline)

Field Description
interStepBufferServiceName
string 		(Optional)
vertices
\[\]AbstractVertex
edges
\[\]Edge

Edges define the relationships between vertices

lifecycle
Lifecycle 		(Optional)

Lifecycle define the Lifecycle properties

limits
PipelineLimits 		(Optional)

Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, they could be overridden by each vertex’s settings

watermark
Watermark 		(Optional)

Watermark enables watermark progression across the entire pipeline.

templates
Templates 		(Optional)

Templates are used to customize additional kubernetes resources required for the Pipeline

sideInputs
\[\]SideInput 		(Optional)

SideInputs defines the Side Inputs of a pipeline.

PipelineStatus

(Appears on: Pipeline)

Field Description
Status
Status

(Members of Status are embedded into this type.)

phase
PipelinePhase
message
string
lastUpdated
Kubernetes meta/v1.Time
vertexCount
uint32
sourceCount
uint32
sinkCount
uint32
udfCount
uint32

RedisBufferService

(Appears on: InterStepBufferServiceSpec)

Field Description
native
NativeRedis

Native brings up a native Redis service

external
RedisConfig

External holds an External Redis config

RedisConfig

(Appears on: BufferServiceConfig, RedisBufferService)

Field Description
url
string 		(Optional)

Redis URL

sentinelUrl
string 		(Optional)

Sentinel URL, will be ignored if Redis URL is provided

masterName
string 		(Optional)

Only required when Sentinel is used

user
string 		(Optional)

Redis user

password
Kubernetes core/v1.SecretKeySelector 		(Optional)

Redis password secret selector

sentinelPassword
Kubernetes core/v1.SecretKeySelector 		(Optional)

Sentinel password secret selector

RedisSettings

(Appears on: NativeRedis)

Field Description
redis
string 		(Optional)

Redis settings shared by both master and slaves, will override the global settings from controller config

master
string 		(Optional)

Special settings for Redis master node, will override the global settings from controller config

replica
string 		(Optional)

Special settings for Redis replica nodes, will override the global settings from controller config

sentinel
string 		(Optional)

Sentinel settings, will override the global settings from controller config

SASL

(Appears on: KafkaSink, KafkaSource)

Field Description
mechanism
SASLType

SASL mechanism to use

gssapi
GSSAPI 		(Optional)

GSSAPI contains the kerberos config

plain
SASLPlain 		(Optional)

SASLPlain contains the sasl plain config

scramsha256
SASLPlain 		(Optional)

SASLSCRAMSHA256 contains the sasl plain config

scramsha512
SASLPlain 		(Optional)

SASLSCRAMSHA512 contains the sasl plain config

SASLPlain

(Appears on: SASL)

Field Description
userSecret
Kubernetes core/v1.SecretKeySelector

UserSecret refers to the secret that contains the user

passwordSecret
Kubernetes core/v1.SecretKeySelector 		(Optional)

PasswordSecret refers to the secret that contains the password

handshake
bool

SASLType (string alias)

(Appears on: SASL)

SASLType describes the SASL type

Scale

(Appears on: AbstractVertex)

Scale defines the parameters for autoscaling.

Field Description
disabled
bool 		(Optional)

Whether to disable autoscaling. Set to “true” when using Kubernetes HPA or any other 3rd party autoscaling strategies.

min
int32 		(Optional)

Minimum replicas.

max
int32 		(Optional)

Maximum replicas.

lookbackSeconds
uint32 		(Optional)

Lookback seconds to calculate the average pending messages and processing rate.

cooldownSeconds
uint32 		(Optional)

Deprecated: Use scaleUpCooldownSeconds and scaleDownCooldownSeconds instead. Cooldown seconds after a scaling operation before another one.

zeroReplicaSleepSeconds
uint32 		(Optional)

After scaling down the source vertex to 0, sleep how many seconds before scaling the source vertex back up to peek.

targetProcessingSeconds
uint32 		(Optional)

TargetProcessingSeconds is used to tune the aggressiveness of autoscaling for source vertices, it measures how fast you want the vertex to process all the pending messages. Typically increasing the value, which leads to lower processing rate, thus less replicas. It’s only effective for source vertices.

targetBufferAvailability
uint32 		(Optional)

TargetBufferAvailability is used to define the target percentage of the buffer availability. A valid and meaningful value should be less than the BufferUsageLimit defined in the Edge spec (or Pipeline spec), for example, 50. It only applies to UDF and Sink vertices because only they have buffers to read.

replicasPerScale
uint32 		(Optional)

ReplicasPerScale defines maximum replicas can be scaled up or down at once. The is use to prevent too aggressive scaling operations

scaleUpCooldownSeconds
uint32 		(Optional)

ScaleUpCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling up. It defaults to the CooldownSeconds if not set.

scaleDownCooldownSeconds
uint32 		(Optional)

ScaleDownCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling down. It defaults to the CooldownSeconds if not set.

SessionWindow

(Appears on: Window)

SessionWindow describes a session window

Field Description
timeout
Kubernetes meta/v1.Duration

Timeout is the duration of inactivity after which a session window closes.

SideInput

(Appears on: PipelineSpec)

SideInput defines information of a Side Input

Field Description
name
string
container
Container
volumes
\[\]Kubernetes core/v1.Volume 		(Optional)
trigger
SideInputTrigger

SideInputTrigger

(Appears on: SideInput)

Field Description
schedule
string

The schedule to trigger the retrievement of the side input data. It supports cron format, for example, “0 30 \* \* \* \*”. Or interval based format, such as “@hourly”, “@every 1h30m”, etc.

timezone
string 		(Optional)

SideInputsManagerTemplate

(Appears on: Templates)

Field Description
AbstractPodTemplate
AbstractPodTemplate

(Members of AbstractPodTemplate are embedded into this type.)

(Optional)
containerTemplate
ContainerTemplate 		(Optional)

Template for the side inputs manager numa container

initContainerTemplate
ContainerTemplate 		(Optional)

Template for the side inputs manager init container

Sink

(Appears on: AbstractVertex)

Field Description
AbstractSink
AbstractSink

(Members of AbstractSink are embedded into this type.)

fallback
AbstractSink 		(Optional)

Fallback sink can be imagined as DLQ for primary Sink. The writes to Fallback sink will only be initiated if the ud-sink response field sets it.

SlidingWindow

(Appears on: Window)

SlidingWindow describes a sliding window

Field Description
length
Kubernetes meta/v1.Duration

Length is the duration of the sliding window.

slide
Kubernetes meta/v1.Duration

Slide is the slide parameter that controls the frequency at which the sliding window is created.

streaming
bool 		(Optional)

Streaming should be set to true if the reduce udf is streaming.

Source

(Appears on: AbstractVertex)

Field Description
generator
GeneratorSource 		(Optional)
kafka
KafkaSource 		(Optional)
http
HTTPSource 		(Optional)
nats
NatsSource 		(Optional)
transformer
UDTransformer 		(Optional)
udsource
UDSource 		(Optional)

Status

(Appears on: InterStepBufferServiceStatus, PipelineStatus)

Status is a common structure which can be used for Status field.

Field Description
conditions
\[\]Kubernetes meta/v1.Condition 		(Optional)

Conditions are the latest available observations of a resource’s current state.

TLS

(Appears on: KafkaSink, KafkaSource, NatsSource)

Field Description
insecureSkipVerify
bool 		(Optional)
caCertSecret
Kubernetes core/v1.SecretKeySelector 		(Optional)

CACertSecret refers to the secret that contains the CA cert

certSecret
Kubernetes core/v1.SecretKeySelector 		(Optional)

CertSecret refers to the secret that contains the cert

keySecret
Kubernetes core/v1.SecretKeySelector 		(Optional)

KeySecret refers to the secret that contains the key

TagConditions

(Appears on: ForwardConditions)

Field Description
operator
LogicOperator 		(Optional)

Operator specifies the type of operation that should be used for conditional forwarding value could be “and”, “or”, “not”

values
\[\]string

Values tag values for conditional forwarding

Templates

(Appears on: PipelineSpec)

Field Description
daemon
DaemonTemplate 		(Optional)

DaemonTemplate is used to customize the Daemon Deployment.

job
JobTemplate 		(Optional)

JobTemplate is used to customize Jobs.

sideInputsManager
SideInputsManagerTemplate 		(Optional)

SideInputsManagerTemplate is used to customize the Side Inputs Manager.

vertex
VertexTemplate 		(Optional)

VertexTemplate is used to customize the vertices of the pipeline.

Transformer

(Appears on: UDTransformer)

Field Description
name
string
args
\[\]string 		(Optional)
kwargs
map\[string\]string 		(Optional)

UDF

(Appears on: AbstractVertex)

Field Description
container
Container 		(Optional)
builtin
Function 		(Optional)
groupBy
GroupBy 		(Optional)

UDSink

(Appears on: AbstractSink)

Field Description
container
Container

UDSource

(Appears on: Source)

Field Description
container
Container

UDTransformer

(Appears on: Source)

Field Description
container
Container 		(Optional)
builtin
Transformer 		(Optional)

Vertex

(Appears on: VertexInstance)

Field Description
metadata
Kubernetes meta/v1.ObjectMeta 		Refer to the Kubernetes API documentation for the fields of the metadata field.
spec
VertexSpec

AbstractVertex
AbstractVertex

(Members of AbstractVertex are embedded into this type.)

pipelineName
string
interStepBufferServiceName
string 		(Optional)
replicas
int32 		(Optional)
fromEdges
\[\]CombinedEdge 		(Optional)
toEdges
\[\]CombinedEdge 		(Optional)
watermark
Watermark 		(Optional)

Watermark indicates watermark progression in the vertex, it’s populated from the pipeline watermark settings.

status
VertexStatus 		(Optional)

VertexInstance

VertexInstance is a wrapper of a vertex instance, which contains the vertex spec and the instance information such as hostname and replica index.

Field Description
vertex
Vertex
hostname
string
replica
int32

VertexLimits

(Appears on: AbstractVertex, CombinedEdge)

Field Description
readBatchSize
uint64 		(Optional)

Read batch size from the source or buffer. It overrides the settings from pipeline limits.

readTimeout
Kubernetes meta/v1.Duration 		(Optional)

Read timeout duration from the source or buffer It overrides the settings from pipeline limits.

bufferMaxLength
uint64 		(Optional)

BufferMaxLength is used to define the max length of a buffer. It overrides the settings from pipeline limits.

bufferUsageLimit
uint32 		(Optional)

BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. It overrides the settings from pipeline limits.

VertexPhase (string alias)

(Appears on: VertexStatus)

VertexSpec

(Appears on: Vertex)

Field Description
AbstractVertex
AbstractVertex

(Members of AbstractVertex are embedded into this type.)

pipelineName
string
interStepBufferServiceName
string 		(Optional)
replicas
int32 		(Optional)
fromEdges
\[\]CombinedEdge 		(Optional)
toEdges
\[\]CombinedEdge 		(Optional)
watermark
Watermark 		(Optional)

Watermark indicates watermark progression in the vertex, it’s populated from the pipeline watermark settings.

VertexStatus

(Appears on: Vertex)

Field Description
phase
VertexPhase
reason
string
message
string
replicas
uint32
selector
string
lastScaledAt
Kubernetes meta/v1.Time

VertexTemplate

(Appears on: Templates)

Field Description
AbstractPodTemplate
AbstractPodTemplate

(Members of AbstractPodTemplate are embedded into this type.)

(Optional)
containerTemplate
ContainerTemplate 		(Optional)

Template for the vertex numa container

initContainerTemplate
ContainerTemplate 		(Optional)

Template for the vertex init container

VertexType (string alias)

(Appears on: CombinedEdge)

Watermark

(Appears on: PipelineSpec, VertexSpec)

Field Description
disabled
bool 		(Optional)

Disabled toggles the watermark propagation, defaults to false.

maxDelay
Kubernetes meta/v1.Duration 		(Optional)

Maximum delay allowed for watermark calculation, defaults to “0s”, which means no delay.

idleSource
IdleSource 		(Optional)

IdleSource defines the idle watermark properties, it could be configured in case source is idling.

Window

(Appears on: GroupBy)

Window describes windowing strategy

Field Description
fixed
FixedWindow 		(Optional)
sliding
SlidingWindow 		(Optional)
session
SessionWindow 		(Optional)

Generated with gen-crd-api-reference-docs.