APIs
Packages:
numaflow.numaproj.io/v1alpha1
Resource Types:
AbstractPodTemplate
(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, NativeRedis, SideInputsManagerTemplate, VertexTemplate)
AbstractPodTemplate provides a template for pod customization in vertices, daemon deployments and so on.
Field | Description |
---|---|
metadata
Metadata
|
(Optional)
Metadata sets the pods’s metadata, i.e. annotations and labels |
nodeSelector map\[string\]string
|
(Optional)
NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ |
tolerations
\[\]Kubernetes core/v1.Toleration
|
(Optional)
If specified, the pod’s tolerations. |
securityContext
Kubernetes core/v1.PodSecurityContext
|
(Optional)
SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field. |
imagePullSecrets
\[\]Kubernetes core/v1.LocalObjectReference
|
(Optional)
ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod |
priorityClassName string
|
(Optional)
If specified, indicates the Redis pod’s priority. “system-node-critical” and “system-cluster-critical” are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ |
priority int32
|
(Optional)
The priority value. Various system components use this field to find the priority of the Redis pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ |
affinity
Kubernetes core/v1.Affinity
|
(Optional)
The pod’s scheduling constraints More info: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ |
serviceAccountName string
|
(Optional)
ServiceAccountName applied to the pod |
runtimeClassName string
|
(Optional)
RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the “legacy” RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class |
automountServiceAccountToken bool
|
(Optional)
AutomountServiceAccountToken indicates whether a service account token should be automatically mounted. |
dnsPolicy
Kubernetes core/v1.DNSPolicy
|
(Optional)
Set DNS policy for the pod. Defaults to “ClusterFirst”. Valid values are ‘ClusterFirstWithHostNet’, ‘ClusterFirst’, ‘Default’ or ‘None’. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to ‘ClusterFirstWithHostNet’. |
dnsConfig
Kubernetes core/v1.PodDNSConfig
|
(Optional)
Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. |
AbstractVertex
(Appears on: PipelineSpec, VertexSpec)
Field | Description |
---|---|
name string
|
|
source
Source
|
(Optional) |
sink
Sink
|
(Optional) |
udf
UDF
|
(Optional) |
containerTemplate
ContainerTemplate
|
(Optional)
Container template for the main numa container. |
initContainerTemplate
ContainerTemplate
|
(Optional)
Container template for all the vertex pod init containers spawned by numaflow, excluding the ones specified by the user. |
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
volumes
\[\]Kubernetes core/v1.Volume
|
(Optional) |
limits
VertexLimits
|
(Optional)
Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, will override pipeline level settings |
scale
Scale
|
(Optional)
Settings for autoscaling |
initContainers
\[\]Kubernetes core/v1.Container
|
(Optional)
List of customized init containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ |
sidecars
\[\]Kubernetes core/v1.Container
|
(Optional)
List of customized sidecar containers belonging to the pod. |
partitions int32
|
(Optional)
Number of partitions of the vertex owned buffers. It applies to udf and sink vertices only. |
sideInputs \[\]string
|
(Optional)
Names of the side inputs used in this vertex. |
sideInputsContainerTemplate
ContainerTemplate
|
(Optional)
Container template for the side inputs watcher container. |
Authorization
(Appears on: HTTPSource)
Field | Description |
---|---|
token
Kubernetes core/v1.SecretKeySelector
|
(Optional)
A secret selector which contains bearer token To use this, the client
needs to add “Authorization: Bearer |
BasicAuth
(Appears on: NatsAuth)
BasicAuth represents the basic authentication approach which contains a user name and a password.
Field | Description |
---|---|
user
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Secret for auth user |
password
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Secret for auth password |
Blackhole
(Appears on: Sink)
Blackhole is a sink to emulate /dev/null
BufferFullWritingStrategy (string
alias)
(Appears on: Edge)
BufferServiceConfig
(Appears on: InterStepBufferServiceStatus)
Field | Description |
---|---|
redis
RedisConfig
|
|
jetstream
JetStreamConfig
|
CombinedEdge
(Appears on: VertexSpec)
CombinedEdge is a combination of Edge and some other properties such as vertex type, partitions, limits. It’s used to decorate the fromEdges and toEdges of the generated Vertex objects, so that in the vertex pod, it knows the properties of the connected vertices, for example, how many partitioned buffers I should write to, what is the write buffer length, etc.
Field | Description |
---|---|
Edge
Edge
|
(Members of |
fromVertexType
VertexType
|
From vertex type. |
fromVertexPartitionCount int32
|
(Optional)
The number of partitions of the from vertex, if not provided, the default value is set to “1”. |
fromVertexLimits
VertexLimits
|
(Optional) |
toVertexType
VertexType
|
To vertex type. |
toVertexPartitionCount int32
|
(Optional)
The number of partitions of the to vertex, if not provided, the default value is set to “1”. |
toVertexLimits
VertexLimits
|
(Optional) |
ConditionType (string
alias)
ConditionType is a valid value of Condition.Type
Container
(Appears on: SideInput, UDF, UDSink, UDSource, UDTransformer)
Container is used to define the container properties for user defined functions, sinks, etc.
Field | Description |
---|---|
image string
|
(Optional) |
command \[\]string
|
(Optional) |
args \[\]string
|
(Optional) |
env
\[\]Kubernetes core/v1.EnvVar
|
(Optional) |
envFrom
\[\]Kubernetes core/v1.EnvFromSource
|
(Optional) |
volumeMounts
\[\]Kubernetes core/v1.VolumeMount
|
(Optional) |
resources
Kubernetes core/v1.ResourceRequirements
|
(Optional) |
securityContext
Kubernetes core/v1.SecurityContext
|
(Optional) |
imagePullPolicy
Kubernetes core/v1.PullPolicy
|
(Optional) |
ContainerTemplate
(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, NativeRedis, SideInputsManagerTemplate, VertexTemplate)
ContainerTemplate defines customized spec for a container
Field | Description |
---|---|
resources
Kubernetes core/v1.ResourceRequirements
|
(Optional) |
imagePullPolicy
Kubernetes core/v1.PullPolicy
|
(Optional) |
securityContext
Kubernetes core/v1.SecurityContext
|
(Optional) |
env
\[\]Kubernetes core/v1.EnvVar
|
(Optional) |
envFrom
\[\]Kubernetes core/v1.EnvFromSource
|
(Optional) |
DaemonTemplate
(Appears on: Templates)
Field | Description |
---|---|
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
replicas int32
|
(Optional)
Replicas is the number of desired replicas of the Deployment. This is a pointer to distinguish between explicit zero and unspecified. Defaults to 1. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#what-is-a-replicationcontroller |
containerTemplate
ContainerTemplate
|
(Optional) |
initContainerTemplate
ContainerTemplate
|
(Optional) |
Edge
(Appears on: CombinedEdge, PipelineSpec)
Field | Description |
---|---|
from string
|
|
to string
|
|
conditions
ForwardConditions
|
(Optional)
Conditional forwarding, only allowed when “From” is a Sink or UDF. |
onFull
BufferFullWritingStrategy
|
(Optional)
OnFull specifies the behaviour for the write actions when the inter step buffer is full. There are currently two options, retryUntilSuccess and discardLatest. if not provided, the default value is set to “retryUntilSuccess” |
FixedWindow
(Appears on: Window)
FixedWindow describes a fixed window
Field | Description |
---|---|
length
Kubernetes meta/v1.Duration
|
ForwardConditions
(Appears on: Edge)
Field | Description |
---|---|
tags
TagConditions
|
Tags used to specify tags for conditional forwarding |
Function
(Appears on: UDF)
Field | Description |
---|---|
name string
|
|
args \[\]string
|
(Optional) |
kwargs map\[string\]string
|
(Optional) |
GSSAPI
(Appears on: SASL)
GSSAPI represents a SASL GSSAPI config
Field | Description |
---|---|
serviceName string
|
|
realm string
|
|
usernameSecret
Kubernetes core/v1.SecretKeySelector
|
UsernameSecret refers to the secret that contains the username |
authType
KRB5AuthType
|
valid inputs - KRB5_USER_AUTH, KRB5_KEYTAB_AUTH |
passwordSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
PasswordSecret refers to the secret that contains the password |
keytabSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
KeytabSecret refers to the secret that contains the keytab |
kerberosConfigSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
KerberosConfigSecret refers to the secret that contains the kerberos config |
GeneratorSource
(Appears on: Source)
Field | Description |
---|---|
rpu int64
|
(Optional) |
duration
Kubernetes meta/v1.Duration
|
(Optional) |
msgSize int32
|
(Optional)
Size of each generated message |
keyCount int32
|
KeyCount is the number of unique keys in the payload |
value uint64
|
Value is an optional uint64 value to be written in to the payload |
GetDaemonDeploymentReq
Field | Description |
---|---|
ISBSvcType
ISBSvcType
|
|
Image string
|
|
PullPolicy
Kubernetes core/v1.PullPolicy
|
|
Env
\[\]Kubernetes core/v1.EnvVar
|
GetJetStreamServiceSpecReq
Field | Description |
---|---|
Labels map\[string\]string
|
|
ClusterPort int32
|
|
ClientPort int32
|
|
MonitorPort int32
|
|
MetricsPort int32
|
GetJetStreamStatefulSetSpecReq
Field | Description |
---|---|
ServiceName string
|
|
Labels map\[string\]string
|
|
NatsImage string
|
|
MetricsExporterImage string
|
|
ConfigReloaderImage string
|
|
ClusterPort int32
|
|
ClientPort int32
|
|
MonitorPort int32
|
|
MetricsPort int32
|
|
ServerAuthSecretName string
|
|
ServerEncryptionSecretName string
|
|
ConfigMapName string
|
|
PvcNameIfNeeded string
|
|
StartCommand string
|
GetRedisServiceSpecReq
Field | Description |
---|---|
Labels map\[string\]string
|
|
RedisContainerPort int32
|
|
SentinelContainerPort int32
|
GetRedisStatefulSetSpecReq
Field | Description |
---|---|
ServiceName string
|
|
Labels map\[string\]string
|
|
RedisImage string
|
|
SentinelImage string
|
|
MetricsExporterImage string
|
|
InitContainerImage string
|
|
RedisContainerPort int32
|
|
SentinelContainerPort int32
|
|
RedisMetricsContainerPort int32
|
|
CredentialSecretName string
|
|
TLSEnabled bool
|
|
PvcNameIfNeeded string
|
|
ConfConfigMapName string
|
|
ScriptsConfigMapName string
|
|
HealthConfigMapName string
|
GetSideInputDeploymentReq
Field | Description |
---|---|
ISBSvcType
ISBSvcType
|
|
Image string
|
|
PullPolicy
Kubernetes core/v1.PullPolicy
|
|
Env
\[\]Kubernetes core/v1.EnvVar
|
GetVertexPodSpecReq
Field | Description |
---|---|
ISBSvcType
ISBSvcType
|
|
Image string
|
|
PullPolicy
Kubernetes core/v1.PullPolicy
|
|
Env
\[\]Kubernetes core/v1.EnvVar
|
|
SideInputsStoreName string
|
GroupBy
(Appears on: UDF)
GroupBy indicates it is a reducer UDF
Field | Description |
---|---|
window
Window
|
Window describes the windowing strategy. |
keyed bool
|
(Optional) |
allowedLateness
Kubernetes meta/v1.Duration
|
(Optional)
AllowedLateness allows late data to be included for the Reduce operation as long as the late data is not later than (Watermark - AllowedLateness). |
storage
PBQStorage
|
Storage is used to define the PBQ storage for a reduce vertex. |
HTTPSource
(Appears on: Source)
Field | Description |
---|---|
auth
Authorization
|
(Optional) |
service bool
|
(Optional)
Whether to create a ClusterIP Service |
ISBSvcPhase (string
alias)
(Appears on: InterStepBufferServiceStatus)
ISBSvcType (string
alias)
(Appears on: GetDaemonDeploymentReq, GetSideInputDeploymentReq, GetVertexPodSpecReq, InterStepBufferServiceStatus)
InterStepBufferService
Field | Description | ||||
---|---|---|---|---|---|
metadata
Kubernetes meta/v1.ObjectMeta
|
Refer to the Kubernetes API documentation for the fields of the
metadata field.
|
||||
spec
InterStepBufferServiceSpec
|
|
||||
status
InterStepBufferServiceStatus
|
(Optional) |
InterStepBufferServiceSpec
(Appears on: InterStepBufferService)
Field | Description |
---|---|
redis
RedisBufferService
|
|
jetstream
JetStreamBufferService
|
InterStepBufferServiceStatus
(Appears on: InterStepBufferService)
Field | Description |
---|---|
Status
Status
|
(Members of |
phase
ISBSvcPhase
|
|
message string
|
|
config
BufferServiceConfig
|
|
type
ISBSvcType
|
JetStreamBufferService
(Appears on: InterStepBufferServiceSpec)
Field | Description |
---|---|
version string
|
JetStream version, such as “2.7.1” |
replicas int32
|
Redis StatefulSet size |
containerTemplate
ContainerTemplate
|
(Optional)
ContainerTemplate contains customized spec for NATS container |
reloaderContainerTemplate
ContainerTemplate
|
(Optional)
ReloaderContainerTemplate contains customized spec for config reloader container |
metricsContainerTemplate
ContainerTemplate
|
(Optional)
MetricsContainerTemplate contains customized spec for metrics container |
persistence
PersistenceStrategy
|
(Optional) |
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
settings string
|
(Optional)
Nats/JetStream configuration, if not specified, global settings in numaflow-controller-config will be used. See https://docs.nats.io/running-a-nats-service/configuration#limits and https://docs.nats.io/running-a-nats-service/configuration#jetstream. For limits, only “max_payload” is supported for configuration, defaults to 1048576 (1MB), not recommended to use values over 8388608 (8MB) but max_payload can be set up to 67108864 (64MB). For jetstream, only “max_memory_store” and “max_file_store” are supported for configuration, do not set “store_dir” as it has been hardcoded. |
startArgs \[\]string
|
(Optional)
Optional arguments to start nats-server. For example, “-D” to enable debugging output, “-DV” to enable debugging and tracing. Check https://docs.nats.io/ for all the available arguments. |
bufferConfig string
|
(Optional)
Optional configuration for the streams, consumers and buckets to be created in this JetStream service, if specified, it will be merged with the default configuration in numaflow-controller-config. It accepts a YAML format configuration, it may include 4 sections, “stream”, “consumer”, “otBucket” and “procBucket”. Available fields under “stream” include “retention” (e.g. interest, limits, workerQueue), “maxMsgs”, “maxAge” (e.g. 72h), “replicas” (1, 3, 5), “duplicates” (e.g. 5m). Available fields under “consumer” include “ackWait” (e.g. 60s) Available fields under “otBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5). Available fields under “procBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5). |
encryption bool
|
(Optional)
Whether encrypt the data at rest, defaults to false Enabling encryption might impact the performance, see https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/encryption_at_rest for the detail Toggling the value will impact encrypting/decrypting existing messages. |
tls bool
|
(Optional)
Whether enable TLS, defaults to false Enabling TLS might impact the performance |
JetStreamConfig
(Appears on: BufferServiceConfig)
Field | Description |
---|---|
url string
|
JetStream (NATS) URL |
auth
NatsAuth
|
|
streamConfig string
|
(Optional) |
tlsEnabled bool
|
TLS enabled or not |
JobTemplate
(Appears on: Templates)
Field | Description |
---|---|
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
containerTemplate
ContainerTemplate
|
(Optional) |
ttlSecondsAfterFinished int32
|
(Optional)
ttlSecondsAfterFinished limits the lifetime of a Job that has finished execution (either Complete or Failed). If this field is set, ttlSecondsAfterFinished after the Job finishes, it is eligible to be automatically deleted. When the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the Job won’t be automatically deleted. If this field is set to zero, the Job becomes eligible to be deleted immediately after it finishes. Numaflow defaults to 30 |
backoffLimit int32
|
(Optional)
Specifies the number of retries before marking this job failed. More info: https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy Numaflow defaults to 20 |
KRB5AuthType (string
alias)
(Appears on: GSSAPI)
KRB5AuthType describes the kerberos auth type
KafkaSink
(Appears on: Sink)
Field | Description |
---|---|
brokers \[\]string
|
|
topic string
|
|
tls
TLS
|
(Optional)
TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. |
config string
|
(Optional) |
sasl
SASL
|
(Optional)
SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL. |
KafkaSource
(Appears on: Source)
Field | Description |
---|---|
brokers \[\]string
|
|
topic string
|
|
consumerGroup string
|
|
tls
TLS
|
(Optional)
TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. |
config string
|
(Optional) |
sasl
SASL
|
(Optional)
SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL. |
Lifecycle
(Appears on: PipelineSpec)
Field | Description |
---|---|
deleteGracePeriodSeconds int32
|
(Optional)
DeleteGracePeriodSeconds used to delete pipeline gracefully |
desiredPhase
PipelinePhase
|
(Optional)
DesiredPhase used to bring the pipeline from current phase to desired phase |
Log
(Appears on: Sink)
LogicOperator (string
alias)
(Appears on: TagConditions)
Metadata
(Appears on: AbstractPodTemplate)
Field | Description |
---|---|
annotations map\[string\]string
|
|
labels map\[string\]string
|
NativeRedis
(Appears on: RedisBufferService)
Field | Description |
---|---|
version string
|
Redis version, such as “6.0.16” |
replicas int32
|
Redis StatefulSet size |
redisContainerTemplate
ContainerTemplate
|
(Optional)
RedisContainerTemplate contains customized spec for Redis container |
sentinelContainerTemplate
ContainerTemplate
|
(Optional)
SentinelContainerTemplate contains customized spec for Redis container |
metricsContainerTemplate
ContainerTemplate
|
(Optional)
MetricsContainerTemplate contains customized spec for metrics container |
initContainerTemplate
ContainerTemplate
|
(Optional) |
persistence
PersistenceStrategy
|
(Optional) |
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
settings
RedisSettings
|
(Optional)
Redis configuration, if not specified, global settings in numaflow-controller-config will be used. |
NatsAuth
(Appears on: JetStreamConfig, NatsSource)
NatsAuth defines how to authenticate the nats access
Field | Description |
---|---|
basic
BasicAuth
|
(Optional)
Basic auth which contains a user name and a password |
token
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Token auth |
nkey
Kubernetes core/v1.SecretKeySelector
|
(Optional)
NKey auth |
NatsSource
(Appears on: Source)
Field | Description |
---|---|
url string
|
URL to connect to NATS cluster, multiple urls could be separated by comma. |
subject string
|
Subject holds the name of the subject onto which messages are published. |
queue string
|
Queue is used for queue subscription. |
tls
TLS
|
(Optional)
TLS configuration for the nats client. |
auth
NatsAuth
|
(Optional)
Auth information |
PBQStorage
(Appears on: GroupBy)
PBQStorage defines the persistence configuration for a vertex.
Field | Description |
---|---|
persistentVolumeClaim
PersistenceStrategy
|
(Optional) |
emptyDir
Kubernetes core/v1.EmptyDirVolumeSource
|
(Optional) |
PersistenceStrategy
(Appears on: JetStreamBufferService, NativeRedis, PBQStorage)
PersistenceStrategy defines the strategy of persistence
Field | Description |
---|---|
storageClassName string
|
(Optional)
Name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1 |
accessMode
Kubernetes core/v1.PersistentVolumeAccessMode
|
(Optional)
Available access modes such as ReadWriteOnce, ReadWriteMany https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes |
volumeSize
k8s.io/apimachinery/pkg/api/resource.Quantity
|
Volume size, e.g. 50Gi |
Pipeline
Field | Description | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
metadata
Kubernetes meta/v1.ObjectMeta
|
Refer to the Kubernetes API documentation for the fields of the
metadata field.
|
||||||||||||||||
spec
PipelineSpec
|
|
||||||||||||||||
status
PipelineStatus
|
(Optional) |
PipelineLimits
(Appears on: PipelineSpec)
Field | Description |
---|---|
readBatchSize uint64
|
(Optional)
Read batch size for all the vertices in the pipeline, can be overridden by the vertex’s limit settings. |
bufferMaxLength uint64
|
(Optional)
BufferMaxLength is used to define the max length of a buffer. Only applies to UDF and Source vertices as only they do buffer write. It can be overridden by the settings in vertex limits. |
bufferUsageLimit uint32
|
(Optional)
BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. Only applies to UDF and Source vertices as only they do buffer write. It will be overridden by the settings in vertex limits. |
readTimeout
Kubernetes meta/v1.Duration
|
(Optional)
Read timeout for all the vertices in the pipeline, can be overridden by the vertex’s limit settings |
PipelinePhase (string
alias)
(Appears on: Lifecycle, PipelineStatus)
PipelineSpec
(Appears on: Pipeline)
Field | Description |
---|---|
interStepBufferServiceName string
|
(Optional) |
vertices
\[\]AbstractVertex
|
|
edges
\[\]Edge
|
Edges define the relationships between vertices |
lifecycle
Lifecycle
|
(Optional)
Lifecycle define the Lifecycle properties |
limits
PipelineLimits
|
(Optional)
Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, they could be overridden by each vertex’s settings |
watermark
Watermark
|
(Optional)
Watermark enables watermark progression across the entire pipeline. |
templates
Templates
|
(Optional)
Templates is used to customize additional kubernetes resources required for the Pipeline |
sideInputs
\[\]SideInput
|
(Optional)
SideInputs defines the Side Inputs of a pipeline. |
PipelineStatus
(Appears on: Pipeline)
Field | Description |
---|---|
Status
Status
|
(Members of |
phase
PipelinePhase
|
|
message string
|
|
lastUpdated
Kubernetes meta/v1.Time
|
|
vertexCount uint32
|
|
sourceCount uint32
|
|
sinkCount uint32
|
|
udfCount uint32
|
RedisBufferService
(Appears on: InterStepBufferServiceSpec)
Field | Description |
---|---|
native
NativeRedis
|
Native brings up a native Redis service |
external
RedisConfig
|
External holds an External Redis config |
RedisConfig
(Appears on: BufferServiceConfig, RedisBufferService, RedisStreamsSource)
Field | Description |
---|---|
url string
|
(Optional)
Redis URL |
sentinelUrl string
|
(Optional)
Sentinel URL, will be ignored if Redis URL is provided |
masterName string
|
(Optional)
Only required when Sentinel is used |
user string
|
(Optional)
Redis user |
password
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Redis password secret selector |
sentinelPassword
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Sentinel password secret selector |
RedisSettings
(Appears on: NativeRedis)
Field | Description |
---|---|
redis string
|
(Optional)
Redis settings shared by both master and slaves, will override the global settings from controller config |
master string
|
(Optional)
Special settings for Redis master node, will override the global settings from controller config |
replica string
|
(Optional)
Special settings for Redis replica nodes, will override the global settings from controller config |
sentinel string
|
(Optional)
Sentinel settings, will override the global settings from controller config |
RedisStreamsSource
(Appears on: Source)
Field | Description |
---|---|
RedisConfig
RedisConfig
|
(Members of RedisConfig contains connectivity info |
stream string
|
|
consumerGroup string
|
|
readFromBeginning bool
|
if true, stream starts being read from the beginning; otherwise, the latest |
tls
TLS
|
(Optional) |
SASL
(Appears on: KafkaSink, KafkaSource)
Field | Description |
---|---|
mechanism
SASLType
|
SASL mechanism to use |
gssapi
GSSAPI
|
(Optional)
GSSAPI contains the kerberos config |
plain
SASLPlain
|
(Optional)
SASLPlain contains the sasl plain config |
SASLPlain
(Appears on: SASL)
Field | Description |
---|---|
userSecret
Kubernetes core/v1.SecretKeySelector
|
UserSecret refers to the secret that contains the user |
passwordSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
PasswordSecret refers to the secret that contains the password |
handshake bool
|
SASLType (string
alias)
(Appears on: SASL)
SASLType describes the SASL type
Scale
(Appears on: AbstractVertex)
Scale defines the parameters for autoscaling.
Field | Description |
---|---|
disabled bool
|
(Optional)
Whether to disable autoscaling. Set to “true” when using Kubernetes HPA or any other 3rd party autoscaling strategies. |
min int32
|
(Optional)
Minimum replicas. |
max int32
|
(Optional)
Maximum replicas. |
lookbackSeconds uint32
|
(Optional)
Lookback seconds to calculate the average pending messages and processing rate. |
cooldownSeconds uint32
|
(Optional)
Deprecated: Use scaleUpCooldownSeconds and scaleDownCooldownSeconds instead. Cooldown seconds after a scaling operation before another one. |
zeroReplicaSleepSeconds uint32
|
(Optional)
After scaling down to 0, sleep how many seconds before scaling up to peek. |
targetProcessingSeconds uint32
|
(Optional)
TargetProcessingSeconds is used to tune the aggressiveness of autoscaling for source vertices, it measures how fast you want the vertex to process all the pending messages. Typically increasing the value, which leads to lower processing rate, thus less replicas. It’s only effective for source vertices. |
targetBufferAvailability uint32
|
(Optional)
TargetBufferAvailability is used to define the target percentage of the buffer availability. A valid and meaningful value should be less than the BufferUsageLimit defined in the Edge spec (or Pipeline spec), for example, 50. It only applies to UDF and Sink vertices because only they have buffers to read. |
replicasPerScale uint32
|
(Optional)
ReplicasPerScale defines maximum replicas can be scaled up or down at once. The is use to prevent too aggressive scaling operations |
scaleUpCooldownSeconds uint32
|
(Optional)
ScaleUpCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling up. It defaults to the CooldownSeconds if not set. |
scaleDownCooldownSeconds uint32
|
(Optional)
ScaleDownCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling down. It defaults to the CooldownSeconds if not set. |
SideInput
(Appears on: PipelineSpec)
SideInput defines information of a Side Input
Field | Description |
---|---|
name string
|
|
container
Container
|
|
volumes
\[\]Kubernetes core/v1.Volume
|
(Optional) |
trigger
SideInputTrigger
|
SideInputTrigger
(Appears on: SideInput)
Field | Description |
---|---|
schedule string
|
The schedule to trigger the retrievement of the side input data. It supports cron format, for example, “0 30 \* \* \* \*”. Or interval based format, such as “@hourly”, “@every 1h30m”, etc. |
timezone string
|
(Optional) |
SideInputsManagerTemplate
(Appears on: Templates)
Field | Description |
---|---|
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
containerTemplate
ContainerTemplate
|
(Optional)
Template for the side inputs manager numa container |
initContainerTemplate
ContainerTemplate
|
(Optional)
Template for the side inputs manager init container |
Sink
(Appears on: AbstractVertex)
Field | Description |
---|---|
log
Log
|
|
kafka
KafkaSink
|
|
blackhole
Blackhole
|
|
udsink
UDSink
|
SlidingWindow
(Appears on: Window)
SlidingWindow describes a sliding window
Field | Description |
---|---|
length
Kubernetes meta/v1.Duration
|
|
slide
Kubernetes meta/v1.Duration
|
Source
(Appears on: AbstractVertex)
Field | Description |
---|---|
generator
GeneratorSource
|
(Optional) |
kafka
KafkaSource
|
(Optional) |
http
HTTPSource
|
(Optional) |
nats
NatsSource
|
(Optional) |
redisStreams
RedisStreamsSource
|
(Optional) |
transformer
UDTransformer
|
(Optional) |
udsource
UDSource
|
(Optional) |
Status
(Appears on: InterStepBufferServiceStatus, PipelineStatus)
Status is a common structure which can be used for Status field.
Field | Description |
---|---|
conditions
\[\]Kubernetes meta/v1.Condition
|
(Optional)
Conditions are the latest available observations of a resource’s current state. |
TLS
(Appears on: KafkaSink, KafkaSource, NatsSource, RedisStreamsSource)
Field | Description |
---|---|
insecureSkipVerify bool
|
(Optional) |
caCertSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
CACertSecret refers to the secret that contains the CA cert |
clientCertSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
CertSecret refers to the secret that contains the cert |
clientKeySecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
KeySecret refers to the secret that contains the key |
TagConditions
(Appears on: ForwardConditions)
Field | Description |
---|---|
operator
LogicOperator
|
(Optional)
Operator specifies the type of operation that should be used for conditional forwarding value could be “and”, “or”, “not” |
values \[\]string
|
Values tag values for conditional forwarding |
Templates
(Appears on: PipelineSpec)
Field | Description |
---|---|
daemon
DaemonTemplate
|
(Optional)
DaemonTemplate is used to customize the Daemon Deployment. |
job
JobTemplate
|
(Optional)
JobTemplate is used to customize Jobs. |
sideInputsManager
SideInputsManagerTemplate
|
(Optional)
SideInputsManagerTemplate is used to customize the Side Inputs Manager. |
vertex
VertexTemplate
|
(Optional)
VertexTemplate is used to customize the vertices of the pipeline. |
Transformer
(Appears on: UDTransformer)
Field | Description |
---|---|
name string
|
|
args \[\]string
|
(Optional) |
kwargs map\[string\]string
|
(Optional) |
UDF
(Appears on: AbstractVertex)
Field | Description |
---|---|
container
Container
|
(Optional) |
builtin
Function
|
(Optional) |
groupBy
GroupBy
|
(Optional) |
UDSink
(Appears on: Sink)
Field | Description |
---|---|
container
Container
|
UDSource
(Appears on: Source)
Field | Description |
---|---|
container
Container
|
UDTransformer
(Appears on: Source)
Field | Description |
---|---|
container
Container
|
(Optional) |
builtin
Transformer
|
(Optional) |
Vertex
(Appears on: VertexInstance)
Field | Description | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
metadata
Kubernetes meta/v1.ObjectMeta
|
Refer to the Kubernetes API documentation for the fields of the
metadata field.
|
||||||||||||||
spec
VertexSpec
|
|
||||||||||||||
status
VertexStatus
|
(Optional) |
VertexInstance
VertexInstance is a wrapper of a vertex instance, which contains the vertex spec and the instance information such as hostname and replica index.
Field | Description |
---|---|
vertex
Vertex
|
|
hostname string
|
|
replica int32
|
VertexLimits
(Appears on: AbstractVertex, CombinedEdge)
Field | Description |
---|---|
readBatchSize uint64
|
(Optional)
Read batch size from the source or buffer. It overrides the settings from pipeline limits. |
readTimeout
Kubernetes meta/v1.Duration
|
(Optional)
Read timeout duration from the source or buffer It overrides the settings from pipeline limits. |
bufferMaxLength uint64
|
(Optional)
BufferMaxLength is used to define the max length of a buffer. It overrides the settings from pipeline limits. |
bufferUsageLimit uint32
|
(Optional)
BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. It overrides the settings from pipeline limits. |
VertexPhase (string
alias)
(Appears on: VertexStatus)
VertexSpec
(Appears on: Vertex)
Field | Description |
---|---|
AbstractVertex
AbstractVertex
|
(Members of |
pipelineName string
|
|
interStepBufferServiceName string
|
(Optional) |
replicas int32
|
(Optional) |
fromEdges
\[\]CombinedEdge
|
(Optional) |
toEdges
\[\]CombinedEdge
|
(Optional) |
watermark
Watermark
|
(Optional)
Watermark indicates watermark progression in the vertex, it’s populated from the pipeline watermark settings. |
VertexStatus
(Appears on: Vertex)
Field | Description |
---|---|
phase
VertexPhase
|
|
reason string
|
|
message string
|
|
replicas uint32
|
|
selector string
|
|
lastScaledAt
Kubernetes meta/v1.Time
|
VertexTemplate
(Appears on: Templates)
Field | Description |
---|---|
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
containerTemplate
ContainerTemplate
|
(Optional)
Template for the vertex numa container |
initContainerTemplate
ContainerTemplate
|
(Optional)
Template for the vertex init container |
VertexType (string
alias)
(Appears on: CombinedEdge)
Watermark
(Appears on: PipelineSpec, VertexSpec)
Field | Description |
---|---|
disabled bool
|
(Optional)
Disabled toggles the watermark propagation, defaults to false. |
maxDelay
Kubernetes meta/v1.Duration
|
(Optional)
Maximum delay allowed for watermark calculation, defaults to “0s”, which means no delay. |
Window
(Appears on: GroupBy)
Window describes windowing strategy
Field | Description |
---|---|
fixed
FixedWindow
|
(Optional) |
sliding
SlidingWindow
|
(Optional) |
Generated with gen-crd-api-reference-docs
.