APIs
Packages:
numaflow.numaproj.io/v1alpha1
Resource Types:
AbstractPodTemplate
(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, MonoVertexSpec, NativeRedis, SideInputsManagerTemplate, VertexTemplate)
AbstractPodTemplate provides a template for pod customization in vertices, daemon deployments and so on.
Field | Description |
---|---|
metadata
Metadata
|
(Optional)
Metadata sets the pods’s metadata, i.e. annotations and labels |
nodeSelector map\[string\]string
|
(Optional)
NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ |
tolerations
\[\]Kubernetes core/v1.Toleration
|
(Optional)
If specified, the pod’s tolerations. |
securityContext
Kubernetes core/v1.PodSecurityContext
|
(Optional)
SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field. |
imagePullSecrets
\[\]Kubernetes core/v1.LocalObjectReference
|
(Optional)
ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod |
priorityClassName string
|
(Optional)
If specified, indicates the Redis pod’s priority. “system-node-critical” and “system-cluster-critical” are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ |
priority int32
|
(Optional)
The priority value. Various system components use this field to find the priority of the Redis pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ |
affinity
Kubernetes core/v1.Affinity
|
(Optional)
The pod’s scheduling constraints More info: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ |
serviceAccountName string
|
(Optional)
ServiceAccountName applied to the pod |
runtimeClassName string
|
(Optional)
RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the “legacy” RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class |
automountServiceAccountToken bool
|
(Optional)
AutomountServiceAccountToken indicates whether a service account token should be automatically mounted. |
dnsPolicy
Kubernetes core/v1.DNSPolicy
|
(Optional)
Set DNS policy for the pod. Defaults to “ClusterFirst”. Valid values are ‘ClusterFirstWithHostNet’, ‘ClusterFirst’, ‘Default’ or ‘None’. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to ‘ClusterFirstWithHostNet’. |
dnsConfig
Kubernetes core/v1.PodDNSConfig
|
(Optional)
Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. |
resourceClaims
\[\]Kubernetes core/v1.PodResourceClaim
|
(Optional)
ResourceClaims defines which ResourceClaims must be allocated and reserved before the Pod is allowed to start. The resources will be made available to those containers which consume them by name. |
AbstractSink
(Appears on: Sink)
Field | Description |
---|---|
log
Log
|
(Optional)
Log sink is used to write the data to the log. |
kafka
KafkaSink
|
(Optional)
Kafka sink is used to write the data to the Kafka. |
blackhole
Blackhole
|
(Optional)
Blackhole sink is used to write the data to the blackhole sink, which is a sink that discards all the data written to it. |
udsink
UDSink
|
(Optional)
UDSink sink is used to write the data to the user-defined sink. |
AbstractVertex
(Appears on: PipelineSpec, VertexSpec)
Field | Description |
---|---|
name string
|
|
source
Source
|
(Optional) |
sink
Sink
|
(Optional) |
udf
UDF
|
(Optional) |
containerTemplate
ContainerTemplate
|
(Optional)
Container template for the main numa container. |
initContainerTemplate
ContainerTemplate
|
(Optional)
Container template for all the vertex pod init containers spawned by numaflow, excluding the ones specified by the user. |
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
volumes
\[\]Kubernetes core/v1.Volume
|
(Optional) |
limits
VertexLimits
|
(Optional)
Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, will override pipeline level settings |
scale
Scale
|
(Optional)
Settings for autoscaling |
initContainers
\[\]Kubernetes core/v1.Container
|
(Optional)
List of customized init containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ |
sidecars
\[\]Kubernetes core/v1.Container
|
(Optional)
List of customized sidecar containers belonging to the pod. |
partitions int32
|
(Optional)
Number of partitions of the vertex owned buffers. It applies to udf and sink vertices only. |
sideInputs \[\]string
|
(Optional)
Names of the side inputs used in this vertex. |
sideInputsContainerTemplate
ContainerTemplate
|
(Optional)
Container template for the side inputs watcher container. |
updateStrategy
UpdateStrategy
|
(Optional)
The strategy to use to replace existing pods with new ones. |
Authorization
(Appears on: HTTPSource, ServingSource)
Field | Description |
---|---|
token
Kubernetes core/v1.SecretKeySelector
|
(Optional)
A secret selector which contains bearer token To use this, the client
needs to add “Authorization: Bearer |
Backoff
(Appears on: RetryStrategy)
Backoff defines parameters used to systematically configure the retry strategy.
Field | Description |
---|---|
interval
Kubernetes meta/v1.Duration
|
(Optional)
Interval sets the delay to wait before retry, after a failure occurs. |
steps uint32
|
(Optional)
Steps defines the number of times to try writing to a sink including retries |
BasicAuth
(Appears on: NatsAuth)
BasicAuth represents the basic authentication approach which contains a user name and a password.
Field | Description |
---|---|
user
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Secret for auth user |
password
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Secret for auth password |
Blackhole
(Appears on: AbstractSink)
Blackhole is a sink to emulate /dev/null
BufferFullWritingStrategy (string
alias)
(Appears on: Edge)
BufferServiceConfig
(Appears on: InterStepBufferServiceStatus)
Field | Description |
---|---|
redis
RedisConfig
|
|
jetstream
JetStreamConfig
|
CombinedEdge
(Appears on: VertexSpec)
CombinedEdge is a combination of Edge and some other properties such as vertex type, partitions, limits. It’s used to decorate the fromEdges and toEdges of the generated Vertex objects, so that in the vertex pod, it knows the properties of the connected vertices, for example, how many partitioned buffers I should write to, what is the write buffer length, etc.
Field | Description |
---|---|
Edge
Edge
|
(Members of |
fromVertexType
VertexType
|
From vertex type. |
fromVertexPartitionCount int32
|
(Optional)
The number of partitions of the from vertex, if not provided, the default value is set to “1”. |
fromVertexLimits
VertexLimits
|
(Optional) |
toVertexType
VertexType
|
To vertex type. |
toVertexPartitionCount int32
|
(Optional)
The number of partitions of the to vertex, if not provided, the default value is set to “1”. |
toVertexLimits
VertexLimits
|
(Optional) |
ConditionType (string
alias)
ConditionType is a valid value of Condition.Type
Container
(Appears on: SideInput, UDF, UDSink, UDSource, UDTransformer)
Container is used to define the container properties for user-defined functions, sinks, etc.
Field | Description |
---|---|
image string
|
(Optional) |
command \[\]string
|
(Optional) |
args \[\]string
|
(Optional) |
env
\[\]Kubernetes core/v1.EnvVar
|
(Optional) |
envFrom
\[\]Kubernetes core/v1.EnvFromSource
|
(Optional) |
volumeMounts
\[\]Kubernetes core/v1.VolumeMount
|
(Optional) |
resources
Kubernetes core/v1.ResourceRequirements
|
(Optional) |
securityContext
Kubernetes core/v1.SecurityContext
|
(Optional) |
imagePullPolicy
Kubernetes core/v1.PullPolicy
|
(Optional) |
readinessProbe
Probe
|
(Optional) |
livenessProbe
Probe
|
(Optional) |
ports
\[\]Kubernetes core/v1.ContainerPort
|
(Optional) |
ContainerTemplate
(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, MonoVertexSpec, NativeRedis, SideInputsManagerTemplate, VertexTemplate)
ContainerTemplate defines customized spec for a container
Field | Description |
---|---|
resources
Kubernetes core/v1.ResourceRequirements
|
(Optional) |
imagePullPolicy
Kubernetes core/v1.PullPolicy
|
(Optional) |
securityContext
Kubernetes core/v1.SecurityContext
|
(Optional) |
env
\[\]Kubernetes core/v1.EnvVar
|
(Optional) |
envFrom
\[\]Kubernetes core/v1.EnvFromSource
|
(Optional) |
readinessProbe
Probe
|
(Optional) |
livenessProbe
Probe
|
(Optional) |
DaemonTemplate
(Appears on: MonoVertexSpec, Templates)
Field | Description |
---|---|
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
replicas int32
|
(Optional)
Replicas is the number of desired replicas of the Deployment. This is a pointer to distinguish between explicit zero and unspecified. Defaults to 1. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#what-is-a-replicationcontroller |
containerTemplate
ContainerTemplate
|
(Optional) |
initContainerTemplate
ContainerTemplate
|
(Optional) |
Edge
(Appears on: CombinedEdge, PipelineSpec)
Field | Description |
---|---|
from string
|
|
to string
|
|
conditions
ForwardConditions
|
(Optional)
Conditional forwarding, only allowed when “From” is a Source or UDF. |
onFull
BufferFullWritingStrategy
|
(Optional)
OnFull specifies the behaviour for the write actions when the inter step buffer is full. There are currently two options, retryUntilSuccess and discardLatest. if not provided, the default value is set to “retryUntilSuccess” |
FixedWindow
(Appears on: Window)
FixedWindow describes a fixed window
Field | Description |
---|---|
length
Kubernetes meta/v1.Duration
|
Length is the duration of the fixed window. |
streaming bool
|
(Optional)
Streaming should be set to true if the reduce udf is streaming. |
ForwardConditions
(Appears on: Edge)
Field | Description |
---|---|
tags
TagConditions
|
Tags used to specify tags for conditional forwarding |
Function
(Appears on: UDF)
Field | Description |
---|---|
name string
|
|
args \[\]string
|
(Optional) |
kwargs map\[string\]string
|
(Optional) |
GSSAPI
(Appears on: SASL)
GSSAPI represents a SASL GSSAPI config
Field | Description |
---|---|
serviceName string
|
|
realm string
|
|
usernameSecret
Kubernetes core/v1.SecretKeySelector
|
UsernameSecret refers to the secret that contains the username |
authType
KRB5AuthType
|
valid inputs - KRB5_USER_AUTH, KRB5_KEYTAB_AUTH |
passwordSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
PasswordSecret refers to the secret that contains the password |
keytabSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
KeytabSecret refers to the secret that contains the keytab |
kerberosConfigSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
KerberosConfigSecret refers to the secret that contains the kerberos config |
GeneratorSource
(Appears on: Source)
Field | Description |
---|---|
rpu int64
|
(Optional) |
duration
Kubernetes meta/v1.Duration
|
(Optional) |
msgSize int32
|
(Optional)
Size of each generated message |
keyCount int32
|
(Optional)
KeyCount is the number of unique keys in the payload |
value uint64
|
(Optional)
Value is an optional uint64 value to be written in to the payload |
jitter
Kubernetes meta/v1.Duration
|
(Optional)
Jitter is the jitter for the message generation, used to simulate out of order messages for example if the jitter is 10s, then the message’s event time will be delayed by a random time between 0 and 10s which will result in the message being out of order by 0 to 10s |
valueBlob string
|
(Optional)
ValueBlob is an optional string which is the base64 encoding of direct payload to send. This is useful for attaching a GeneratorSource to a true pipeline to test load behavior with true messages without requiring additional work to generate messages through the external source if present, the Value and MsgSize fields will be ignored. |
GetDaemonDeploymentReq
Field | Description |
---|---|
ISBSvcType
ISBSvcType
|
|
Image string
|
|
PullPolicy
Kubernetes core/v1.PullPolicy
|
|
Env
\[\]Kubernetes core/v1.EnvVar
|
|
DefaultResources
Kubernetes core/v1.ResourceRequirements
|
GetJetStreamServiceSpecReq
Field | Description |
---|---|
Labels map\[string\]string
|
|
ClusterPort int32
|
|
ClientPort int32
|
|
MonitorPort int32
|
|
MetricsPort int32
|
GetJetStreamStatefulSetSpecReq
Field | Description |
---|---|
ServiceName string
|
|
Labels map\[string\]string
|
|
NatsImage string
|
|
MetricsExporterImage string
|
|
ConfigReloaderImage string
|
|
ClusterPort int32
|
|
ClientPort int32
|
|
MonitorPort int32
|
|
MetricsPort int32
|
|
ServerAuthSecretName string
|
|
ServerEncryptionSecretName string
|
|
ConfigMapName string
|
|
PvcNameIfNeeded string
|
|
StartCommand string
|
|
DefaultResources
Kubernetes core/v1.ResourceRequirements
|
GetMonoVertexDaemonDeploymentReq
Field | Description |
---|---|
Image string
|
|
PullPolicy
Kubernetes core/v1.PullPolicy
|
|
Env
\[\]Kubernetes core/v1.EnvVar
|
|
DefaultResources
Kubernetes core/v1.ResourceRequirements
|
GetMonoVertexPodSpecReq
Field | Description |
---|---|
Image string
|
|
PullPolicy
Kubernetes core/v1.PullPolicy
|
|
Env
\[\]Kubernetes core/v1.EnvVar
|
|
DefaultResources
Kubernetes core/v1.ResourceRequirements
|
GetRedisServiceSpecReq
Field | Description |
---|---|
Labels map\[string\]string
|
|
RedisContainerPort int32
|
|
SentinelContainerPort int32
|
GetRedisStatefulSetSpecReq
Field | Description |
---|---|
ServiceName string
|
|
Labels map\[string\]string
|
|
RedisImage string
|
|
SentinelImage string
|
|
MetricsExporterImage string
|
|
InitContainerImage string
|
|
RedisContainerPort int32
|
|
SentinelContainerPort int32
|
|
RedisMetricsContainerPort int32
|
|
CredentialSecretName string
|
|
TLSEnabled bool
|
|
PvcNameIfNeeded string
|
|
ConfConfigMapName string
|
|
ScriptsConfigMapName string
|
|
HealthConfigMapName string
|
|
DefaultResources
Kubernetes core/v1.ResourceRequirements
|
GetSideInputDeploymentReq
Field | Description |
---|---|
ISBSvcType
ISBSvcType
|
|
Image string
|
|
PullPolicy
Kubernetes core/v1.PullPolicy
|
|
Env
\[\]Kubernetes core/v1.EnvVar
|
|
DefaultResources
Kubernetes core/v1.ResourceRequirements
|
GetVertexPodSpecReq
Field | Description |
---|---|
ISBSvcType
ISBSvcType
|
|
Image string
|
|
PullPolicy
Kubernetes core/v1.PullPolicy
|
|
Env
\[\]Kubernetes core/v1.EnvVar
|
|
SideInputsStoreName string
|
|
ServingSourceStreamName string
|
|
PipelineSpec
PipelineSpec
|
|
DefaultResources
Kubernetes core/v1.ResourceRequirements
|
GroupBy
(Appears on: UDF)
GroupBy indicates it is a reducer UDF
Field | Description |
---|---|
window
Window
|
Window describes the windowing strategy. |
keyed bool
|
(Optional) |
allowedLateness
Kubernetes meta/v1.Duration
|
(Optional)
AllowedLateness allows late data to be included for the Reduce operation as long as the late data is not later than (Watermark - AllowedLateness). |
storage
PBQStorage
|
Storage is used to define the PBQ storage for a reduce vertex. |
HTTPSource
(Appears on: Source)
Field | Description |
---|---|
auth
Authorization
|
(Optional) |
service bool
|
(Optional)
Whether to create a ClusterIP Service |
ISBSvcPhase (string
alias)
(Appears on: InterStepBufferServiceStatus)
ISBSvcType (string
alias)
(Appears on: GetDaemonDeploymentReq, GetSideInputDeploymentReq, GetVertexPodSpecReq, InterStepBufferServiceStatus)
IdleSource
(Appears on: Watermark)
Field | Description |
---|---|
threshold
Kubernetes meta/v1.Duration
|
Threshold is the duration after which a source is marked as Idle due to
lack of data. Ex: If watermark found to be idle after the Threshold
duration then the watermark is progressed by |
stepInterval
Kubernetes meta/v1.Duration
|
(Optional)
StepInterval is the duration between the subsequent increment of the
watermark as long the source remains Idle. The default value is 0s which
means that once we detect idle source, we will be incrementing the
watermark by |
incrementBy
Kubernetes meta/v1.Duration
|
IncrementBy is the duration to be added to the current watermark to progress the watermark when source is idling. |
InterStepBufferService
Field | Description | ||||
---|---|---|---|---|---|
metadata
Kubernetes meta/v1.ObjectMeta
|
Refer to the Kubernetes API documentation for the fields of the
metadata field.
|
||||
spec
InterStepBufferServiceSpec
|
|
||||
status
InterStepBufferServiceStatus
|
(Optional) |
InterStepBufferServiceSpec
(Appears on: InterStepBufferService)
Field | Description |
---|---|
redis
RedisBufferService
|
|
jetstream
JetStreamBufferService
|
InterStepBufferServiceStatus
(Appears on: InterStepBufferService)
Field | Description |
---|---|
Status
Status
|
(Members of |
phase
ISBSvcPhase
|
|
message string
|
|
config
BufferServiceConfig
|
|
type
ISBSvcType
|
|
observedGeneration int64
|
JetStreamBufferService
(Appears on: InterStepBufferServiceSpec)
Field | Description |
---|---|
version string
|
JetStream version, such as “2.7.1” |
replicas int32
|
JetStream StatefulSet size |
containerTemplate
ContainerTemplate
|
(Optional)
ContainerTemplate contains customized spec for NATS container |
reloaderContainerTemplate
ContainerTemplate
|
(Optional)
ReloaderContainerTemplate contains customized spec for config reloader container |
metricsContainerTemplate
ContainerTemplate
|
(Optional)
MetricsContainerTemplate contains customized spec for metrics container |
persistence
PersistenceStrategy
|
(Optional) |
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
settings string
|
(Optional)
Nats/JetStream configuration, if not specified, global settings in numaflow-controller-config will be used. See https://docs.nats.io/running-a-nats-service/configuration#limits and https://docs.nats.io/running-a-nats-service/configuration#jetstream. For limits, only “max_payload” is supported for configuration, defaults to 1048576 (1MB), not recommended to use values over 8388608 (8MB) but max_payload can be set up to 67108864 (64MB). For jetstream, only “max_memory_store” and “max_file_store” are supported for configuration, do not set “store_dir” as it has been hardcoded. |
startArgs \[\]string
|
(Optional)
Optional arguments to start nats-server. For example, “-D” to enable debugging output, “-DV” to enable debugging and tracing. Check https://docs.nats.io/ for all the available arguments. |
bufferConfig string
|
(Optional)
Optional configuration for the streams, consumers and buckets to be created in this JetStream service, if specified, it will be merged with the default configuration in numaflow-controller-config. It accepts a YAML format configuration, it may include 4 sections, “stream”, “consumer”, “otBucket” and “procBucket”. Available fields under “stream” include “retention” (e.g. interest, limits, workerQueue), “maxMsgs”, “maxAge” (e.g. 72h), “replicas” (1, 3, 5), “duplicates” (e.g. 5m). Available fields under “consumer” include “ackWait” (e.g. 60s) Available fields under “otBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5). Available fields under “procBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5). |
encryption bool
|
(Optional)
Whether encrypt the data at rest, defaults to false Enabling encryption might impact the performance, see https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/encryption_at_rest for the detail Toggling the value will impact encrypting/decrypting existing messages. |
tls bool
|
(Optional)
Whether enable TLS, defaults to false Enabling TLS might impact the performance |
JetStreamConfig
(Appears on: BufferServiceConfig)
Field | Description |
---|---|
url string
|
JetStream (NATS) URL |
auth
NatsAuth
|
|
streamConfig string
|
(Optional) |
tlsEnabled bool
|
TLS enabled or not |
JetStreamSource
(Appears on: Source)
Field | Description |
---|---|
url string
|
URL to connect to NATS cluster, multiple urls could be separated by comma. |
stream string
|
Stream represents the name of the stream. |
tls
TLS
|
(Optional)
TLS configuration for the nats client. |
auth
NatsAuth
|
(Optional)
Auth information |
JobTemplate
(Appears on: Templates)
Field | Description |
---|---|
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
containerTemplate
ContainerTemplate
|
(Optional) |
ttlSecondsAfterFinished int32
|
(Optional)
ttlSecondsAfterFinished limits the lifetime of a Job that has finished execution (either Complete or Failed). If this field is set, ttlSecondsAfterFinished after the Job finishes, it is eligible to be automatically deleted. When the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the Job won’t be automatically deleted. If this field is set to zero, the Job becomes eligible to be deleted immediately after it finishes. Numaflow defaults to 30 |
backoffLimit int32
|
(Optional)
Specifies the number of retries before marking this job failed. More info: https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy Numaflow defaults to 20 |
KRB5AuthType (string
alias)
(Appears on: GSSAPI)
KRB5AuthType describes the kerberos auth type
KafkaSink
(Appears on: AbstractSink)
Field | Description |
---|---|
brokers \[\]string
|
|
topic string
|
|
setKey bool
|
(Optional)
SetKey sets the Kafka key to the keys passed in the Message. When the key is null (default), the record is sent randomly to one of the available partitions of the topic. If a key exists, Kafka hashes the key, and the result is used to map the message to a specific partition. This ensures that messages with the same key end up in the same partition. |
tls
TLS
|
(Optional)
TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. |
config string
|
(Optional) |
sasl
SASL
|
(Optional)
SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL. |
KafkaSource
(Appears on: Source)
Field | Description |
---|---|
brokers \[\]string
|
|
topic string
|
|
consumerGroup string
|
|
tls
TLS
|
(Optional)
TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. |
config string
|
(Optional) |
sasl
SASL
|
(Optional)
SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL. |
kafkaVersion string
|
Lifecycle
(Appears on: PipelineSpec)
Field | Description |
---|---|
deletionGracePeriodSeconds int64
|
(Optional)
DeletionGracePeriodSeconds used to delete pipeline gracefully |
desiredPhase
PipelinePhase
|
(Optional)
DesiredPhase used to bring the pipeline from current phase to desired phase |
pauseGracePeriodSeconds int64
|
(Optional)
PauseGracePeriodSeconds used to pause pipeline gracefully |
deleteGracePeriodSeconds int64
|
(Optional)
DeleteGracePeriodSeconds used to delete pipeline gracefully Deprecated: Use DeletionGracePeriodSeconds instead |
Log
(Appears on: AbstractSink)
LogicOperator (string
alias)
(Appears on: TagConditions)
Metadata
(Appears on: AbstractPodTemplate)
Field | Description |
---|---|
annotations map\[string\]string
|
|
labels map\[string\]string
|
MonoVertex
Field | Description | ||||||||||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
metadata
Kubernetes meta/v1.ObjectMeta
|
Refer to the Kubernetes API documentation for the fields of the
metadata field.
|
||||||||||||||||||||||||||
spec
MonoVertexSpec
|
|
||||||||||||||||||||||||||
status
MonoVertexStatus
|
(Optional) |
MonoVertexLifecycle
(Appears on: MonoVertexSpec)
Field | Description |
---|---|
desiredPhase
MonoVertexPhase
|
(Optional)
DesiredPhase used to bring the pipeline from current phase to desired phase |
MonoVertexLimits
(Appears on: MonoVertexSpec)
Field | Description |
---|---|
readBatchSize uint64
|
(Optional)
Read batch size from the source. |
readTimeout
Kubernetes meta/v1.Duration
|
(Optional)
Read timeout duration from the source. |
MonoVertexPhase (string
alias)
(Appears on: MonoVertexLifecycle, MonoVertexStatus)
MonoVertexSpec
(Appears on: MonoVertex)
Field | Description |
---|---|
replicas int32
|
(Optional) |
source
Source
|
|
sink
Sink
|
|
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
containerTemplate
ContainerTemplate
|
(Optional)
Container template for the main numa container. |
volumes
\[\]Kubernetes core/v1.Volume
|
(Optional) |
limits
MonoVertexLimits
|
(Optional)
Limits define the limitations such as read batch size for the mono vertex. |
scale
Scale
|
(Optional)
Settings for autoscaling |
initContainers
\[\]Kubernetes core/v1.Container
|
(Optional)
List of customized init containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ |
sidecars
\[\]Kubernetes core/v1.Container
|
(Optional)
List of customized sidecar containers belonging to the pod. |
daemonTemplate
DaemonTemplate
|
(Optional)
Template for the daemon service deployment. |
updateStrategy
UpdateStrategy
|
(Optional)
The strategy to use to replace existing pods with new ones. |
lifecycle
MonoVertexLifecycle
|
(Optional)
Lifecycle defines the Lifecycle properties of a MonoVertex |
MonoVertexStatus
(Appears on: MonoVertex)
Field | Description |
---|---|
Status
Status
|
(Members of |
phase
MonoVertexPhase
|
(Optional) |
replicas uint32
|
(Optional)
Total number of non-terminated pods targeted by this MonoVertex (their labels match the selector). |
desiredReplicas uint32
|
(Optional)
The number of desired replicas. |
selector string
|
(Optional) |
reason string
|
(Optional) |
message string
|
(Optional) |
lastUpdated
Kubernetes meta/v1.Time
|
(Optional) |
lastScaledAt
Kubernetes meta/v1.Time
|
(Optional)
Time of last scaling operation. |
observedGeneration int64
|
(Optional)
The generation observed by the MonoVertex controller. |
readyReplicas uint32
|
(Optional)
The number of pods targeted by this MonoVertex with a Ready Condition. |
updatedReplicas uint32
|
The number of Pods created by the controller from the MonoVertex version indicated by updateHash. |
updatedReadyReplicas uint32
|
The number of ready Pods created by the controller from the MonoVertex version indicated by updateHash. |
currentHash string
|
If not empty, indicates the current version of the MonoVertex used to generate Pods. |
updateHash string
|
If not empty, indicates the updated version of the MonoVertex used to generate Pods. |
NativeRedis
(Appears on: RedisBufferService)
Field | Description |
---|---|
version string
|
Redis version, such as “6.0.16” |
replicas int32
|
Redis StatefulSet size |
redisContainerTemplate
ContainerTemplate
|
(Optional)
RedisContainerTemplate contains customized spec for Redis container |
sentinelContainerTemplate
ContainerTemplate
|
(Optional)
SentinelContainerTemplate contains customized spec for Redis container |
metricsContainerTemplate
ContainerTemplate
|
(Optional)
MetricsContainerTemplate contains customized spec for metrics container |
initContainerTemplate
ContainerTemplate
|
(Optional) |
persistence
PersistenceStrategy
|
(Optional) |
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
settings
RedisSettings
|
(Optional)
Redis configuration, if not specified, global settings in numaflow-controller-config will be used. |
NatsAuth
(Appears on: JetStreamConfig, JetStreamSource, NatsSource)
NatsAuth defines how to authenticate the nats access
Field | Description |
---|---|
basic
BasicAuth
|
(Optional)
Basic auth which contains a username and a password |
token
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Token auth |
nkey
Kubernetes core/v1.SecretKeySelector
|
(Optional)
NKey auth |
NatsSource
(Appears on: Source)
Field | Description |
---|---|
url string
|
URL to connect to NATS cluster, multiple urls could be separated by comma. |
subject string
|
Subject holds the name of the subject onto which messages are published. |
queue string
|
Queue is used for queue subscription. |
tls
TLS
|
(Optional)
TLS configuration for the nats client. |
auth
NatsAuth
|
(Optional)
Auth information |
NoStore
(Appears on: PBQStorage)
NoStore means there will be no persistence storage and there will be data loss during pod restarts. Use this option only if you do not care about correctness (e.g., approx statistics pipeline like sampling rate, etc.).
OnFailureRetryStrategy (string
alias)
(Appears on: RetryStrategy)
PBQStorage
(Appears on: GroupBy)
PBQStorage defines the persistence configuration for a vertex.
Field | Description |
---|---|
persistentVolumeClaim
PersistenceStrategy
|
(Optional) |
emptyDir
Kubernetes core/v1.EmptyDirVolumeSource
|
(Optional) |
no_store
NoStore
|
(Optional) |
PersistenceStrategy
(Appears on: JetStreamBufferService, NativeRedis, PBQStorage)
PersistenceStrategy defines the strategy of persistence
Field | Description |
---|---|
storageClassName string
|
(Optional)
Name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1 |
accessMode
Kubernetes core/v1.PersistentVolumeAccessMode
|
(Optional)
Available access modes such as ReadWriteOnce, ReadWriteMany https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes |
volumeSize
k8s.io/apimachinery/pkg/api/resource.Quantity
|
Volume size, e.g. 50Gi |
Pipeline
Field | Description | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
metadata
Kubernetes meta/v1.ObjectMeta
|
Refer to the Kubernetes API documentation for the fields of the
metadata field.
|
||||||||||||||||
spec
PipelineSpec
|
|
||||||||||||||||
status
PipelineStatus
|
(Optional) |
PipelineLimits
(Appears on: PipelineSpec)
Field | Description |
---|---|
readBatchSize uint64
|
(Optional)
Read batch size for all the vertices in the pipeline, can be overridden by the vertex’s limit settings. |
bufferMaxLength uint64
|
(Optional)
BufferMaxLength is used to define the max length of a buffer. Only applies to UDF and Source vertices as only they do buffer write. It can be overridden by the settings in vertex limits. |
bufferUsageLimit uint32
|
(Optional)
BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. Only applies to UDF and Source vertices as only they do buffer write. It will be overridden by the settings in vertex limits. |
readTimeout
Kubernetes meta/v1.Duration
|
(Optional)
Read timeout for all the vertices in the pipeline, can be overridden by the vertex’s limit settings |
PipelinePhase (string
alias)
(Appears on: Lifecycle, PipelineStatus)
PipelineSpec
(Appears on: GetVertexPodSpecReq, Pipeline)
Field | Description |
---|---|
interStepBufferServiceName string
|
(Optional) |
vertices
\[\]AbstractVertex
|
|
edges
\[\]Edge
|
Edges define the relationships between vertices |
lifecycle
Lifecycle
|
(Optional)
Lifecycle define the Lifecycle properties |
limits
PipelineLimits
|
(Optional)
Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, they could be overridden by each vertex’s settings |
watermark
Watermark
|
(Optional)
Watermark enables watermark progression across the entire pipeline. |
templates
Templates
|
(Optional)
Templates are used to customize additional kubernetes resources required for the Pipeline |
sideInputs
\[\]SideInput
|
(Optional)
SideInputs defines the Side Inputs of a pipeline. |
PipelineStatus
(Appears on: Pipeline)
Field | Description |
---|---|
Status
Status
|
(Members of |
phase
PipelinePhase
|
(Optional) |
message string
|
(Optional) |
lastUpdated
Kubernetes meta/v1.Time
|
(Optional) |
vertexCount uint32
|
(Optional) |
sourceCount uint32
|
(Optional) |
sinkCount uint32
|
(Optional) |
udfCount uint32
|
(Optional) |
mapUDFCount uint32
|
(Optional) |
reduceUDFCount uint32
|
(Optional) |
observedGeneration int64
|
(Optional)
The generation observed by the Pipeline controller. |
drainedOnPause bool
|
(Optional)
Field to indicate if a pipeline drain successfully occurred, only meaningful when the pipeline is paused. True means it has been successfully drained. |
Probe
(Appears on: Container, ContainerTemplate)
Probe is used to customize the configuration for Readiness and Liveness probes.
Field | Description |
---|---|
initialDelaySeconds int32
|
(Optional)
Number of seconds after the container has started before liveness probes are initiated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes |
timeoutSeconds int32
|
(Optional)
Number of seconds after which the probe times out. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes |
periodSeconds int32
|
(Optional)
How often (in seconds) to perform the probe. |
successThreshold int32
|
(Optional)
Minimum consecutive successes for the probe to be considered successful after having failed. Defaults to 1. Must be 1 for liveness and startup. Minimum value is 1. |
failureThreshold int32
|
(Optional)
Minimum consecutive failures for the probe to be considered failed after having succeeded. Defaults to 3. Minimum value is 1. |
RedisBufferService
(Appears on: InterStepBufferServiceSpec)
Field | Description |
---|---|
native
NativeRedis
|
Native brings up a native Redis service |
external
RedisConfig
|
External holds an External Redis config |
RedisConfig
(Appears on: BufferServiceConfig, RedisBufferService)
Field | Description |
---|---|
url string
|
(Optional)
Redis URL |
sentinelUrl string
|
(Optional)
Sentinel URL, will be ignored if Redis URL is provided |
masterName string
|
(Optional)
Only required when Sentinel is used |
user string
|
(Optional)
Redis user |
password
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Redis password secret selector |
sentinelPassword
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Sentinel password secret selector |
RedisSettings
(Appears on: NativeRedis)
Field | Description |
---|---|
redis string
|
(Optional)
Redis settings shared by both master and slaves, will override the global settings from controller config |
master string
|
(Optional)
Special settings for Redis master node, will override the global settings from controller config |
replica string
|
(Optional)
Special settings for Redis replica nodes, will override the global settings from controller config |
sentinel string
|
(Optional)
Sentinel settings, will override the global settings from controller config |
RetryStrategy
(Appears on: Sink)
RetryStrategy struct encapsulates the settings for retrying operations in the event of failures. It includes a BackOff strategy to manage the timing of retries and defines the action to take upon failure.
Field | Description |
---|---|
backoff
Backoff
|
(Optional)
BackOff specifies the parameters for the backoff strategy, controlling how delays between retries should increase. |
onFailure
OnFailureRetryStrategy
|
(Optional)
OnFailure specifies the action to take when a retry fails. The default action is to retry. |
RollingUpdateStrategy
(Appears on: UpdateStrategy)
RollingUpdateStrategy is used to communicate parameter for RollingUpdateStrategyType.
Field | Description |
---|---|
maxUnavailable
k8s.io/apimachinery/pkg/util/intstr.IntOrString
|
(Optional)
The maximum number of pods that can be unavailable during the update. Value can be an absolute number (ex: 5) or a percentage of desired pods (ex: 10%). Absolute number is calculated from percentage by rounding down. Defaults to 25%. Example: when this is set to 30%, the old pods can be scaled down to 70% of desired pods immediately when the rolling update starts. Once new pods are ready, old pods can be scaled down further, followed by scaling up the new pods, ensuring that the total number of pods available at all times during the update is at least 70% of desired pods. |
SASL
(Appears on: KafkaSink, KafkaSource)
Field | Description |
---|---|
mechanism
SASLType
|
SASL mechanism to use |
gssapi
GSSAPI
|
(Optional)
GSSAPI contains the kerberos config |
plain
SASLPlain
|
(Optional)
SASLPlain contains the sasl plain config |
scramsha256
SASLPlain
|
(Optional)
SASLSCRAMSHA256 contains the sasl plain config |
scramsha512
SASLPlain
|
(Optional)
SASLSCRAMSHA512 contains the sasl plain config |
SASLPlain
(Appears on: SASL)
Field | Description |
---|---|
userSecret
Kubernetes core/v1.SecretKeySelector
|
UserSecret refers to the secret that contains the user |
passwordSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
PasswordSecret refers to the secret that contains the password |
handshake bool
|
SASLType (string
alias)
(Appears on: SASL)
SASLType describes the SASL type
Scale
(Appears on: AbstractVertex, MonoVertexSpec)
Scale defines the parameters for autoscaling.
Field | Description |
---|---|
disabled bool
|
(Optional)
Whether to disable autoscaling. Set to “true” when using Kubernetes HPA or any other 3rd party autoscaling strategies. |
min int32
|
(Optional)
Minimum replicas. |
max int32
|
(Optional)
Maximum replicas. |
lookbackSeconds uint32
|
(Optional)
Lookback seconds to calculate the average pending messages and processing rate. |
zeroReplicaSleepSeconds uint32
|
(Optional)
After scaling down the source vertex to 0, sleep how many seconds before scaling the source vertex back up to peek. |
targetProcessingSeconds uint32
|
(Optional)
TargetProcessingSeconds is used to tune the aggressiveness of autoscaling for source vertices, it measures how fast you want the vertex to process all the pending messages. Typically increasing the value, which leads to lower processing rate, thus less replicas. It’s only effective for source vertices. |
targetBufferAvailability uint32
|
(Optional)
TargetBufferAvailability is used to define the target percentage of the buffer availability. A valid and meaningful value should be less than the BufferUsageLimit defined in the Edge spec (or Pipeline spec), for example, 50. It only applies to UDF and Sink vertices because only they have buffers to read. |
replicasPerScale uint32
|
(Optional)
DeprecatedReplicasPerScale defines the number of maximum replicas that can be changed in a single scale up or down operation. The is use to prevent from too aggressive scaling operations Deprecated: Use ReplicasPerScaleUp and ReplicasPerScaleDown instead |
scaleUpCooldownSeconds uint32
|
(Optional)
ScaleUpCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling up. It defaults to the CooldownSeconds if not set. |
scaleDownCooldownSeconds uint32
|
(Optional)
ScaleDownCooldownSeconds defines the cooldown seconds after a scaling operation, before a follow-up scaling down. It defaults to the CooldownSeconds if not set. |
replicasPerScaleUp uint32
|
(Optional)
ReplicasPerScaleUp defines the number of maximum replicas that can be changed in a single scaled up operation. The is use to prevent from too aggressive scaling up operations |
replicasPerScaleDown uint32
|
(Optional)
ReplicasPerScaleDown defines the number of maximum replicas that can be changed in a single scaled down operation. The is use to prevent from too aggressive scaling down operations |
ServingSource
(Appears on: Source)
ServingSource is the HTTP endpoint for Numaflow.
Field | Description |
---|---|
auth
Authorization
|
(Optional) |
service bool
|
(Optional)
Whether to create a ClusterIP Service |
msgIDHeaderKey string
|
The header key from which the message id will be extracted |
store
ServingStore
|
Persistent store for the callbacks for serving and tracking |
ServingStore
(Appears on: ServingSource)
ServingStore to track and store data and metadata for tracking and serving.
Field | Description |
---|---|
url string
|
URL of the persistent store to write the callbacks |
ttl
Kubernetes meta/v1.Duration
|
(Optional)
TTL for the data in the store and tracker |
SessionWindow
(Appears on: Window)
SessionWindow describes a session window
Field | Description |
---|---|
timeout
Kubernetes meta/v1.Duration
|
Timeout is the duration of inactivity after which a session window closes. |
SideInput
(Appears on: PipelineSpec)
SideInput defines information of a Side Input
Field | Description |
---|---|
name string
|
|
container
Container
|
|
volumes
\[\]Kubernetes core/v1.Volume
|
(Optional) |
trigger
SideInputTrigger
|
SideInputTrigger
(Appears on: SideInput)
Field | Description |
---|---|
schedule string
|
The schedule to trigger the retrievement of the side input data. It supports cron format, for example, “0 30 \* \* \* \*”. Or interval based format, such as “@hourly”, “@every 1h30m”, etc. |
timezone string
|
(Optional) |
SideInputsManagerTemplate
(Appears on: Templates)
Field | Description |
---|---|
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
containerTemplate
ContainerTemplate
|
(Optional)
Template for the side inputs manager numa container |
initContainerTemplate
ContainerTemplate
|
(Optional)
Template for the side inputs manager init container |
Sink
(Appears on: AbstractVertex, MonoVertexSpec)
Field | Description |
---|---|
AbstractSink
AbstractSink
|
(Members of |
fallback
AbstractSink
|
(Optional)
Fallback sink can be imagined as DLQ for primary Sink. The writes to Fallback sink will only be initiated if the ud-sink response field sets it. |
retryStrategy
RetryStrategy
|
(Optional)
RetryStrategy struct encapsulates the settings for retrying operations in the event of failures. |
SlidingWindow
(Appears on: Window)
SlidingWindow describes a sliding window
Field | Description |
---|---|
length
Kubernetes meta/v1.Duration
|
Length is the duration of the sliding window. |
slide
Kubernetes meta/v1.Duration
|
Slide is the slide parameter that controls the frequency at which the sliding window is created. |
streaming bool
|
(Optional)
Streaming should be set to true if the reduce udf is streaming. |
Source
(Appears on: AbstractVertex, MonoVertexSpec)
Field | Description |
---|---|
generator
GeneratorSource
|
(Optional) |
kafka
KafkaSource
|
(Optional) |
http
HTTPSource
|
(Optional) |
nats
NatsSource
|
(Optional) |
transformer
UDTransformer
|
(Optional) |
udsource
UDSource
|
(Optional) |
jetstream
JetStreamSource
|
(Optional) |
serving
ServingSource
|
(Optional) |
Status
(Appears on: InterStepBufferServiceStatus, MonoVertexStatus, PipelineStatus, VertexStatus)
Status is a common structure which can be used for Status field.
Field | Description |
---|---|
conditions
\[\]Kubernetes meta/v1.Condition
|
(Optional)
Conditions are the latest available observations of a resource’s current state. |
TLS
(Appears on: JetStreamSource, KafkaSink, KafkaSource, NatsSource)
Field | Description |
---|---|
insecureSkipVerify bool
|
(Optional) |
caCertSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
CACertSecret refers to the secret that contains the CA cert |
certSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
CertSecret refers to the secret that contains the cert |
keySecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
KeySecret refers to the secret that contains the key |
TagConditions
(Appears on: ForwardConditions)
Field | Description |
---|---|
operator
LogicOperator
|
(Optional)
Operator specifies the type of operation that should be used for conditional forwarding value could be “and”, “or”, “not” |
values \[\]string
|
Values tag values for conditional forwarding |
Templates
(Appears on: PipelineSpec)
Field | Description |
---|---|
daemon
DaemonTemplate
|
(Optional)
DaemonTemplate is used to customize the Daemon Deployment. |
job
JobTemplate
|
(Optional)
JobTemplate is used to customize Jobs. |
sideInputsManager
SideInputsManagerTemplate
|
(Optional)
SideInputsManagerTemplate is used to customize the Side Inputs Manager. |
vertex
VertexTemplate
|
(Optional)
VertexTemplate is used to customize the vertices of the pipeline. |
Transformer
(Appears on: UDTransformer)
Field | Description |
---|---|
name string
|
|
args \[\]string
|
(Optional) |
kwargs map\[string\]string
|
(Optional) |
UDF
(Appears on: AbstractVertex)
Field | Description |
---|---|
container
Container
|
(Optional) |
builtin
Function
|
(Optional) |
groupBy
GroupBy
|
(Optional) |
UDSink
(Appears on: AbstractSink)
Field | Description |
---|---|
container
Container
|
UDSource
(Appears on: Source)
Field | Description |
---|---|
container
Container
|
UDTransformer
(Appears on: Source)
Field | Description |
---|---|
container
Container
|
(Optional) |
builtin
Transformer
|
(Optional) |
UpdateStrategy
(Appears on: AbstractVertex, MonoVertexSpec)
UpdateStrategy indicates the strategy that the controller will use to perform updates for Vertex or MonoVertex.
Field | Description |
---|---|
type
UpdateStrategyType
|
(Optional)
Type indicates the type of the StatefulSetUpdateStrategy. Default is RollingUpdate. |
rollingUpdate
RollingUpdateStrategy
|
(Optional)
RollingUpdate is used to communicate parameters when Type is RollingUpdateStrategy. |
UpdateStrategyType (string
alias)
(Appears on: UpdateStrategy)
UpdateStrategyType is a string enumeration type that enumerates all possible update strategies.
Vertex
(Appears on: VertexInstance)
Field | Description | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
metadata
Kubernetes meta/v1.ObjectMeta
|
Refer to the Kubernetes API documentation for the fields of the
metadata field.
|
||||||||||||||
spec
VertexSpec
|
|
||||||||||||||
status
VertexStatus
|
(Optional) |
VertexInstance
VertexInstance is a wrapper of a vertex instance, which contains the vertex spec and the instance information such as hostname and replica index.
Field | Description |
---|---|
vertex
Vertex
|
|
hostname string
|
|
replica int32
|
VertexLimits
(Appears on: AbstractVertex, CombinedEdge)
Field | Description |
---|---|
readBatchSize uint64
|
(Optional)
Read batch size from the source or buffer. It overrides the settings from pipeline limits. |
readTimeout
Kubernetes meta/v1.Duration
|
(Optional)
Read timeout duration from the source or buffer It overrides the settings from pipeline limits. |
bufferMaxLength uint64
|
(Optional)
BufferMaxLength is used to define the max length of a buffer. It overrides the settings from pipeline limits. |
bufferUsageLimit uint32
|
(Optional)
BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. It overrides the settings from pipeline limits. |
VertexPhase (string
alias)
(Appears on: VertexStatus)
VertexSpec
(Appears on: Vertex)
Field | Description |
---|---|
AbstractVertex
AbstractVertex
|
(Members of |
pipelineName string
|
|
interStepBufferServiceName string
|
(Optional) |
replicas int32
|
(Optional) |
fromEdges
\[\]CombinedEdge
|
(Optional) |
toEdges
\[\]CombinedEdge
|
(Optional) |
watermark
Watermark
|
(Optional)
Watermark indicates watermark progression in the vertex, it’s populated from the pipeline watermark settings. |
VertexStatus
(Appears on: Vertex)
Field | Description |
---|---|
Status
Status
|
(Members of |
phase
VertexPhase
|
(Optional) |
replicas uint32
|
(Optional)
Total number of non-terminated pods targeted by this Vertex (their labels match the selector). |
desiredReplicas uint32
|
(Optional)
The number of desired replicas. |
selector string
|
(Optional) |
reason string
|
(Optional) |
message string
|
(Optional) |
lastScaledAt
Kubernetes meta/v1.Time
|
(Optional)
Time of last scaling operation. |
observedGeneration int64
|
(Optional)
The generation observed by the Vertex controller. |
readyReplicas uint32
|
(Optional)
The number of pods targeted by this Vertex with a Ready Condition. |
updatedReplicas uint32
|
The number of Pods created by the controller from the Vertex version indicated by updateHash. |
updatedReadyReplicas uint32
|
The number of ready Pods created by the controller from the Vertex version indicated by updateHash. |
currentHash string
|
If not empty, indicates the current version of the Vertex used to generate Pods. |
updateHash string
|
If not empty, indicates the updated version of the Vertex used to generate Pods. |
VertexTemplate
(Appears on: Templates)
Field | Description |
---|---|
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
containerTemplate
ContainerTemplate
|
(Optional)
Template for the vertex numa container |
initContainerTemplate
ContainerTemplate
|
(Optional)
Template for the vertex init container |
VertexType (string
alias)
(Appears on: CombinedEdge)
Watermark
(Appears on: PipelineSpec, VertexSpec)
Field | Description |
---|---|
disabled bool
|
(Optional)
Disabled toggles the watermark propagation, defaults to false. |
maxDelay
Kubernetes meta/v1.Duration
|
(Optional)
Maximum delay allowed for watermark calculation, defaults to “0s”, which means no delay. |
idleSource
IdleSource
|
(Optional)
IdleSource defines the idle watermark properties, it could be configured in case source is idling. |
Window
(Appears on: GroupBy)
Window describes windowing strategy
Field | Description |
---|---|
fixed
FixedWindow
|
(Optional) |
sliding
SlidingWindow
|
(Optional) |
session
SessionWindow
|
(Optional) |
Generated with gen-crd-api-reference-docs
.