APIs
Packages:
numaflow.numaproj.io/v1alpha1
Resource Types:
AbstractPodTemplate
(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, NativeRedis)
AbstractPodTemplate provides a template for pod customization in vertices, daemon deployments and so on.
Field | Description |
---|---|
metadata
Metadata
|
(Optional)
Metadata sets the pods’s metadata, i.e. annotations and labels |
nodeSelector map\[string\]string
|
(Optional)
NodeSelector is a selector which must be true for the pod to fit on a node. Selector which must match a node’s labels for the pod to be scheduled on that node. More info: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/ |
tolerations
\[\]Kubernetes core/v1.Toleration
|
(Optional)
If specified, the pod’s tolerations. |
securityContext
Kubernetes core/v1.PodSecurityContext
|
(Optional)
SecurityContext holds pod-level security attributes and common container settings. Optional: Defaults to empty. See type description for default values of each field. |
imagePullSecrets
\[\]Kubernetes core/v1.LocalObjectReference
|
(Optional)
ImagePullSecrets is an optional list of references to secrets in the same namespace to use for pulling any of the images used by this PodSpec. If specified, these secrets will be passed to individual puller implementations for them to use. For example, in the case of docker, only DockerConfig type secrets are honored. More info: https://kubernetes.io/docs/concepts/containers/images#specifying-imagepullsecrets-on-a-pod |
priorityClassName string
|
(Optional)
If specified, indicates the Redis pod’s priority. “system-node-critical” and “system-cluster-critical” are two special keywords which indicate the highest priorities with the former being the highest priority. Any other name must be defined by creating a PriorityClass object with that name. If not specified, the pod priority will be default or zero if there is no default. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ |
priority int32
|
(Optional)
The priority value. Various system components use this field to find the priority of the Redis pod. When Priority Admission Controller is enabled, it prevents users from setting this field. The admission controller populates this field from PriorityClassName. The higher the value, the higher the priority. More info: https://kubernetes.io/docs/concepts/configuration/pod-priority-preemption/ |
affinity
Kubernetes core/v1.Affinity
|
(Optional)
The pod’s scheduling constraints More info: https://kubernetes.io/docs/concepts/scheduling-eviction/assign-pod-node/ |
serviceAccountName string
|
(Optional)
ServiceAccountName applied to the pod |
runtimeClassName string
|
(Optional)
RuntimeClassName refers to a RuntimeClass object in the node.k8s.io group, which should be used to run this pod. If no RuntimeClass resource matches the named class, the pod will not be run. If unset or empty, the “legacy” RuntimeClass will be used, which is an implicit class with an empty definition that uses the default runtime handler. More info: https://git.k8s.io/enhancements/keps/sig-node/585-runtime-class |
automountServiceAccountToken bool
|
(Optional)
AutomountServiceAccountToken indicates whether a service account token should be automatically mounted. |
dnsPolicy
Kubernetes core/v1.DNSPolicy
|
(Optional)
Set DNS policy for the pod. Defaults to “ClusterFirst”. Valid values are ‘ClusterFirstWithHostNet’, ‘ClusterFirst’, ‘Default’ or ‘None’. DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy. To have DNS options set along with hostNetwork, you have to specify DNS policy explicitly to ‘ClusterFirstWithHostNet’. |
dnsConfig
Kubernetes core/v1.PodDNSConfig
|
(Optional)
Specifies the DNS parameters of a pod. Parameters specified here will be merged to the generated DNS configuration based on DNSPolicy. |
AbstractVertex
(Appears on: PipelineSpec, VertexSpec)
Field | Description |
---|---|
name string
|
|
source
Source
|
(Optional) |
sink
Sink
|
(Optional) |
udf
UDF
|
(Optional) |
containerTemplate
ContainerTemplate
|
(Optional) |
initContainerTemplate
ContainerTemplate
|
(Optional) |
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
volumes
\[\]Kubernetes core/v1.Volume
|
(Optional) |
limits
VertexLimits
|
(Optional)
Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, will override pipeline level settings |
scale
Scale
|
(Optional)
Settings for autoscaling |
initContainers
\[\]Kubernetes core/v1.Container
|
(Optional)
List of init containers belonging to the pod. More info: https://kubernetes.io/docs/concepts/workloads/pods/init-containers/ |
sidecars
\[\]Kubernetes core/v1.Container
|
(Optional)
List of sidecar containers belonging to the pod. |
Authorization
(Appears on: HTTPSource)
Field | Description |
---|---|
token
Kubernetes core/v1.SecretKeySelector
|
(Optional)
A secret selector which contains bearer token To use this, the client
needs to add “Authorization: Bearer |
BasicAuth
(Appears on: NatsAuth)
BasicAuth represents the basic authentication approach which contains a user name and a password.
Field | Description |
---|---|
user
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Secret for auth user |
password
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Secret for auth password |
Blackhole
(Appears on: Sink)
Blackhole is a sink to emulate /dev/null
Buffer
Field | Description |
---|---|
Name string
|
|
Type
BufferType
|
BufferFullWritingStrategy (string
alias)
(Appears on: Edge)
BufferServiceConfig
(Appears on: InterStepBufferServiceStatus)
Field | Description |
---|---|
redis
RedisConfig
|
|
jetstream
JetStreamConfig
|
BufferType (string
alias)
(Appears on: Buffer)
ConditionType (string
alias)
ConditionType is a valid value of Condition.Type
Container
(Appears on: UDF, UDSink, UDTransformer)
Container is used to define the container properties for user defined functions, sinks, etc.
Field | Description |
---|---|
image string
|
(Optional) |
command \[\]string
|
(Optional) |
args \[\]string
|
(Optional) |
env
\[\]Kubernetes core/v1.EnvVar
|
(Optional) |
envFrom
\[\]Kubernetes core/v1.EnvFromSource
|
(Optional) |
volumeMounts
\[\]Kubernetes core/v1.VolumeMount
|
(Optional) |
resources
Kubernetes core/v1.ResourceRequirements
|
(Optional) |
securityContext
Kubernetes core/v1.SecurityContext
|
(Optional) |
imagePullPolicy
Kubernetes core/v1.PullPolicy
|
(Optional) |
ContainerTemplate
(Appears on: AbstractVertex, DaemonTemplate, JetStreamBufferService, JobTemplate, NativeRedis)
ContainerTemplate defines customized spec for a container
Field | Description |
---|---|
resources
Kubernetes core/v1.ResourceRequirements
|
(Optional) |
imagePullPolicy
Kubernetes core/v1.PullPolicy
|
(Optional) |
securityContext
Kubernetes core/v1.SecurityContext
|
(Optional) |
env
\[\]Kubernetes core/v1.EnvVar
|
(Optional) |
envFrom
\[\]Kubernetes core/v1.EnvFromSource
|
(Optional) |
DaemonTemplate
(Appears on: Templates)
Field | Description |
---|---|
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
replicas int32
|
(Optional)
Replicas is the number of desired replicas of the Deployment. This is a pointer to distinguish between explicit zero and unspecified. Defaults to 1. More info: https://kubernetes.io/docs/concepts/workloads/controllers/replicationcontroller#what-is-a-replicationcontroller |
containerTemplate
ContainerTemplate
|
(Optional) |
initContainerTemplate
ContainerTemplate
|
(Optional) |
Edge
(Appears on: PipelineSpec, VertexSpec)
Field | Description |
---|---|
from string
|
|
to string
|
|
conditions
ForwardConditions
|
(Optional)
Conditional forwarding, only allowed when “From” is a Sink or UDF. |
limits
EdgeLimits
|
(Optional)
Limits define the limitations such as buffer read batch size for the edge, will override pipeline level settings. |
parallelism int32
|
(Optional)
Parallelism is only effective when the “to” vertex is a reduce vertex, if it’s not provided, the default value is set to “1”. Parallelism is ignored when the “to” vertex is not a reduce vertex. |
onFull
BufferFullWritingStrategy
|
(Optional)
OnFull specifies the behaviour for the write actions when the inter step buffer is full. There are currently two options, retryUntilSuccess and discardLatest. if not provided, the default value is set to “retryUntilSuccess” |
EdgeLimits
(Appears on: Edge)
Field | Description |
---|---|
bufferMaxLength uint64
|
(Optional)
BufferMaxLength is used to define the max length of a buffer. It overrides the settings from pipeline limits. |
bufferUsageLimit uint32
|
(Optional)
BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. It overrides the settings from pipeline limits. |
FixedWindow
(Appears on: Window)
FixedWindow describes a fixed window
Field | Description |
---|---|
length
Kubernetes meta/v1.Duration
|
ForwardConditions
(Appears on: Edge)
Field | Description |
---|---|
keyIn \[\]string
|
Function
(Appears on: UDF)
Field | Description |
---|---|
name string
|
|
args \[\]string
|
(Optional) |
kwargs map\[string\]string
|
(Optional) |
GSSAPI
(Appears on: SASL)
GSSAPI represents a SASL GSSAPI config
Field | Description |
---|---|
serviceName string
|
|
realm string
|
|
username string
|
|
authType
KRB5AuthType
|
valid inputs - KRB5_USER_AUTH, KRB5_KEYTAB_AUTH |
passwordSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
PasswordSecret refers to the secret that contains the password |
keytabSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
KeytabSecret refers to the secret that contains the keytab |
kerberosConfigSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
KerberosConfigSecret refers to the secret that contains the kerberos config |
GeneratorSource
(Appears on: Source)
Field | Description |
---|---|
rpu int64
|
(Optional) |
duration
Kubernetes meta/v1.Duration
|
(Optional) |
msgSize int32
|
(Optional)
Size of each generated message |
keyCount int32
|
KeyCount is the number of unique keys in the payload |
value uint64
|
Value is an optional uint64 value to be written in to the payload |
GetDaemonDeploymentReq
Field | Description |
---|---|
ISBSvcType
ISBSvcType
|
|
Image string
|
|
PullPolicy
Kubernetes core/v1.PullPolicy
|
|
Env
\[\]Kubernetes core/v1.EnvVar
|
GetJetStreamServiceSpecReq
Field | Description |
---|---|
Labels map\[string\]string
|
|
ClusterPort int32
|
|
ClientPort int32
|
|
MonitorPort int32
|
|
MetricsPort int32
|
GetJetStreamStatefulSetSpecReq
Field | Description |
---|---|
ServiceName string
|
|
Labels map\[string\]string
|
|
NatsImage string
|
|
MetricsExporterImage string
|
|
ConfigReloaderImage string
|
|
ClusterPort int32
|
|
ClientPort int32
|
|
MonitorPort int32
|
|
MetricsPort int32
|
|
ServerAuthSecretName string
|
|
ServerEncryptionSecretName string
|
|
ConfigMapName string
|
|
PvcNameIfNeeded string
|
|
StartCommand string
|
GetRedisServiceSpecReq
Field | Description |
---|---|
Labels map\[string\]string
|
|
RedisContainerPort int32
|
|
SentinelContainerPort int32
|
GetRedisStatefulSetSpecReq
Field | Description |
---|---|
ServiceName string
|
|
Labels map\[string\]string
|
|
RedisImage string
|
|
SentinelImage string
|
|
MetricsExporterImage string
|
|
InitContainerImage string
|
|
RedisContainerPort int32
|
|
SentinelContainerPort int32
|
|
RedisMetricsContainerPort int32
|
|
CredentialSecretName string
|
|
TLSEnabled bool
|
|
PvcNameIfNeeded string
|
|
ConfConfigMapName string
|
|
ScriptsConfigMapName string
|
|
HealthConfigMapName string
|
GetVertexPodSpecReq
Field | Description |
---|---|
ISBSvcType
ISBSvcType
|
|
Image string
|
|
PullPolicy
Kubernetes core/v1.PullPolicy
|
|
Env
\[\]Kubernetes core/v1.EnvVar
|
GroupBy
(Appears on: UDF)
GroupBy indicates it is a reducer UDF
Field | Description |
---|---|
window
Window
|
Window describes the windowing strategy. |
keyed bool
|
(Optional) |
storage
PBQStorage
|
Storage is used to define the PBQ storage for a reduce vertex. |
HTTPSource
(Appears on: Source)
Field | Description |
---|---|
auth
Authorization
|
(Optional) |
service bool
|
(Optional)
Whether to create a ClusterIP Service |
ISBSvcPhase (string
alias)
(Appears on: InterStepBufferServiceStatus)
ISBSvcType (string
alias)
(Appears on: GetDaemonDeploymentReq, GetVertexPodSpecReq, InterStepBufferServiceStatus)
InterStepBufferService
Field | Description | ||||
---|---|---|---|---|---|
metadata
Kubernetes meta/v1.ObjectMeta
|
Refer to the Kubernetes API documentation for the fields of the
metadata field.
|
||||
spec
InterStepBufferServiceSpec
|
|
||||
status
InterStepBufferServiceStatus
|
(Optional) |
InterStepBufferServiceSpec
(Appears on: InterStepBufferService)
Field | Description |
---|---|
redis
RedisBufferService
|
|
jetstream
JetStreamBufferService
|
InterStepBufferServiceStatus
(Appears on: InterStepBufferService)
Field | Description |
---|---|
Status
Status
|
(Members of |
phase
ISBSvcPhase
|
|
message string
|
|
config
BufferServiceConfig
|
|
type
ISBSvcType
|
JetStreamBufferService
(Appears on: InterStepBufferServiceSpec)
Field | Description |
---|---|
version string
|
JetStream version, such as “2.7.1” |
replicas int32
|
Redis StatefulSet size |
containerTemplate
ContainerTemplate
|
(Optional)
ContainerTemplate contains customized spec for NATS container |
reloaderContainerTemplate
ContainerTemplate
|
(Optional)
ReloaderContainerTemplate contains customized spec for config reloader container |
metricsContainerTemplate
ContainerTemplate
|
(Optional)
MetricsContainerTemplate contains customized spec for metrics container |
persistence
PersistenceStrategy
|
(Optional) |
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
settings string
|
(Optional)
JetStream configuration, if not specified, global settings in numaflow-controller-config will be used. See https://docs.nats.io/running-a-nats-service/configuration#jetstream. Only configure “max_memory_store” or “max_file_store”, do not set “store_dir” as it has been hardcoded. |
startArgs \[\]string
|
(Optional)
Optional arguments to start nats-server. For example, “-D” to enable debugging output, “-DV” to enable debugging and tracing. Check https://docs.nats.io/ for all the available arguments. |
bufferConfig string
|
(Optional)
Optional configuration for the streams, consumers and buckets to be created in this JetStream service, if specified, it will be merged with the default configuration in numaflow-controller-config. It accepts a YAML format configuration, it may include 4 sections, “stream”, “consumer”, “otBucket” and “procBucket”. Available fields under “stream” include “retention” (e.g. interest, limits, workerQueue), “maxMsgs”, “maxAge” (e.g. 72h), “replicas” (1, 3, 5), “duplicates” (e.g. 5m). Available fields under “consumer” include “ackWait” (e.g. 60s) Available fields under “otBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5). Available fields under “procBucket” include “maxValueSize”, “history”, “ttl” (e.g. 72h), “maxBytes”, “replicas” (1, 3, 5). |
encryption bool
|
(Optional)
Whether encrypt the data at rest, defaults to false Enabling encryption might impact the performance, see https://docs.nats.io/running-a-nats-service/nats_admin/jetstream_admin/encryption_at_rest for the detail Toggling the value will impact encypting/decrypting existing messages. |
tls bool
|
(Optional)
Whether enable TLS, defaults to false Enabling TLS might impact the performance |
JetStreamConfig
(Appears on: BufferServiceConfig)
Field | Description |
---|---|
url string
|
JetStream (NATS) URL |
auth
NatsAuth
|
|
bufferConfig string
|
(Optional) |
tlsEnabled bool
|
TLS enabled or not |
JobTemplate
(Appears on: Templates)
Field | Description |
---|---|
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
containerTemplate
ContainerTemplate
|
(Optional) |
ttlSecondsAfterFinished int32
|
(Optional)
ttlSecondsAfterFinished limits the lifetime of a Job that has finished execution (either Complete or Failed). If this field is set, ttlSecondsAfterFinished after the Job finishes, it is eligible to be automatically deleted. When the Job is being deleted, its lifecycle guarantees (e.g. finalizers) will be honored. If this field is unset, the Job won’t be automatically deleted. If this field is set to zero, the Job becomes eligible to be deleted immediately after it finishes. Numaflow defaults to 30 |
backoffLimit int32
|
(Optional)
Specifies the number of retries before marking this job failed. More info: https://kubernetes.io/docs/concepts/workloads/controllers/job/#pod-backoff-failure-policy Numaflow defaults to 20 |
KRB5AuthType (string
alias)
(Appears on: GSSAPI)
KRB5AuthType describes the kerberos auth type
KafkaSink
(Appears on: Sink)
Field | Description |
---|---|
brokers \[\]string
|
|
topic string
|
|
tls
TLS
|
(Optional)
TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. |
config string
|
(Optional) |
KafkaSource
(Appears on: Source)
Field | Description |
---|---|
brokers \[\]string
|
|
topic string
|
|
consumerGroup string
|
|
tls
TLS
|
(Optional)
TLS user to configure TLS connection for kafka broker TLS.enable=true default for TLS. |
config string
|
(Optional) |
sasl
SASL
|
(Optional)
SASL user to configure SASL connection for kafka broker SASL.enable=true default for SASL. |
Lifecycle
(Appears on: PipelineSpec)
Field | Description |
---|---|
deleteGracePeriodSeconds int32
|
(Optional)
DeleteGracePeriodSeconds used to delete pipeline gracefully |
desiredPhase
PipelinePhase
|
(Optional)
DesiredPhase used to bring the pipeline from current phase to desired phase |
Log
(Appears on: Sink)
Metadata
(Appears on: AbstractPodTemplate)
Field | Description |
---|---|
annotations map\[string\]string
|
|
labels map\[string\]string
|
NativeRedis
(Appears on: RedisBufferService)
Field | Description |
---|---|
version string
|
Redis version, such as “6.0.16” |
replicas int32
|
Redis StatefulSet size |
redisContainerTemplate
ContainerTemplate
|
(Optional)
RedisContainerTemplate contains customized spec for Redis container |
sentinelContainerTemplate
ContainerTemplate
|
(Optional)
SentinelContainerTemplate contains customized spec for Redis container |
metricsContainerTemplate
ContainerTemplate
|
(Optional)
MetricsContainerTemplate contains customized spec for metrics container |
initContainerTemplate
ContainerTemplate
|
(Optional) |
persistence
PersistenceStrategy
|
(Optional) |
AbstractPodTemplate
AbstractPodTemplate
|
(Members of |
settings
RedisSettings
|
(Optional)
Redis configuration, if not specified, global settings in numaflow-controller-config will be used. |
NatsAuth
(Appears on: JetStreamConfig, NatsSource)
NatsAuth defines how to authenticate the nats access
Field | Description |
---|---|
basic
BasicAuth
|
(Optional)
Basic auth which contains a user name and a password |
token
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Token auth |
nkey
Kubernetes core/v1.SecretKeySelector
|
(Optional)
NKey auth |
NatsSource
(Appears on: Source)
Field | Description |
---|---|
url string
|
URL to connect to NATS cluster, multiple urls could be separated by comma. |
subject string
|
Subject holds the name of the subject onto which messages are published. |
queue string
|
Queue is used for queue subscription. |
tls
TLS
|
(Optional)
TLS configuration for the nats client. |
auth
NatsAuth
|
(Optional)
Auth information |
PBQStorage
(Appears on: GroupBy)
PBQStorage defines the persistence configuration for a vertex.
Field | Description |
---|---|
persistentVolumeClaim
PersistenceStrategy
|
(Optional) |
emptyDir
Kubernetes core/v1.EmptyDirVolumeSource
|
(Optional) |
PersistenceStrategy
(Appears on: JetStreamBufferService, NativeRedis, PBQStorage)
PersistenceStrategy defines the strategy of persistence
Field | Description |
---|---|
storageClassName string
|
(Optional)
Name of the StorageClass required by the claim. More info: https://kubernetes.io/docs/concepts/storage/persistent-volumes#class-1 |
accessMode
Kubernetes core/v1.PersistentVolumeAccessMode
|
(Optional)
Available access modes such as ReadWriteOnce, ReadWriteMany https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes |
volumeSize
k8s.io/apimachinery/pkg/api/resource.Quantity
|
Volume size, e.g. 50Gi |
Pipeline
Field | Description | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
metadata
Kubernetes meta/v1.ObjectMeta
|
Refer to the Kubernetes API documentation for the fields of the
metadata field.
|
||||||||||||||
spec
PipelineSpec
|
|
||||||||||||||
status
PipelineStatus
|
(Optional) |
PipelineLimits
(Appears on: PipelineSpec)
Field | Description |
---|---|
readBatchSize uint64
|
(Optional)
Read batch size for all the vertices in the pipeline, can be overridden by the vertex’s limit settings |
bufferMaxLength uint64
|
(Optional)
BufferMaxLength is used to define the max length of a buffer Only applies to UDF and Source vertice as only they do buffer write. It can be overridden by the settings in vertex limits. |
bufferUsageLimit uint32
|
(Optional)
BufferUsageLimit is used to define the percentage of the buffer usage limit, a valid value should be less than 100, for example, 85. Only applies to UDF and Source vertice as only they do buffer write. It will be overridden by the settings in vertex limits. |
readTimeout
Kubernetes meta/v1.Duration
|
(Optional)
Read timeout for all the vertices in the pipeline, can be overridden by the vertex’s limit settings |
PipelinePhase (string
alias)
(Appears on: Lifecycle, PipelineStatus)
PipelineSpec
(Appears on: Pipeline)
Field | Description |
---|---|
interStepBufferServiceName string
|
(Optional) |
vertices
\[\]AbstractVertex
|
|
edges
\[\]Edge
|
Edges define the relationships between vertices |
lifecycle
Lifecycle
|
(Optional)
Lifecycle define the Lifecycle properties |
limits
PipelineLimits
|
(Optional)
Limits define the limitations such as buffer read batch size for all the vertices of a pipeline, they could be overridden by each vertex’s settings |
watermark
Watermark
|
(Optional)
Watermark enables watermark progression across the entire pipeline. |
templates
Templates
|
(Optional)
Templates is used to customize additional kubernetes resources required for the Pipeline |
PipelineStatus
(Appears on: Pipeline)
Field | Description |
---|---|
Status
Status
|
(Members of |
phase
PipelinePhase
|
|
message string
|
|
lastUpdated
Kubernetes meta/v1.Time
|
|
vertexCount uint32
|
|
sourceCount uint32
|
|
sinkCount uint32
|
|
udfCount uint32
|
RedisBufferService
(Appears on: InterStepBufferServiceSpec)
Field | Description |
---|---|
native
NativeRedis
|
Native brings up a native Redis service |
external
RedisConfig
|
External holds an External Redis config |
RedisConfig
(Appears on: BufferServiceConfig, RedisBufferService, RedisStreamsSource)
Field | Description |
---|---|
url string
|
(Optional)
Redis URL |
sentinelUrl string
|
(Optional)
Sentinel URL, will be ignored if Redis URL is provided |
masterName string
|
(Optional)
Only required when Sentinel is used |
user string
|
(Optional)
Redis user |
password
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Redis password secret selector |
sentinelPassword
Kubernetes core/v1.SecretKeySelector
|
(Optional)
Sentinel password secret selector |
RedisSettings
(Appears on: NativeRedis)
Field | Description |
---|---|
redis string
|
(Optional)
Redis settings shared by both master and slaves, will override the global settings from controller config |
master string
|
(Optional)
Special settings for Redis master node, will override the global settings from controller config |
replica string
|
(Optional)
Special settings for Redis replica nodes, will override the global settings from controller config |
sentinel string
|
(Optional)
Sentinel settings, will override the global settings from controller config |
RedisStreamsSource
(Appears on: Source)
Field | Description |
---|---|
RedisConfig
RedisConfig
|
(Members of RedisConfig contains connectivity info |
stream string
|
|
consumerGroup string
|
|
readFromBeginning bool
|
if true, stream starts being read from the beginning; otherwise, the latest |
tls
TLS
|
(Optional) |
SASL
(Appears on: KafkaSource)
Field | Description |
---|---|
mechanism
SASLType
|
SASL mechanism to use |
gssapi
GSSAPI
|
(Optional)
GSSAPI contains the kerberos config |
plain
SASLPlain
|
(Optional)
SASLPlain contains the sasl plain config |
SASLPlain
(Appears on: SASL)
Field | Description |
---|---|
user string
|
|
passwordSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
PasswordSecret refers to the secret that contains the password |
handshake bool
|
SASLType (string
alias)
(Appears on: SASL)
SASLType describes the SASL type
Scale
(Appears on: AbstractVertex)
Scale defines the parameters for autoscaling.
Field | Description |
---|---|
disabled bool
|
(Optional)
Whether to disable autoscaling. Set to “true” when using Kubernetes HPA or any other 3rd party autoscaling strategies. |
min int32
|
(Optional)
Minimum replicas. |
max int32
|
(Optional)
Maximum replicas. |
lookbackSeconds uint32
|
(Optional)
Lookback seconds to calculate the average pending messages and processing rate. |
cooldownSeconds uint32
|
(Optional)
Cooldown seconds after a scaling operation before another one. |
zeroReplicaSleepSeconds uint32
|
(Optional)
After scaling down to 0, sleep how many seconds before scaling up to peek. |
targetProcessingSeconds uint32
|
(Optional)
TargetProcessingSeconds is used to tune the aggressiveness of autoscaling for source vertices, it measures how fast you want the vertex to process all the pending messages. Typically increasing the value, which leads to lower processing rate, thus less replicas. It’s only effective for source vertices. |
targetBufferUsage uint32
|
(Optional)
TargetBufferUsage is used to define the target percentage of the buffer availability. A valid and meaningful value should be less than the BufferUsageLimit defined in the Edge spec (or Pipeline spec), for example, 50. It only applies to UDF and Sink vertices because only they have buffers to read. Deprecated: use targetBufferAvailability instead. Will be removed in v0.9 |
replicasPerScale uint32
|
(Optional)
ReplicasPerScale defines maximum replicas can be scaled up or down at once. The is use to prevent too aggressive scaling operations |
targetBufferAvailability uint32
|
(Optional)
TargetBufferAvailability is used to define the target percentage of the buffer availability. A valid and meaningful value should be less than the BufferUsageLimit defined in the Edge spec (or Pipeline spec), for example, 50. It only applies to UDF and Sink vertices because only they have buffers to read. |
Sink
(Appears on: AbstractVertex)
Field | Description |
---|---|
log
Log
|
|
kafka
KafkaSink
|
|
blackhole
Blackhole
|
|
udsink
UDSink
|
SlidingWindow
(Appears on: Window)
SlidingWindow describes a sliding window
Field | Description |
---|---|
length
Kubernetes meta/v1.Duration
|
|
slide
Kubernetes meta/v1.Duration
|
Source
(Appears on: AbstractVertex)
Field | Description |
---|---|
generator
GeneratorSource
|
(Optional) |
kafka
KafkaSource
|
(Optional) |
http
HTTPSource
|
(Optional) |
nats
NatsSource
|
(Optional) |
redisStreams
RedisStreamsSource
|
(Optional) |
transformer
UDTransformer
|
(Optional) |
Status
(Appears on: InterStepBufferServiceStatus, PipelineStatus)
Status is a common structure which can be used for Status field.
Field | Description |
---|---|
conditions
\[\]Kubernetes meta/v1.Condition
|
(Optional)
Conditions are the latest available observations of a resource’s current state. |
TLS
(Appears on: KafkaSink, KafkaSource, NatsSource, RedisStreamsSource)
Field | Description |
---|---|
insecureSkipVerify bool
|
(Optional) |
caCertSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
CACertSecret refers to the secret that contains the CA cert |
clientCertSecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
CertSecret refers to the secret that contains the cert |
clientKeySecret
Kubernetes core/v1.SecretKeySelector
|
(Optional)
KeySecret refers to the secret that contains the key |
Templates
(Appears on: PipelineSpec)
Field | Description |
---|---|
daemon
DaemonTemplate
|
(Optional)
DaemonTemplate is used to customize the Daemon Deployment |
job
JobTemplate
|
(Optional)
JobTemplate is used to customize Jobs |
Transformer
(Appears on: UDTransformer)
Field | Description |
---|---|
name string
|
|
args \[\]string
|
(Optional) |
kwargs map\[string\]string
|
(Optional) |
UDF
(Appears on: AbstractVertex)
Field | Description |
---|---|
container
Container
|
(Optional) |
builtin
Function
|
(Optional) |
groupBy
GroupBy
|
(Optional) |
UDSink
(Appears on: Sink)
Field | Description |
---|---|
container
Container
|
UDTransformer
(Appears on: Source)
Field | Description |
---|---|
container
Container
|
(Optional) |
builtin
Transformer
|
(Optional) |
Vertex
(Appears on: VertexInstance)
Field | Description | ||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
metadata
Kubernetes meta/v1.ObjectMeta
|
Refer to the Kubernetes API documentation for the fields of the
metadata field.
|
||||||||||||||
spec
VertexSpec
|
|
||||||||||||||
status
VertexStatus
|
(Optional) |
VertexInstance
VertexInstance is a wrapper of a vertex instance, which contains the vertex spec and the instance information such as hostname and replica index.
Field | Description |
---|---|
vertex
Vertex
|
|
hostname string
|
|
replica int32
|
VertexLimits
(Appears on: AbstractVertex)
Field | Description |
---|---|
readBatchSize uint64
|
(Optional)
Read batch size from the source or buffer. It overrides the settings from pipeline limits. |
readTimeout
Kubernetes meta/v1.Duration
|
(Optional)
Read timeout duration from the source or buffer It overrides the settings from pipeline limits. |
VertexPhase (string
alias)
(Appears on: VertexStatus)
VertexSpec
(Appears on: Vertex)
Field | Description |
---|---|
AbstractVertex
AbstractVertex
|
(Members of |
pipelineName string
|
|
interStepBufferServiceName string
|
(Optional) |
replicas int32
|
(Optional) |
fromEdges
\[\]Edge
|
(Optional) |
toEdges
\[\]Edge
|
(Optional) |
watermark
Watermark
|
(Optional)
Watermark indicates watermark progression in the vertex, it’s populated from the pipeline watermark settings. |
VertexStatus
(Appears on: Vertex)
Field | Description |
---|---|
phase
VertexPhase
|
|
reason string
|
|
message string
|
|
replicas uint32
|
|
selector string
|
|
lastScaledAt
Kubernetes meta/v1.Time
|
VertexType (string
alias)
Watermark
(Appears on: PipelineSpec, VertexSpec)
Field | Description |
---|---|
disabled bool
|
(Optional)
Disabled toggles the watermark propagation, defaults to false. |
maxDelay
Kubernetes meta/v1.Duration
|
(Optional)
Maximum delay allowed for watermark calculation, defaults to “0s”, which means no delay. |
Window
(Appears on: GroupBy)
Window describes windowing strategy
Field | Description |
---|---|
fixed
FixedWindow
|
(Optional) |
sliding
SlidingWindow
|
(Optional) |
Generated with gen-crd-api-reference-docs
.