User Defined Sources¶
Pipeline may have multiple Sources, those sources could either be a pre-defined source such as
http, etc., or a
User Defined Source.
With no source data transformer, A pre-defined source vertex runs single-container pods; a user-defined source runs two-container pods.
Build Your Own User Defined Sources¶
You can build your own user defined sources in multiple languages.
Check the links below to see the examples for different languages.
After building a docker image for the written user-defined source, specify the image as below in the vertex spec.
- name: input
Available Environment Variables¶
Some environment variables are available in the user defined source container:
NUMAFLOW_POD- Pod name.
NUMAFLOW_REPLICA- Replica index.
NUMAFLOW_PIPELINE_NAME- Name of the pipeline.
NUMAFLOW_VERTEX_NAME- Name of the vertex.