Skip to content

Development

This doc explains how to set up a development environment for Numaflow.

Install required tools

  1. go 1.24+.
  2. git.
  3. kubectl.
  4. protoc 3.19 for compiling protocol buffers.
  5. pandoc 3.2.1 for generating API markdown.
  6. Node.js® for running the UI.
  7. yarn.
  8. A local Kubernetes cluster for development usage, pick either one of k3d, kind, or minikube.

Example: Create a local Kubernetes cluster with kind

# Install kind on macOS
brew install kind

# Create a cluster with default name kind
kind create cluster

# Get kubeconfig for the cluster
kind export kubeconfig

Metrics Server

Please install the metrics server if your local Kubernetes cluster does not bring it by default (e.g., Kind). Without the metrics-server, we will not be able to see the pods in the UI.

kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl patch -n kube-system deployment metrics-server --type=json -p '[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'

Useful Commands

  • make start Build the source code, image, and install the Numaflow controller in the numaflow-system namespace.

  • make build Binaries are placed in ./dist.

  • make manifests Regenerate all the manifests after making any base manifest changes. This is also covered by make codegen.

  • make codegen Run after making changes to ./pkg/api/.

  • make test Run unit tests.

  • make test-* Run one e2e test suite. e.g. make test-kafka-e2e to run the kafka e2e suite.

  • make Test* Run one e2e test case. e.g. make TestKafkaSourceSink to run the TestKafkaSourceSink case in the kafka e2e suite.

  • make image Build container image, and import it to k3d, kind, or minikube cluster if corresponding KUBECONFIG is sourced.

  • make docs Convert the docs to GitHub pages, check if there's any error.

  • make docs-serve Start an HTTP server on your local to host the docs generated Github pages.