Drop message onFull¶
We need to have an edge level setting to drop the messages if the
buffer.isFull == true. Even if the UDF or UDSink drops
a message due to some internal error in the user-defined code, the processing latency will spike up causing a natural
back pressure. A kill switch to drop messages can help alleviate/avoid any repercussions on the rest of the DAG.
This setting is an edge-level setting and can be enabled by
onFull and the default is
retryUntilSuccess (other option
This is a data loss scenario but can be useful in cases where we are doing user-introduced experimentations, like A/B testing, on the pipeline. It is totally okay for the experimentation side of the DAG to have data loss while the production is unaffected.
discardLatest will drop the message on the floor if the edge is full.
- from: a
The default setting for
retryUntilSuccess which will make sure the message is retried until successful.
- from: a