Skip to main content

Documentation Index

Fetch the complete documentation index at: https://goldrush.dev/docs/llms.txt

Use this file to discover all available pages before exploring further.

The Kafka destination forwards raw bytes from the pipeline source directly to a target Kafka topic. Unlike other destinations, Kafka operates in Raw pipeline mode - no normalization, ABI decoding, or transforms are applied.

​
Configuration

destination:
  type: "kafka"
  bootstrap_servers: "broker1:9092,broker2:9092"
  sasl:
    mechanism: "SCRAM-SHA-512"
    username: "${KAFKA_USER}"
    password: "${KAFKA_PASSWORD}"
  output_topic: "my-output-topic"

​
Fields

FieldTypeRequiredDescription
bootstrap_serversstringyesTarget Kafka broker addresses
sasl.mechanismstringyesSASL mechanism (e.g., SCRAM-SHA-512)
sasl.usernamestringyesSASL username
sasl.passwordstringyesSASL password
output_topicstringyesTarget topic name

​
Key Behavior

  • Raw passthrough - bytes from the source are forwarded without any processing. No normalization, ABI decoding, or transform stages run.
  • Exactly-once delivery - the destination uses transactional producing to guarantee exactly-once semantics end to end.
  • Validation at startup - if your pipeline configuration includes abi or transforms sections alongside a Kafka destination, the pipeline will reject the configuration and fail to start.
ABI decoding and transform configurations are not compatible with the Kafka destination. Remove those sections from your pipeline definition before deploying.