Skip to main content

Documentation Index

Fetch the complete documentation index at: https://goldrush.dev/docs/llms.txt

Use this file to discover all available pages before exploring further.

The AWS SQS destination delivers structured pipeline output as messages to an Amazon SQS queue. It supports both standard and FIFO queues with built-in deduplication.

​
Configuration

destination:
  type: "sqs"
  queue_url: "https://sqs.us-east-1.amazonaws.com/123456789/my-queue.fifo"
  region: "us-east-1"
  access_key_id: "${AWS_ACCESS_KEY_ID}"
  secret_access_key: "${AWS_SECRET_ACCESS_KEY}"
  batch_size: 10
  message_group_id: ""

​
Fields

FieldTypeRequiredDefaultDescription
queue_urlstringyes—SQS queue URL
regionstringyes—AWS region
access_key_idstringyes—AWS access key
secret_access_keystringyes—AWS secret key
batch_sizeintno10Messages per batch (max 10)
message_group_idstringnoauto-derivedFIFO message group ID

​
FIFO Queue Detection

FIFO mode is automatically activated when either condition is met:
  • The queue_url ends with .fifo
  • A message_group_id value is explicitly set
When FIFO mode is active, the destination sets the MessageGroupId and MessageDeduplicationId on every message.

​
Message Format

Each message is a JSON object containing the project, stream name, and the record payload: 
{
  "project": "analytics-prod",
  "stream": "hl_fills",
  "record": {
    "block_number": 12345,
    "...": "..."
  }
}

​
Message Attributes

Each SQS message includes the following message attributes:
AttributeTypeDescription
projectStringProject name
streamStringStream name
block_numberStringBlock number, if present in the record

​
Deduplication

For FIFO queues, the deduplication ID is deterministically derived from source metadata using the format: 
{topic}-{partition}-{offset}
This ensures that retried deliveries do not produce duplicate messages in the queue.