Amazon SQS

Amazon AWS Message Queue

Synopsis

Creates a target that writes log messages to Amazon Simple Queue Service (SQS) with support for batching and AWS authentication. The target handles message delivery efficiently with configurable batch limits. Amazon SQS is a fully managed message queuing service that enables decoupling and scaling of distributed systems and serverless applications.

Schema

- name: <string>
  description: <string>
  type: amazonsqs
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    key: <string>
    secret: <string>
    session: <string>
    region: <string>
    endpoint: <string>
    queue: <string>
    queue_url: <string>
    max_events: <numeric>
    timeout: <numeric>
    field_format: <string>
    interval: <string|numeric>
    cron: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be amazonsqs
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

AWS Credentials

Field	Required	Default	Description
key	N*	-	AWS access key ID for authentication
secret	N*	-	AWS secret access key for authentication
session	N	-	Optional session token for temporary credentials
region	Y	-	AWS region (e.g., us-east-1, eu-west-1)
endpoint	N	-	Custom SQS endpoint URL (for testing or local development)

* = Conditionally required. AWS credentials (key and secret) are required unless using IAM role-based authentication on AWS infrastructure.

Queue Configuration

Field	Required	Default	Description
queue	N*	-	SQS queue name (will be resolved to queue URL)
queue_url	N*	-	Direct SQS queue URL
max_events	N	10	Maximum number of events per batch (1-10)
timeout	N	30	Connection timeout in seconds
field_format	N	-	Data normalization format. See applicable Normalization section

* = Either queue or queue_url must be specified. Using queue_url is more efficient as it skips the queue name resolution step.

note

Amazon SQS supports a maximum of 10 messages per SendMessageBatch request. The max_events parameter must be between 1 and 10.

Scheduler

Field	Required	Default	Description
interval	N	realtime	Execution frequency. See Interval for details
cron	N	-	Cron expression for scheduled execution. See Cron for details

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

Amazon SQS is a fully managed message queuing service that enables asynchronous communication between distributed system components. This target allows you to send log messages to SQS queues for processing by downstream applications.

Authentication Methods

Supports static credentials (access key and secret key) with optional session tokens for temporary credentials. When deployed on AWS infrastructure, can leverage IAM role-based authentication without explicit credentials.

Queue Types

Amazon SQS supports two types of queues:

Standard Queues

• Nearly unlimited throughput
• At-least-once delivery
• Best-effort ordering
• Default and most common type

FIFO Queues

• Exactly-once processing
• First-in-first-out delivery
• Limited to 300 messages per second (3000 with batching)
• Queue name must end with .fifo

Message Properties

SQS messages have the following characteristics:

• Maximum message size: 256 KB
• Message retention: 1 minute to 14 days (default 4 days)
• Delivery delay: 0 seconds to 15 minutes
• Visibility timeout: 0 seconds to 12 hours

Batch Processing

The target accumulates messages in memory and sends them in batches using the SendMessageBatch API. Batches are sent when the event count limit (max_events) is reached or during finalization. The maximum batch size is 10 messages per request (AWS SQS limit).

Message Deduplication

For FIFO queues, SQS uses message deduplication to prevent duplicate messages. Messages with the same deduplication ID within a 5-minute interval are treated as duplicates and only one copy is delivered.

Dead Letter Queues

SQS supports dead letter queues (DLQ) for handling messages that cannot be processed successfully. Configure DLQ settings in the AWS Console or via infrastructure as code.

Encryption

SQS supports server-side encryption using AWS KMS. Messages are encrypted at rest. All connections to SQS use HTTPS endpoints for encryption in transit.

Integration with AWS Services

SQS integrates with other AWS services:

• AWS Lambda for serverless message processing
• Amazon EC2 and ECS for consumer applications
• Amazon CloudWatch for monitoring and alarms
• AWS Step Functions for workflow orchestration
• Amazon EventBridge for event-driven architectures

Examples

Basic Configuration with Queue Name

The minimum configuration using queue name:

targets:
  - name: basic_sqs
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue: "application-logs"

With Queue URL

Configuration using direct queue URL (more efficient):

targets:
  - name: sqs_with_url
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/application-logs"

With IAM Role

Configuration using IAM role authentication (no explicit credentials):

targets:
  - name: iam_sqs
    type: amazonsqs
    properties:
      region: "us-east-1"
      queue: "application-logs"

note

When using IAM role authentication, ensure the EC2 instance, ECS task, or Lambda function has an IAM role with appropriate SQS permissions attached.

FIFO Queue

Configuration for FIFO queue:

targets:
  - name: fifo_sqs
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue: "application-logs.fifo"

High Throughput

Configuration optimized for high-volume data:

targets:
  - name: high_volume_sqs
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/high-volume-logs"
      max_events: 10
      timeout: 60

With Temporary Credentials

Configuration using temporary session credentials:

targets:
  - name: temp_creds_sqs
    type: amazonsqs
    properties:
      key: "ASIATEMP1234567890AB"
      secret: "tempSecretKeyExample1234567890"
      session: "FwoGZXIvYXdzEBYaDH...temporary-session-token"
      region: "us-west-2"
      queue: "temporary-logs"

With Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_sqs
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue: "normalized-logs"
      field_format: "cim"

With Checkpoint Pipeline

Configuration with checkpoint pipeline for reliability:

targets:
  - name: reliable_sqs
    type: amazonsqs
    pipelines:
      - checkpoint
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue: "critical-logs"
      max_events: 5

Multiple Regions

Configuration for SQS queue in different region:

targets:
  - name: eu_sqs
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "eu-west-1"
      queue: "eu-application-logs"

Scheduled Batching

Configuration with scheduled batch delivery:

targets:
  - name: scheduled_sqs
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue: "scheduled-logs"
      max_events: 10
      interval: "5m"

Debug Configuration

Configuration with debugging enabled:

targets:
  - name: debug_sqs
    type: amazonsqs
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue: "test-logs"
      debug:
        status: true
        dont_send_logs: true

Local Development

Configuration with custom endpoint for local testing (e.g., LocalStack, ElasticMQ):

targets:
  - name: local_sqs
    type: amazonsqs
    properties:
      key: "test"
      secret: "test"
      region: "us-east-1"
      endpoint: "http://localhost:4566"
      queue: "local-test-queue"

Production Configuration

Configuration for production with optimal settings:

targets:
  - name: production_sqs
    type: amazonsqs
    pipelines:
      - checkpoint
    properties:
      key: "AKIAIOSFODNN7EXAMPLE"
      secret: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
      region: "us-east-1"
      queue_url: "https://sqs.us-east-1.amazonaws.com/123456789012/production-logs"
      max_events: 10
      timeout: 60
      field_format: "cim"