Scaleway Object Storage
Synopsis
Creates a target that writes log messages to Scaleway Object Storage with support for various file formats and authentication methods. The target handles large file uploads efficiently with configurable rotation based on size or event count. Scaleway Object Storage provides European-based, GDPR-compliant cloud storage with multi-region availability and competitive pricing.
Schema
- name: <string>
description: <string>
type: scaleways3
pipelines: <pipeline[]>
status: <boolean>
properties:
key: <string>
secret: <string>
region: <string>
endpoint: <string>
part_size: <numeric>
bucket: <string>
buckets:
- bucket: <string>
name: <string>
format: <string>
compression: <string>
extension: <string>
schema: <string>
name: <string>
format: <string>
compression: <string>
extension: <string>
schema: <string>
max_size: <numeric>
batch_size: <numeric>
timeout: <numeric>
field_format: <string>
interval: <string|numeric>
cron: <string>
debug:
status: <boolean>
dont_send_logs: <boolean>
Configuration
The following fields are used to define the target:
| Field | Required | Default | Description |
|---|---|---|---|
name | Y | Target name | |
description | N | - | Optional description |
type | Y | Must be scaleways3 | |
pipelines | N | - | Optional post-processor pipelines |
status | N | true | Enable/disable the target |
Scaleway Object Storage Credentials
| Field | Required | Default | Description |
|---|---|---|---|
key | Y | - | Scaleway Object Storage access key |
secret | Y | - | Scaleway Object Storage secret key |
region | Y | - | Scaleway region (e.g., fr-par, nl-ams, pl-waw) |
endpoint | Y | - | Scaleway Object Storage endpoint (format: https://s3.<region>.scw.cloud) |
Connection
| Field | Required | Default | Description |
|---|---|---|---|
part_size | N | 5 | Multipart upload part size in megabytes (minimum 5MB) |
timeout | N | 30 | Connection timeout in seconds |
field_format | N | - | Data normalization format. See applicable Normalization section |
Files
| Field | Required | Default | Description |
|---|---|---|---|
bucket | N* | - | Default Scaleway bucket name (used if buckets not specified) |
buckets | N* | - | Array of bucket configurations for file distribution |
buckets.bucket | Y | - | Scaleway bucket name |
buckets.name | Y | - | File name template |
buckets.format | N | "json" | Output format: json, multijson, avro, parquet |
buckets.compression | N | - | Compression algorithm. See Compression below |
buckets.extension | N | Matches format | File extension override |
buckets.schema | N* | - | Schema definition file path (required for Avro and Parquet formats) |
name | N | "vmetric.{{.Timestamp}}.{{.Extension}}" | Default file name template when buckets not used |
format | N | "json" | Default output format when buckets not used |
compression | N | - | Default compression when buckets not used |
extension | N | Matches format | Default file extension when buckets not used |
schema | N | - | Default schema path when buckets not used |
max_size | N | 0 | Maximum file size in bytes before rotation |
batch_size | N | 100000 | Maximum number of messages per file |
* = Either bucket or buckets must be specified. When using buckets, schema is conditionally required for Avro and Parquet formats.
When max_size is reached, the current file is uploaded to Scaleway Object Storage and a new file is created. For unlimited file size, set the field to 0.
Scheduler
| Field | Required | Default | Description |
|---|---|---|---|
interval | N | realtime | Execution frequency. See Interval for details |
cron | N | - | Cron expression for scheduled execution. See Cron for details |
Debug Options
| Field | Required | Default | Description |
|---|---|---|---|
debug.status | N | false | Enable debug logging |
debug.dont_send_logs | N | false | Process logs but don't send to target (testing) |
Details
The Scaleway Object Storage target provides European-focused cloud storage integration with comprehensive file format support. Scaleway's infrastructure is entirely based in Europe, making it an ideal choice for organizations requiring GDPR compliance and data sovereignty.
Authentication
Requires Scaleway Object Storage API credentials. API keys can be generated through the Scaleway Console under Credentials. Keys can be scoped with specific permissions and access policies for security.
Endpoint Configuration
The endpoint URL follows the pattern https://s3.<region>.scw.cloud where <region> is your chosen Scaleway region identifier. Each bucket is created in a specific region.
Available Regions
Scaleway Object Storage is available in the following European regions:
| Region Code | Location |
|---|---|
fr-par | France (Paris) |
nl-ams | Netherlands (Amsterdam) |
pl-waw | Poland (Warsaw) |
File Formats
| Format | Description |
|---|---|
json | Each log entry is written as a separate JSON line (JSONL format) |
multijson | All log entries are written as a single JSON array |
avro | Apache Avro format with schema |
parquet | Apache Parquet columnar format with schema |
Compression
Some formats support built-in compression to reduce storage costs and transfer times. When supported, compression is applied at the file/block level before upload.
| Format | Default | Compression Codecs |
|---|---|---|
| JSON | - | Not supported |
| MultiJSON | - | Not supported |
| Avro | zstd | deflate, snappy, zstd |
| Parquet | zstd | gzip, snappy, zstd, brotli, lz4 |
File Management
Files are rotated based on size (max_size parameter) or event count (batch_size parameter), whichever limit is reached first. Template variables in file names enable dynamic file naming for time-based partitioning.
Templates
The following template variables can be used in file names:
| Variable | Description | Example |
|---|---|---|
{{.Year}} | Current year | 2024 |
{{.Month}} | Current month | 01 |
{{.Day}} | Current day | 15 |
{{.Timestamp}} | Current timestamp in nanoseconds | 1703688533123456789 |
{{.Format}} | File format | json |
{{.Extension}} | File extension | json |
{{.Compression}} | Compression type | zstd |
{{.TargetName}} | Target name | my_logs |
{{.TargetType}} | Target type | scaleways3 |
{{.Table}} | Bucket name | logs |
Multipart Upload
Large files automatically use multipart upload protocol with configurable part size (part_size parameter). Default 5MB part size balances upload efficiency and memory usage.
Multiple Buckets
Single target can write to multiple Scaleway buckets with different configurations, enabling data distribution strategies (e.g., raw data to one bucket, processed data to another).
Schema Requirements
Avro and Parquet formats require schema definition files. Schema files must be accessible at the path specified in the schema parameter during target initialization.
GDPR Compliance
Scaleway's European infrastructure ensures full GDPR compliance with data residency guarantees. All data centers are located within the European Union, providing strong data sovereignty protections.
Storage Classes
Scaleway Object Storage offers Standard and Glacier storage classes for different access patterns and retention requirements, enabling cost optimization.
Multi-Region Capabilities
Deploy data across multiple European regions for enhanced availability and disaster recovery without leaving the European jurisdiction.
Examples
Basic Configuration
The minimum configuration for a JSON Scaleway target:
targets:
- name: basic_scaleway
type: scaleways3
properties:
key: "SCW1A2B3C4D5E6F7G8H9"
secret: "a1b2c3d4-e5f6-7890-a1b2-c3d4e5f6g7h8"
region: "fr-par"
endpoint: "https://s3.fr-par.scw.cloud"
bucket: "datastream-logs"
Multiple Buckets
Configuration for distributing data across multiple Scaleway buckets with different formats:
targets:
- name: multi_bucket_export
type: scaleways3
properties:
key: "SCW1A2B3C4D5E6F7G8H9"
secret: "a1b2c3d4-e5f6-7890-a1b2-c3d4e5f6g7h8"
region: "nl-ams"
endpoint: "https://s3.nl-ams.scw.cloud"
buckets:
- bucket: "raw-data-archive"
name: "raw-{{.Year}}-{{.Month}}-{{.Day}}.json"
format: "multijson"
compression: "gzip"
- bucket: "analytics-data"
name: "analytics-{{.Year}}/{{.Month}}/{{.Day}}/data_{{.Timestamp}}.parquet"
format: "parquet"
schema: "<schema definition>"
compression: "snappy"
Parquet Format
Configuration for daily partitioned Parquet files:
targets:
- name: parquet_analytics
type: scaleways3
properties:
key: "SCW1A2B3C4D5E6F7G8H9"
secret: "a1b2c3d4-e5f6-7890-a1b2-c3d4e5f6g7h8"
region: "pl-waw"
endpoint: "https://s3.pl-waw.scw.cloud"
bucket: "analytics-lake"
name: "events/year={{.Year}}/month={{.Month}}/day={{.Day}}/part-{{.Timestamp}}.parquet"
format: "parquet"
schema: "<schema definition>"
compression: "snappy"
max_size: 536870912
High Reliability
Configuration with enhanced settings:
targets:
- name: reliable_scaleway
type: scaleways3
pipelines:
- checkpoint
properties:
key: "SCW1A2B3C4D5E6F7G8H9"
secret: "a1b2c3d4-e5f6-7890-a1b2-c3d4e5f6g7h8"
region: "fr-par"
endpoint: "https://s3.fr-par.scw.cloud"
bucket: "critical-logs"
name: "logs-{{.Timestamp}}.json"
format: "json"
timeout: 60
part_size: 10
With Field Normalization
Using field normalization for standard format:
targets:
- name: normalized_scaleway
type: scaleways3
properties:
key: "SCW1A2B3C4D5E6F7G8H9"
secret: "a1b2c3d4-e5f6-7890-a1b2-c3d4e5f6g7h8"
region: "nl-ams"
endpoint: "https://s3.nl-ams.scw.cloud"
bucket: "normalized-logs"
name: "logs-{{.Timestamp}}.json"
format: "json"
field_format: "cim"
Debug Configuration
Configuration with debugging enabled:
targets:
- name: debug_scaleway
type: scaleways3
properties:
key: "SCW1A2B3C4D5E6F7G8H9"
secret: "a1b2c3d4-e5f6-7890-a1b2-c3d4e5f6g7h8"
region: "fr-par"
endpoint: "https://s3.fr-par.scw.cloud"
bucket: "test-logs"
name: "test-{{.Timestamp}}.json"
format: "json"
debug:
status: true
dont_send_logs: true
GDPR-Compliant Archive
Configuration optimized for GDPR-compliant long-term storage:
targets:
- name: gdpr_archive
type: scaleways3
properties:
key: "SCW1A2B3C4D5E6F7G8H9"
secret: "a1b2c3d4-e5f6-7890-a1b2-c3d4e5f6g7h8"
region: "fr-par"
endpoint: "https://s3.fr-par.scw.cloud"
bucket: "compliance-archive"
name: "archive/{{.Year}}/{{.Month}}/logs-{{.Day}}.json"
format: "json"
compression: "zstd"
max_size: 1073741824