Oracle Cloud Object Storage

Oracle Cloud Long Term Storage

Synopsis

Creates a target that writes log messages to Oracle Cloud Infrastructure (OCI) Object Storage with support for various file formats and authentication methods. The target handles large file uploads efficiently with configurable rotation based on size or event count. OCI Object Storage provides enterprise-grade durability, security, and performance with strong integration into Oracle's cloud ecosystem.

Schema

- name: <string>
  description: <string>
  type: oracles3
  pipelines: <pipeline[]>
  status: <boolean>
  properties:
    key: <string>
    secret: <string>
    region: <string>
    endpoint: <string>
    part_size: <numeric>
    bucket: <string>
    buckets:
      - bucket: <string>
        name: <string>
        format: <string>
        compression: <string>
        extension: <string>
        schema: <string>
    name: <string>
    format: <string>
    compression: <string>
    extension: <string>
    schema: <string>
    max_size: <numeric>
    batch_size: <numeric>
    timeout: <numeric>
    field_format: <string>
    interval: <string|numeric>
    cron: <string>
    debug:
      status: <boolean>
      dont_send_logs: <boolean>

Configuration

The following fields are used to define the target:

Field	Required	Default	Description
name	Y		Target name
description	N	-	Optional description
type	Y		Must be oracles3
pipelines	N	-	Optional post-processor pipelines
status	N	true	Enable/disable the target

Oracle Cloud Object Storage Credentials

Field	Required	Default	Description
key	Y	-	Oracle Cloud access key ID for Customer Secret Keys
secret	Y	-	Oracle Cloud secret access key
region	Y	-	OCI region identifier (e.g., us-ashburn-1, eu-frankfurt-1, ap-tokyo-1)
endpoint	Y	-	OCI Object Storage endpoint (format: https://<namespace>.compat.objectstorage.<region>.oraclecloud.com)

Connection

Field	Required	Default	Description
part_size	N	5	Multipart upload part size in megabytes (minimum 5MB)
timeout	N	30	Connection timeout in seconds
field_format	N	-	Data normalization format. See applicable Normalization section

Files

Field	Required	Default	Description
bucket	N*	-	Default OCI bucket name (used if buckets not specified)
buckets	N*	-	Array of bucket configurations for file distribution
buckets.bucket	Y	-	OCI bucket name
buckets.name	Y	-	File name template
buckets.format	N	"json"	Output format: json, multijson, avro, parquet
buckets.compression	N	-	Compression algorithm. See Compression below
buckets.extension	N	Matches format	File extension override
buckets.schema	N*	-	Schema definition file path (required for Avro and Parquet formats)
name	N	"vmetric.{{.Timestamp}}.{{.Extension}}"	Default file name template when buckets not used
format	N	"json"	Default output format when buckets not used
compression	N	-	Default compression when buckets not used
extension	N	Matches format	Default file extension when buckets not used
schema	N	-	Default schema path when buckets not used
max_size	N	0	Maximum file size in bytes before rotation
batch_size	N	100000	Maximum number of messages per file

* = Either bucket or buckets must be specified. When using buckets, schema is conditionally required for Avro and Parquet formats.

note

When max_size is reached, the current file is uploaded to OCI Object Storage and a new file is created. For unlimited file size, set the field to 0.

Scheduler

Field	Required	Default	Description
interval	N	realtime	Execution frequency. See Interval for details
cron	N	-	Cron expression for scheduled execution. See Cron for details

Debug Options

Field	Required	Default	Description
debug.status	N	false	Enable debug logging
debug.dont_send_logs	N	false	Process logs but don't send to target (testing)

Details

The Oracle Cloud Infrastructure Object Storage target provides enterprise-grade cloud storage integration with comprehensive file format support. OCI Object Storage offers strong data durability, automatic encryption, and seamless integration with Oracle Database and analytics services.

Authentication

Requires OCI Customer Secret Keys for S3 compatibility. Customer Secret Keys can be generated through the OCI Console under User Settings. Each user can have up to two active Customer Secret Keys at a time.

Endpoint Configuration

The endpoint URL follows the pattern https://<namespace>.compat.objectstorage.<region>.oraclecloud.com where <namespace> is your OCI Object Storage namespace (typically your tenancy name) and <region> is your chosen OCI region identifier.

Finding Your Namespace

Your Object Storage namespace can be found in the OCI Console under Tenancy Details or by using the OCI CLI command oci os ns get.

Available Regions

Oracle Cloud Infrastructure is available in numerous regions worldwide:

Region Code	Location
us-ashburn-1	US East (Ashburn)
us-phoenix-1	US West (Phoenix)
us-sanjose-1	US West (San Jose)
ca-toronto-1	Canada Southeast (Toronto)
ca-montreal-1	Canada Southeast (Montreal)
eu-frankfurt-1	Germany Central (Frankfurt)
eu-zurich-1	Switzerland North (Zurich)
eu-amsterdam-1	Netherlands Northwest (Amsterdam)
uk-london-1	UK South (London)
ap-tokyo-1	Japan East (Tokyo)
ap-osaka-1	Japan Central (Osaka)
ap-seoul-1	South Korea Central (Seoul)
ap-mumbai-1	India West (Mumbai)
ap-hyderabad-1	India South (Hyderabad)
ap-sydney-1	Australia East (Sydney)
ap-melbourne-1	Australia Southeast (Melbourne)
sa-saopaulo-1	Brazil East (Sao Paulo)
me-jeddah-1	Saudi Arabia West (Jeddah)
me-dubai-1	UAE East (Dubai)

File Formats

Format	Description
json	Each log entry is written as a separate JSON line (JSONL format)
multijson	All log entries are written as a single JSON array
avro	Apache Avro format with schema
parquet	Apache Parquet columnar format with schema

Compression

Some formats support built-in compression to reduce storage costs and transfer times. When supported, compression is applied at the file/block level before upload.

Format	Default	Compression Codecs
JSON	-	Not supported
MultiJSON	-	Not supported
Avro	zstd	deflate, snappy, zstd
Parquet	zstd	gzip, snappy, zstd, brotli, lz4

File Management

Files are rotated based on size (max_size parameter) or event count (batch_size parameter), whichever limit is reached first. Template variables in file names enable dynamic file naming for time-based partitioning.

Templates

The following template variables can be used in file names:

Variable	Description	Example
{{.Year}}	Current year	2024
{{.Month}}	Current month	01
{{.Day}}	Current day	15
{{.Timestamp}}	Current timestamp in nanoseconds	1703688533123456789
{{.Format}}	File format	json
{{.Extension}}	File extension	json
{{.Compression}}	Compression type	zstd
{{.TargetName}}	Target name	my_logs
{{.TargetType}}	Target type	oracles3
{{.Table}}	Bucket name	logs

Multipart Upload

Large files automatically use multipart upload protocol with configurable part size (part_size parameter). Default 5MB part size balances upload efficiency and memory usage.

Multiple Buckets

Single target can write to multiple OCI buckets with different configurations, enabling data distribution strategies (e.g., raw data to one bucket, processed data to another).

Schema Requirements

Avro and Parquet formats require schema definition files. Schema files must be accessible at the path specified in the schema parameter during target initialization.

Storage Tiers

OCI Object Storage supports multiple storage tiers including Standard, Infrequent Access, and Archive for cost optimization based on access patterns and retention requirements.

Integration with Oracle Services

OCI Object Storage integrates seamlessly with Oracle Autonomous Database, Oracle Analytics Cloud, and other Oracle Cloud services for comprehensive data processing pipelines.

Examples

Basic Configuration

The minimum configuration for a JSON OCI Object Storage target:

targets:
  - name: basic_oci
    type: oracles3
    properties:
      key: "0a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p"
      secret: "AbCdEfGhIjKlMnOpQrStUvWxYz0123456789+/=="
      region: "us-ashburn-1"
      endpoint: "https://mytenancy.compat.objectstorage.us-ashburn-1.oraclecloud.com"
      bucket: "datastream-logs"

Multiple Buckets

Configuration for distributing data across multiple OCI buckets with different formats:

targets:
  - name: multi_bucket_export
    type: oracles3
    properties:
      key: "0a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p"
      secret: "AbCdEfGhIjKlMnOpQrStUvWxYz0123456789+/=="
      region: "eu-frankfurt-1"
      endpoint: "https://mytenancy.compat.objectstorage.eu-frankfurt-1.oraclecloud.com"
      buckets:
        - bucket: "raw-data-archive"
          name: "raw-{{.Year}}-{{.Month}}-{{.Day}}.json"
          format: "multijson"
          compression: "gzip"
        - bucket: "analytics-data"
          name: "analytics-{{.Year}}/{{.Month}}/{{.Day}}/data_{{.Timestamp}}.parquet"
          format: "parquet"
          schema: "<schema definition>"
          compression: "snappy"

Parquet Format

Configuration for daily partitioned Parquet files:

targets:
  - name: parquet_analytics
    type: oracles3
    properties:
      key: "0a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p"
      secret: "AbCdEfGhIjKlMnOpQrStUvWxYz0123456789+/=="
      region: "ap-tokyo-1"
      endpoint: "https://mytenancy.compat.objectstorage.ap-tokyo-1.oraclecloud.com"
      bucket: "analytics-lake"
      name: "events/year={{.Year}}/month={{.Month}}/day={{.Day}}/part-{{.Timestamp}}.parquet"
      format: "parquet"
      schema: "<schema definition>"
      compression: "snappy"
      max_size: 536870912

High Reliability

Configuration with enhanced settings:

targets:
  - name: reliable_oci
    type: oracles3
    pipelines:
      - checkpoint
    properties:
      key: "0a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p"
      secret: "AbCdEfGhIjKlMnOpQrStUvWxYz0123456789+/=="
      region: "uk-london-1"
      endpoint: "https://mytenancy.compat.objectstorage.uk-london-1.oraclecloud.com"
      bucket: "critical-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      timeout: 60
      part_size: 10

With Field Normalization

Using field normalization for standard format:

targets:
  - name: normalized_oci
    type: oracles3
    properties:
      key: "0a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p"
      secret: "AbCdEfGhIjKlMnOpQrStUvWxYz0123456789+/=="
      region: "ap-sydney-1"
      endpoint: "https://mytenancy.compat.objectstorage.ap-sydney-1.oraclecloud.com"
      bucket: "normalized-logs"
      name: "logs-{{.Timestamp}}.json"
      format: "json"
      field_format: "cim"

Debug Configuration

Configuration with debugging enabled:

targets:
  - name: debug_oci
    type: oracles3
    properties:
      key: "0a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p"
      secret: "AbCdEfGhIjKlMnOpQrStUvWxYz0123456789+/=="
      region: "us-phoenix-1"
      endpoint: "https://mytenancy.compat.objectstorage.us-phoenix-1.oraclecloud.com"
      bucket: "test-logs"
      name: "test-{{.Timestamp}}.json"
      format: "json"
      debug:
        status: true
        dont_send_logs: true