External Storage - TypeScript SDK

Release, stability, and dependency info

External Storage is in Public Preview. APIs and configuration may change before General Availability. Join the #large-payloads Slack channel to provide feedback or ask for help.

The Temporal Service enforces a 2 MB per-payload limit by default. This limit is configurable on self-hosted deployments. When your Workflows or Activities handle data larger than the limit, you can offload payloads to external storage, such as Amazon S3, and pass a small reference token through the Event History instead. This page shows you how to set up External Storage with Amazon S3 and how to implement a custom storage driver.

For a conceptual overview of External Storage and its use cases, see External Storage.

Store and retrieve large payloads with Amazon S3

The TypeScript SDK includes an S3 storage driver. Follow these steps to set it up:

Prerequisites

An Amazon S3 bucket that you have read and write access to. Refer to lifecycle management to ensure that your payloads remain available for the entire lifetime of the Workflow. For multi-region durability, see Durable External Storage.

Install the S3 driver package and its dependencies.

# TODO(external-storage): confirm the S3 driver package name.
npm install @temporalio/external-storage-s3 @aws-sdk/client-s3

Procedure

Create an S3 client and pass it to the S3 storage driver. The driver uses your standard AWS credentials from the environment (environment variables, IAM role, or AWS config file):

// TODO(external-storage): confirm the S3 driver import path and constructor options.
import { S3StorageDriver } from '@temporalio/external-storage-s3';
import { S3Client } from '@aws-sdk/client-s3';

const s3Client = new S3Client({ region: 'us-east-2' });

const driver = new S3StorageDriver({
  client: s3Client,
  bucket: 'my-temporal-payloads',
});

Configure the driver on your Data Converter and pass the converter to your Client and Worker. External Storage runs outside the Workflow sandbox, so you can pass the driver object directly on the dataConverter option (unlike a custom Payload Converter, which must be referenced by path):

// TODO(external-storage): confirm the externalStorage field name and shape on the Data Converter.
import { Client, Connection } from '@temporalio/client';
import { Worker } from '@temporalio/worker';

const dataConverter = {
  externalStorage: {
    drivers: [driver],
  },
};

const connection = await Connection.connect();
const client = new Client({ connection, dataConverter });

const worker = await Worker.create({
  workflowsPath: require.resolve('./workflows'),
  taskQueue: 'my-task-queue',
  dataConverter,
});

By default, payloads larger than 256 KiB are offloaded to external storage. You can adjust this with the payloadSizeThreshold option, even setting it to 0 to externalize all payloads regardless of size. Refer to Configure payload size threshold for more information.

All Workflows and Activities running on the Worker use the storage driver automatically without changes to your business logic. The driver uploads and downloads payloads concurrently and validates payload integrity on retrieve.

The S3 driver includes diagnostic metadata, such as the AWS region, in error messages to help troubleshoot storage failures.

Implement a custom storage driver

If you need a storage backend other than what the built-in drivers allow, you can implement your own storage driver. Refer to Choose a storage system for guidance on selecting a backing store and Lifecycle management for retention requirements.

The following example shows a custom driver that uses local disk as the backing store. This example is for local development and testing only. In production, use a durable storage system that is accessible to all Workers.

// TODO(external-storage): confirm import paths and exact exported names for StorageDriver, StorageDriverClaim,
// StorageDriverStoreContext, and StorageDriverRetrieveContext.
import { Payload } from '@temporalio/common';
import { temporal } from '@temporalio/proto';
import {
  StorageDriver,
  StorageDriverClaim,
  StorageDriverStoreContext,
  StorageDriverRetrieveContext,
} from '@temporalio/external-storage';
import { randomUUID } from 'node:crypto';
import { mkdir, readFile, writeFile } from 'node:fs/promises';
import path from 'node:path';

export class LocalDiskStorageDriver implements StorageDriver {
  constructor(private readonly storeDir = '/tmp/temporal-payload-store') {}

  name(): string {
    return 'local-disk';
  }

  async store(context: StorageDriverStoreContext, payloads: Payload[]): Promise<StorageDriverClaim[]> {
    // TODO(external-storage): confirm the shape of context.target (Workflow vs Activity info).
    let prefix = this.storeDir;
    const target = context.target;
    if (target?.workflowId) {
      prefix = path.join(this.storeDir, target.namespace, target.workflowId);
    }
    await mkdir(prefix, { recursive: true });

    const claims: StorageDriverClaim[] = [];
    for (const payload of payloads) {
      const key = `${randomUUID()}.bin`;
      const filePath = path.join(prefix, key);
      const bytes = temporal.api.common.v1.Payload.encode(payload).finish();
      await writeFile(filePath, bytes);
      claims.push({ claimData: { path: filePath } });
    }
    return claims;
  }

  async retrieve(context: StorageDriverRetrieveContext, claims: StorageDriverClaim[]): Promise<Payload[]> {
    const payloads: Payload[] = [];
    for (const claim of claims) {
      const bytes = await readFile(claim.claimData.path);
      payloads.push(temporal.api.common.v1.Payload.decode(bytes));
    }
    return payloads;
  }
}

The following sections walk through the key parts of the driver implementation.

1. Implement the StorageDriver interface

A custom driver implements the StorageDriver interface with three methods:

name() returns a unique string that identifies the driver. The SDK stores this name in the claim check reference so it can route retrieval requests to the correct driver. Changing the name after payloads have been stored breaks retrieval.
store() receives an array of payloads and returns one StorageDriverClaim per payload. A claim is a set of string key-value pairs that the driver uses to locate the payload later.
retrieve() receives the claims that store() produced and returns the original payloads.

2. Store payloads

In store(), serialize each Payload protobuf message to bytes and write the bytes to your storage system. The application data has already been serialized by the Payload Converter and Payload Codec before it reaches the driver. See the data conversion pipeline for more details.

Return a StorageDriverClaim for each payload with enough information to retrieve it later. The context.target provides identity information (namespace, Workflow ID, or Activity ID) depending on the operation. Consider structuring your storage keys to include this information so that you can identify which Workflow owns each payload. Within that scope, content-addressable keys (such as a SHA-256 hash of the payload bytes) can help deduplicate identical payloads.

3. Retrieve payloads

In retrieve(), download the bytes using the claim data, then reconstruct the Payload protobuf message. The Payload Converter handles deserializing the application data after the driver returns the payload.

4. Configure the Data Converter

Pass your driver on the Data Converter's External Storage configuration and use the converter when creating your Client and Worker. You can also package your driver as a plugin for easier reuse across services:

// TODO(external-storage): confirm the externalStorage field name and shape on the Data Converter.
const dataConverter = {
  externalStorage: {
    drivers: [new LocalDiskStorageDriver()],
  },
};

Configure payload size threshold

You can configure the payload size threshold that triggers external storage. By default, payloads larger than 256 KiB are offloaded to external storage. You can adjust this with the payloadSizeThreshold option, or set it to 0 to externalize all payloads regardless of size.

const dataConverter = {
  externalStorage: {
    drivers: [driver],
    payloadSizeThreshold: 0,
  },
};

Use multiple storage drivers

When you register multiple drivers, you must provide a driverSelector function that chooses which driver stores each payload. Any driver in the list that is not selected for storing is still available for retrieval, which is useful when migrating between storage backends. Return a nullish value from the selector to keep a specific payload inline in Event History.

Multiple drivers are useful in scenarios such as:

Driver migration. Your Worker needs to retrieve payloads created by clients that use a different driver than the one you prefer. Register both drivers and use the selector to always pick your preferred driver for new payloads. The old driver remains available for retrieving existing claims.
Multi-cloud storage. Route payloads to different storage backends based on your cloud environment. For example, use S3 for Workers running on AWS and GCS for Workers running on Google Cloud. The selector chooses the appropriate driver based on the runtime environment.

The following example registers two drivers but always selects preferredDriver for new payloads. The legacyDriver is only registered so the Worker can retrieve payloads that were previously stored with it:

// TODO(external-storage): confirm the selector signature and return-value semantics.
const preferredDriver = new S3StorageDriver({ client: s3Client, bucket: 'my-bucket' });
const legacyDriver = new LegacyStorageDriver();

const externalStorage = {
  drivers: [preferredDriver, legacyDriver],
  driverSelector: (context, payload) => preferredDriver,
};

Multi-region durability

To make your S3-backed External Storage tolerant of regional failures, configure the AWS side with Cross-Region Replication (CRR) and an S3 Multi-Region Access Point (MRAP), then point the driver at the MRAP ARN instead of a bucket name. See Durable External Storage for the full pattern and trade-offs.

In code, the only change is the value you pass as bucket:

const driver = new S3StorageDriver({
  client: s3Client,
  bucket: 'arn:aws:s3::123456789012:accesspoint/mfzwi23gnjvgw.mrap',
});

The AWS SDK for JavaScript automatically uses SigV4A signing when the bucket value is an MRAP ARN, so no additional client configuration is required.

Store and retrieve large payloads with Amazon S3​

Prerequisites​

Procedure​

Implement a custom storage driver​

1. Implement the StorageDriver interface​

2. Store payloads​

3. Retrieve payloads​

4. Configure the Data Converter​

Configure payload size threshold​

Use multiple storage drivers​

Multi-region durability​