How to set up Redpanda Tiered Storage on S3 with external keys
The best of both worlds to meet even the strictest data compliance requirements
Author: Michael Maltese
Redpanda Tiered Storage is a powerful tool that lets you keep infinite retention on your topics by automatically offloading historical data into the cloud and calling it back as needed. It also offers unique capabilities around disaster recovery and read replicas.
Many of our customers use Redpanda in environments with data compliance requirements, where uploading unencrypted data into the cloud is a security risk. Luckily, most cloud providers offer transparent storage encryption by default, which works seamlessly with Redpanda.
However, some customers have even stricter data compliance requirements, where using cloud-provider-managed keys can be a security risk. These customers need to manage and store encryption keys separately from their cloud provider.
In this post, we’ll cover how you can achieve the best of both worlds: setting up Redpanda Tiered Storage against an Amazon S3 bucket where all data will be encrypted with an external customer-controlled key.
How to set up Redpanda Tiered Storage to an S3 bucket encrypted with an external key
AWS S3 supports server-side encryption of objects via AWS KMS keys, which includes supporting external keys accessible via the AWS KMS External Keystore (XKS) Proxy API.
We’ll first set up an external key store (XKS) in AWS KMS, import our keys, configure the S3 bucket, and finally set up Redpanda Tiered Storage.
1. Configure an External Key Store and add your external keys
First, you’ll need an external key manager that implements the AWS KMS XKS Proxy API. Vendors including Atos, Entrust, Fortanix, HashiCorp, Salesforce, Thales, and T-Systems offer XKS solutions. You can also test using the open-source reference implementation, or write your own according to the protocol specification.
Your XKS proxy must be accessible to AWS KMS, either over the public internet, or via a VPC endpoint service.
The AWS documentation has additional information on considerations for creating and configuring your XKS proxy.
Once you have that set up, you’ll create a new KMS external key store configured to point at your XKS proxy. KMS creates new external key stores as disconnected by default, so you’ll also need to choose to connect it afterward.
Now, you can import an external key by creating a new KMS customer managed key, choosing to use the external key store as the source of key material, and then passing a key ID corresponding to a key known to your XKS proxy.
This will create a KMS customer-managed key that effectively acts as an intermediary between AWS services and your external key manager, calling your proxy API anytime it needs to encrypt or decrypt information. In this way, you retain complete control of the key, and can shut off access to AWS at any time.
Example: using aws-kms-xks-proxy
Let’s walk through an example of creating an external key store and importing an external key into KMS, using the open-source XKS proxy reference implementation.
We’ll download the proxy source, edit the configuration to match our needs, build and run a Docker container with SoftHSMv2 and a key named foo, expose it to the public internet, and then connect to it from AWS KMS.
First, download the source for the proxy from https://github.com/aws-samples/aws-kms-xks-proxy:
git clone https://github.com/aws-samples/aws-kms-xks-proxy
If necessary, edit the configuration file that the Docker image will . At the time of writing, the file is located at xks-axum/configuration/settings_docker.toml in the repository root. For instance, if you’re not using the AWS region us-east-1, you’ll need to edit the region key to match.
Build and run the docker container:
docker build -t xks-proxy:latest .
docker run - rm - name xks-proxy -p 0.0.0.0:9000:80 xks-proxy:latestFor this example, we’ll expose the proxy to AWS KMS over the public internet using ngrok. Ngrok accomplishes two things:
- A public endpoint that proxies to our local port
- TLS termination using a well-known certificate.
If you don’t have ngrok, you can download it here. Then, expose the proxy with: ngrok http 9000
Ngrok will give you a public endpoint that forwards to your local proxy, such as:
https://5fa8-70-19-71-78.ngrok.io -> http://localhost:9000.
This endpoint is what we’ll pass to KMS.
If you’re using the AWS Console, go to AWS KMS, and select the “External key stores” section. Then click “Create external key store” and pass in the configuration information for your XKS proxy:
- “Proxy connectivity” should be set to “Public endpoint”
- “Proxy URI endpoint” should be set to the ngrok endpoint.
- “Proxy URI path prefix” should be set to
/example/uri/path/prefix(fromsettings_docker.toml) - “Proxy credential: Access key ID” should be set to
BETWEEN2TENAND3TENCHARACTERS(fromsettings_docker.toml)
“Proxy credential: Secret access key” should be set to PleaseReplaceThisWithSomeSecretOfLength43To64 (from settings_docker.toml)
Once created, there’s one more step: select the key store, and tell KMS to “Connect.”
Now you can create keys pointing at this external key store. Our XKS proxy currently exposes a single key: foo. Let’s add it to KMS.
Go to the KMS section for customer-managed keys, and create a new key:
- “Key type” must be “Symmetric”
- “Key usage” must be “Encrypt and decrypt”
- “Advanced options: Key material origin” must be “External key store”
On the next page, select which external key store to use, and specify the external ID of the key. The external ID here should be foo, as that is the only key the proxy is currently configured with.
Follow the remainder of the key configuration steps, and you now have an external key available through AWS KMS. You can use this key anywhere inside AWS that supports using a symmetric customer-managed key from KMS.
2. Configure Amazon S3 to encrypt using your external key
Now that you have KMS connected to your external key store, you can configure S3 to do server-side encryption and decryption of your data using your external keys. Server-side encryption is fast, transparent, and needs zero configuration changes to your Redpanda cluster.
Go into your S3 bucket configuration, and change the encryption type to “Server-side encryption with AWS Key Management Service keys (SSE-KMS)” if it isn’t already. S3 buckets default to “Server-side encryption with Amazon S3 managed keys (SSE-S3)”, where your objects are encrypted at rest but using keys controlled by Amazon.
Choose to encrypt the bucket using the KMS key corresponding to your external key. We also recommend enabling Bucket Keys, which reduces calls to AWS KMS and your XKS proxy, saving costs, and decreasing latency. You should judge if this is acceptable depending on your compliance requirements.
3. Enable Redpanda Tiered Storage
Finally, configure Redpanda to enable Tiered Storage to your S3 bucket. Note that the IAM role or access key being used by Redpanda to connect to S3 must have permission to use the KMS key, as well as access the S3 bucket.
That’s it! As the encryption is being handled transparently by S3, Redpanda needs no additional changes. Segment data will now be uploaded to S3 and encrypted using your external key. As Redpanda uploads and downloads segments to cloud storage, you will see your XKS proxy being called.
Additionally, any previously uploaded data will still be available for download (but remain encrypted using the S3 bucket’s previous config), ensuring no interruption of service.
Wrapping up
This post introduced you to AWS KMS external key stores and S3 encryption. You learned how to use KMS and S3 to configure a Redpanda Tiered Storage setup where all data is encrypted with an external customer-controlled key.
Redpanda Tiered Storage can also be used with Google Cloud Storage and Azure Blob Storage / Data Lake Storage. GCS supports a similar solution to the one described in this post using Cloud External Key Manager. Azure does not currently have support for transparent encryption using external keys.
To learn more about Redpanda, browse the Redpanda blog for more tutorials. If you have questions about this post or about Redpanda in general, ask away in the Redpanda Community on Slack.
