How we engineered our CLI to improve developer productivity

5 Ways Redpanda’s rpk command line interface makes it easier to install Redpanda and manage your workflows.

Redpanda Keeper (rpk) is a single binary application written in Go to interact with your Redpanda clusters. It’s designed to be a Swiss Army Knife CLI application with simple commands that make it easy to use. It is handy not only for developers who want to evaluate Redpanda on their local machine, but also for developers who want to build, maintain, and debug a production streaming system using Redpanda.

This blog highlights five key benefits that rpk offers developers, outlined below:

1. Easy installation and quickstart on a local machine
2. Topic, group, and ACL management capabilities
3. Rich formatting for producing and consuming
4. Tuning the host to adopt production workload
5. Easy configuration changes

1. Provides easy installation and quickstart on local machines

As mentioned above, rpk is a single binary application. It is installed with the Redpanda broker, which is contained in all Redpanda packages. It also provides autocompletion, which helps you interact with rpk easily using the tab button. Please review the rpk documentation for the most up-to-date information on using rpk with the latest version of Redpanda.

On Linux

If you want to run rpk in a Linux distribution, there are two ways.

To install rpk with the Redpanda server binary, here’s an example on Ubuntu:

$ curl -1sLf 'https://packages.vectorized.io/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash && \
sudo apt install redpanda -y && \
sudo systemctl start redpanda

To install the rpk binary independently,

• Download the rpk archive on Ubuntu with: curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-linux-amd64.zip
• Unpack the archive to /usr/local/bin or any location in your $PATH with: unzip rpk-linux-amd64.zip

Once it’s completed, run rpk version, which shows the rpk binary version, not the Redpanda server version.

$ rpk version
v22.1.2 (rev 449eef2)

Follow the instructions in our Linux quickstart documentation, which will show you how to install rpk on Fedora/RedHat systems too.

On macOS

For individual developers who want to evaluate Redpanda on macOS, there are two ways to install rpk: via Homebrew or by downloading the binary.

To install rpk with Homebrew,

• Run: brew install redpanda-data/tap/redpanda

To install the rpk binary,

• Download the rpk archive with: curl -LO https://github.com/redpanda-data/redpanda/releases/latest/download/rpk-darwin-amd64.zip
• Unpack the archive to /usr/local/bin or any location in your$PATH with:unzip rpk-darwin-amd64.zip

rpk also provides a way to spin up a cluster using Docker on your machine. To create a 3-node cluster, run:rpk container start -n 3, and you are ready to interact with Redpanda! For further steps, please follow the same instructions in the Docker quickstart documentation.

We’re in the process of revising therpk container command to make it better. You can expect updated command usage in the future.

On Windows

We don’t provide anrpk binary for Windows at this moment. Instead, follow the instructions in the Windows quickstart documentation, which utilizes WSL2 and Docker for Windows to setup a Redpanda cluster on your machine.

2. Simplifies topic, group, and ACL management capabilities

rpk gives you cluster management capabilities for topics, groups, ACLs, and more. For example, in order to manage consumer groups, rpk gives a group subcommand , where you can delete groups, describe groups, list groups, or seek to modify an existing group's offsets:

$ rpk group --help
Describe, list, and delete consumer groups and manage their offsets.
...
Available Commands:
  delete               Delete groups from brokers.
  describe           Describe group offset status & lag.
  list                   List all groups.
  seek                 Modify a group's current offsets.
...

rpk makes adding partitions to an existing topic easily:

$ rpk topic add-partitions test.blog --num 10
TOPIC      	ERROR
test.blog  OK

Just like groups, you can delete, describe, and list topics, as well, you can create topics with configurations. For example, the following command creates a new topic with 5GiB for retention.bytes, and confirms the topic configurations.

$ rpk topic create test.blog2 -c retention.bytes=5373952000
TOPIC       	STATUS
test.blog2  OK
$ rpk topic describe test.blog2
=======
NAME    	test.blog2
PARTITIONS  1
REPLICAS	1
CONFIGS
=======
KEY                 	VALUE                      	SOURCE
...
retention.bytes     	5373952000                 	DYNAMIC_TOPIC_CONFIG
...

rpk supports topic modification, and we take the viewpoint that incremental (KIP-339) config modification is more bullet proof and the only recommended way to modify configurations on modern brokers:

$ rpk topic alter-config --help
Set, delete, add, and remove key/value configs for a topic.

This command allows you to incrementally alter the configuration for multiple topics at a time.

Incremental altering supports four operations:

1. Setting a key=value pair
2. Deleting a key’s value
3. Appending a new value to a list-of-values key
4. Subtracting (removing) an existing value from a list-of-values key

Here’s an example:

$ rpk topic alter-config test.blog2 --set retention.bytes=10747904000
TOPIC       	STATUS
test.blog2  OK

Here are all topic commands:

$ rpk topic --help
Create, delete, produce to and consume from Redpanda topics.
...
Available Commands:
  add-partitions Add partitions to existing topics.
  alter-config     Set, delete, add, and remove key/value configs for a topic.
  consume         Consume records from topics.
  create              Create topics.
  delete              Delete topics.
  describe          Describe a topic.
  list                   List topics, optionally listing specific topics.
  produce           Produce records to a topic.
...

ACL creation follows roughly the same flags as kafka-acls.sh. Listing and deleting use the same flags as creation, with missing flags defaulting to an all filter (word this better). We make deletion more bulletproof by prompting a confirmation for the ACLs you will be deleting:

$ rpk acl delete --allow-principal User:daisuke --user admin --password admin --sasl-mechanism SCRAM-SHA-256
MATCHES
=======
PRINCIPAL 	HOST  RESOURCE-TYPE  RESOURCE-NAME  RESOURCE-PATTERN-TYPE  OPERATION  PERMISSION  ERROR
User:daisuke  * 	TOPIC      	*          	LITERAL            	WRITE  	ALLOW
? Confirm deletion of the above matching ACLs? (Y/n)

Once you type ‘Y’, the console updates like this:

? Confirm deletion of the above matching ACLs? Yes
DELETIONS
=========
PRINCIPAL 	HOST  RESOURCE-TYPE  RESOURCE-NAME  RESOURCE-PATTERN-TYPE  OPERATION  PERMISSION  ERROR
User:daisuke  * 	TOPIC      	*          	LITERAL            	WRITE  	ALLOW

Here are all ACL commands:

$ rpk acl --help
Manage ACLs and SASL users.
...
Available commands for ACLs:
  create              Create ACLs.
  delete              Delete ACLs.
  list                   List ACLs.
  user                 Manage SASL users.
...

For more information for available commands, please refer to the rpk documentation.

3. Enables rich formatting on producing and consuming

rpk provides rich formatting capabilities for both producing and consuming. Here are some parts of them as an example.

Create a topic:

$ rpk topic create my.test.topic
TOPIC      	STATUS
my.test.topic  OK

Produce records from STDIN with specifying the record format as key and value with the-f option:

$ cat <<EOF | rpk topic produce my.test.topic -f '%k,%v\n'
key1,value1
key2,value2
key3,value3
EOF
Produced to partition 0 at offset 0 with timestamp 1649921155004.
Produced to partition 0 at offset 1 with timestamp 1649921155004.
Produced to partition 0 at offset 2 with timestamp 1649921155004.

Let’s consume the records. By default, rpk shows them in the following fashion, including their metadata:

$ rpk topic consume my.test.topic
{
  "topic": "my.test.topic",
  "key": "key1",
  "value": "value1",
  "timestamp": 1649921155004,
  "partition": 0,
  "offset": 0
}
{
  "topic": "my.test.topic",
  "key": "key2",
  "value": "value2",
  "timestamp": 1649921155004,
  "partition": 0,
  "offset": 1
}
{
  "topic": "my.test.topic",
  "key": "key3",
  "value": "value3",
  "timestamp": 1649921155004,
  "partition": 0,
  "offset": 2
}

In order to print only the keys and the values, you can use %k and %v to represent keys and values in each record respectively:

$ rpk topic consume my.test.topic -f '%k,%v\n'
key1,value1
key2,value2
key3,value3

By default, the timestamp field is printed as Epoch time number value. It can be printed with either

Go formatting or strftime formatting:

$ rpk topic consume my.test.topic -f '%k,%v,%d{go[2006-01-02T15:04:05Z07:00]}\n'
key1,value1,2022-04-14T07:25:55Z
key2,value2,2022-04-14T07:25:55Z
key3,value3,2022-04-14T07:25:55Z
$ rpk topic consume my.test.topic -f '%k,%v,%d{strftime[%Y-%m-%d %H:%M:%S]}\n'
key1,value1,2022-04-14 07:25:55
key2,value2,2022-04-14 07:25:55
key3,value3,2022-04-14 07:25:55

You can pipe the consumer and the other produce sincerpk topic produce reads from STDIN. In the following example, it consumes keys and values frommy.test.topic and produces them into the other topic,my.production.topic.

$ rpk topic consume my.test.topic -f '%k,%v\n' | rpk topic produce my.production.topic -f '%k,%v\n'

rpk also provides binary encoding capabilities for consuming and producing and data unpacking on consume. The full formatting options are available with rpk topic produce --help or rpk topic consume --help.

4. Tunes the host to adopt production workload

Even when you’re ready to run a production workload on your Redpanda cluster, rpk is continues to be valuable. rpk has a capability of configuring your Linux host to optimize it to run Redpanda most efficiently. Here’s the command, followed by the available options along with brief descriptions.

rpk redpanda tune [command]

For better interaction with these rpk commands, view this table in the original article on the Redpanda blog.

For more details of the commands, use the help command as follows:

rpk redpanda tune help [command]

If you want to dig deeper intorpk’s autotuning capabilities, check out this blog post.

5. Allows for easy configuration changes

In Redpanda’s latest major version update, Redpanda 22.1, our new centralized cluster configuration system is available and makes it easy to manage and edit the cluster-wide configuration using rpk. rpk centralized configuration carries the following subcommands:

\$ rpk cluster config --help
Interact with cluster configuration properties.
...
Available Commands:
edit Edit cluster configuration properties.
export Export cluster configuration.
force-reset Forcibly clear a cluster configuration property on this node.
get Get a cluster configuration property
import Import cluster configuration from a file.
lint Remove any deprecated content from redpanda.yaml.
set Set a single cluster configuration property
status Get configuration status of redpanda nodes.
...

In order to change the broker configuration, simply type rpk cluster config edit, which launches an editor (via $EDITOR) and allows you to edit the configuration on the fly. With this feature, you don’t need to manage the individual configuration files on each broker anymore.

What will you build with rpk and Redpanda?

In summary, rpk makes it easier for devs to interact with Redpanda clusters. Among other benefits, rpk makes it simple to:

1. Install and start Redpanda on local machines
2. Manage, topics, groups, and ACLs
3. Use rich formatting for producing and consuming
4. Tune the host to adopt production workload
5. Make configuration changes

To learn more about rpk and how it can make your life simpler, join us on GitHub, or interact with us in our Slack community. For a more detailed list of how you can interact with rpk, view our rpk documentation.