Running a NATS service

• Installing, running and deploying a NATS Server
• Installing a NATS Server
• Running and deploying a NATS Server
• Windows Service
• Flags
• Environmental considerations
• NATS and Docker
• Tutorial
• Docker Swarm
• Python and NGS Running in Docker
• JetStream
• NGS Leaf Nodes
• NATS and Kubernetes
• NATS Server Clients
• Configuring NATS Server
• Configuring JetStream
• Configuration Management
• NATS Admin CLI
• Terraform
• GitHub Actions
• Kubernetes Controller
• Clustering
• Clustering Configuration
• v2 Routes
• JetStream Clustering
• Administration
• Troubleshooting
• Super-cluster with Gateways
• Configuration
• Leaf Nodes
• Configuration
• JetStream on Leaf Nodes
• Securing NATS
• Enabling TLS
• Authentication
• Tokens
• Username/Password
• TLS Authentication
• TLS Authentication in clusters
• NKeys
• Authentication Timeout
• Decentralized JWT Authentication/Authorization
• Account lookup using Resolver
• Memory Resolver Tutorial
• Mixed Authentication/Authorization Setup
• Authorization
• Multi Tenancy using Accounts
• OCSP Stapling
• Auth Callout
• Logging
• Enabling Monitoring
• MQTT
• Configuration
• Configuring Subject Mapping
• System Events
• System Events & Decentralized JWT Tutorial
• WebSocket
• Configuration
• Managing and Monitoring your NATS Server Infrastructure
• Monitoring
• Monitoring JetStream
• Managing JetStream
• Account Information
• Naming Streams, Consumers, and Accounts
• Streams
• Consumers
• Data Replication
• Disaster Recovery
• Encryption at Rest
• Managing JWT Security
• In Depth JWT Guide
• Upgrading a Cluster
• Slow Consumers
• Signals
• Lame Duck Mode
• Profiling

Installing, running and deploying a NATS Server

The NATS Server is highly optimized and its binary is very compact (less than 20 MB), it can run on any machine from a lowly Raspberry Pi to the largest of servers, in the cloud, on premise or at the edge, on bare metal, on VMs or in containers.

NATS Servers can cluster together to provide fault-tolerance and scalability. NATS servers can run as Leaf Nodes connecting to a cluster transparently proxying and expanding NATS and JetStream service to any location, be it a VPC, a VLAN, a remote office, a single machine at home or even an edge location with partial connectivity.

NATS clusters in different regions or cloud providers can be connected together by gateways that filter unneeded traffic and allow for disaster recovery and queued distributed processing with geo-location affinity.

Even better, if you have build your application with NATS you don't need to run your own server, cluster, or super-cluster and can simply leverage Synadia's NGS, the global NATS super-cluster service provided by Synadia.

This section of the documentation explains how to install, deploy, run, configure, orchestrate, test and manage NATS servers.

Installing a NATS Server

NATS philosophy is simplicity. Installation is just decompressing a zip file and copying the binary to an appropriate directory; you can also use your favorite package manager. Here's a list of different ways you can install or run NATS:

• Command Line
• Docker
• Kubernetes
• Package Manager
• Release Zip
• Development Build

See also installing the NATS client

Supported operating systems and architectures

The following table indicates the current supported NATS server build combinations for operating systems and architectures.

Note, not all installation methods below have distributions for all OS and architecture combinations.

Hardware requirements

The NATS server itself has minimal hardware requirements to support small edge devices, but can take advantage of more resources if available.

CPU should be considered in accepting TLS connections. After a network partition, every disconnected client will attempt to connect to a NATS server in the cluster simultaneously, so CPU on those servers will momentarily spike. When there are many clients this can be mitigated with reconnect jitter settings, and errors can be reduced with longer TLS timeouts, and scaling up cluster sizes.

We highly recommend testing to see if smaller, cheaper machines suffice for your workload - often they do! We suggest starting here and adjusting resources after load testing specific to your environment. When using cloud provider instance types make sure the node has a sufficient NIC to support the required bandwidth for the application needs.

For high throughput use cases, the network interface card (NIC) or the available bandwidth are often the bottleneck, so ensure the hardware or cloud provider instance types are sufficient for your needs.

Core NATS

The tables below outline the minimum number of cores and memory for stable cluster performance with different combinations of publishers, subscribers, and message rates.
Stability is defined as the system avoiding slowdowns or running out of memory.
These tests were conducted inside containers with GOMEMLIMIT set to 90% of the memory allocation, utilizing a 2021-era CPU and SSD for JetStream storage.
Note that these are minimum configurations, and actual production environments may require additional resources.

With JetStream

This table follows the same pattern, with published messages received by a stream using file storage. For a cluster size of three, the stream uses three replicas. Subscribers rely on a "pull consumer" for fetching messages.

For production deployment of JetStream, we recommend starting with at least 4 CPU cores and 8 GiB of memory to reduce the risk of resource-related issues.

For recommendations on configuring limits in Kubernetes, see: https://github.com/nats-io/k8s/tree/main/helm/charts/nats#nats-container-resources

Getting the binary from the command line

The simplest way to get the nats-server binary for your machine is to use the following shell command.

For example, to get the binary for version 2.11.6:

curl -fsSL https://binaries.nats.dev/nats-io/nats-server/v2@v2.11.6 | sh

To get the latest released version, use @latest. You can also use @main to get the tip, or use a tag, specific branch, or commit hash after the @.

Installing via Docker

With Docker, you can install the server easily without scattering binaries and other artifacts on your system. The only pre-requisite is to install docker.

docker pull nats:latest

To run NATS on Docker:

docker run -p 4222:4222 -ti nats:latest

[1] 2019/05/24 15:42:58.228063 [INF] Starting nats-server version #.#.#
[1] 2019/05/24 15:42:58.228115 [INF] Git commit [#######]
[1] 2019/05/24 15:42:58.228201 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2019/05/24 15:42:58.228740 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2019/05/24 15:42:58.228765 [INF] Server is ready

More information on containerized NATS is available here.

Installing via a Package Manager

On Windows, using scoop.sh:

scoop install main/nats-server

On Mac OS:

brew install nats-server

Arch Linux:

For Arch users, there is an AUR package that you can install with:

yay -S nats-server

To test your installation (provided the executable is visible to your shell):

Typing nats-server should output something like

[41634] 2019/05/13 09:42:11.745919 [INF] Starting nats-server version 2.*.*
[41634] 2019/05/13 09:42:11.746240 [INF] Listening for client connections on 0.0.0.0:4222
...
[41634] 2019/05/13 09:42:11.746249 [INF] Server id is NBNYNR4ZNTH4N2UQKSAAKBAFLDV3PZO4OUYONSUIQASTQT7BT4ZF6WX7
[41634] 2019/05/13 09:42:11.746252 [INF] Server is ready

Downloading a Release Build

You can find the latest release of nats-server on the nats-io/nats-server GitHub releases page.

From the releases page, copy the link to the release archive file of your choice and download it using curl -L.

For example, assuming version X.Y.Z of the server and a Linux AMD64:

curl -L https://github.com/nats-io/nats-server/releases/download/vX.Y.Z/nats-server-vX.Y.Z-linux-amd64.tar.gz -o nats-server.tar.gz

tar -xvzf nats-server.tar.gz

Archive:  nats-server.zip
   creating: nats-server-vX.Y.Z-linux-amd64/
...

and finally:

sudo cp nats-server-vX.Y.Z-linux-amd64/nats-server /usr/bin

Installing From the Source

If you have Go installed, installing the binary is easy:

go install github.com/nats-io/nats-server/v2@main

This mechanism will install a build of the main branch, which almost certainly will not be a released version. If you are a developer and want to play with the latest, this is the easiest way.

To test your installation (provided $GOPATH/bin is in your path) by typing nats-server which should output something like

[2397474] 2023/09/27 10:32:02.709019 [INF] Starting nats-server
[2397474] 2023/09/27 10:32:02.709165 [INF]   Version:  2.11.0-dev
[2397474] 2023/09/27 10:32:02.709182 [INF]   Git:      [not set]
[2397474] 2023/09/27 10:32:02.709185 [INF]   Name:     NDQU7SGA4ECW4PHL4KNBY42AFQEZDAPMMQZVSQDKGTARZI5JHJV6KO2N
[2397474] 2023/09/27 10:32:02.709187 [INF]   ID:       NDQU7SGA4ECW4PHL4KNBY42AFQEZDAPMMQZVSQDKGTARZI5JHJV6KO2N
[2397474] 2023/09/27 10:32:02.709795 [INF] Listening for client connections on 0.0.0.0:4222
[2397474] 2023/09/27 10:32:02.710173 [INF] Server is ready

Building From the Source

We use goreleaser to build assets published on GitHub releases.
Our builds are fully reproducible, so with Go installed, one can execute the following commands to build from source:

go install github.com/goreleaser/goreleaser/v2@latest

git clone git@github.com:nats-io/nats-server.git
cd nats-server
git checkout v2.12.0 
[[ `git status --porcelain` ]] && echo "Must have repo in clean state before building"

goreleaser release --skip=announce,publish,validate --clean -f .goreleaser.yml

And to verify SHASUMs against our release:

wget https://github.com/nats-io/nats-server/releases/download/v2.12.0/SHA256SUMS
diff --color --minimal --context=0 SHA256SUMS dist/SHA256SUMS

Running and deploying a NATS Server

The nats-server has many command line options. To get started, you don't have to specify anything. In the absence of any flags, the NATS server will start listening for NATS client connections on port 4222. By default, security is disabled.

Standalone

When the server starts it will print some information including where the server is listening for client connections:

nats-server

[61052] 2021/10/28 16:53:38.003205 [INF] Starting nats-server
[61052] 2021/10/28 16:53:38.003329 [INF]   Version:  2.6.1
[61052] 2021/10/28 16:53:38.003333 [INF]   Git:      [not set]
[61052] 2021/10/28 16:53:38.003339 [INF]   Name:     NDUP6JO4T5LRUEXZUHWXMJYMG4IZAJDNWETTA4GPJ7DKXLJUXBN3UP3M
[61052] 2021/10/28 16:53:38.003342 [INF]   ID:       NDUP6JO4T5LRUEXZUHWXMJYMG4IZAJDNWETTA4GPJ7DKXLJUXBN3UP3M
[61052] 2021/10/28 16:53:38.004046 [INF] Listening for client connections on 0.0.0.0:4222
[61052] 2021/10/28 16:53:38.004683 [INF] Server is ready
...

Docker

If you are running your NATS server in a docker container:

docker run -p 4222:4222 -ti nats:latest

[1] 2021/10/28 23:51:52.705376 [INF] Starting nats-server
[1] 2021/10/28 23:51:52.705428 [INF]   Version:  2.6.1
[1] 2021/10/28 23:51:52.705432 [INF]   Git:      [c91f0fe]
[1] 2021/10/28 23:51:52.705439 [INF]   Name:     NB32AP7VSM3FTKTVEGPQ3OZWSE4T7PQDVJSJMGYFIDKJA6TQEZMV2JNN
[1] 2021/10/28 23:51:52.705446 [INF]   ID:       NB32AP7VSM3FTKTVEGPQ3OZWSE4T7PQDVJSJMGYFIDKJA6TQEZMV2JNN
[1] 2021/10/28 23:51:52.705448 [INF] Using configuration file: nats-server.conf
[1] 2021/10/28 23:51:52.709505 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2021/10/28 23:51:52.709590 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2021/10/28 23:51:52.709882 [INF] Server is ready
[1] 2021/10/28 23:51:52.710394 [INF] Cluster name is 3tlKqFWx91wnnAekR76U9V
[1] 2021/10/28 23:51:52.710419 [WRN] Cluster name was dynamically generated, consider setting one
[1] 2021/10/28 23:51:52.710446 [INF] Listening for route connections on 0.0.0.0:6222
...

Running nats-server as a systemd service on Linux

You can easily and quickly use systemd to start (and restart if needed) the nats-server process.

Please see the example files located in the util directory of the nats-server repo that you can use to generate your own /etc/systemd/system/nats.service file.

Exit Status

As a long-running service, it's important for administrators to understand what guarantees the nats-server makes about how it exits and what that means.

The approach chosen by nats-server is that "exited cleanly after being asked to shutdown" is a successful exit. Even on platforms where that shutdown request is a POSIX signal. It is not an error to successfully exit when asked to shut down.

When configuring a service manager, whether systemd or any other, we recommend that it be configured to restart the nats-server on non-zero exit status or death by signal or any other abnormal exit, so that the service manager does what service managers do best: keeping essential services available when wanted. The service manager probably should not restart the nats-server if it exits successfully. If your environment does not provide any means to interact with the nats-server except through the service agent, then it doesn't matter either way; this distinction only becomes noticeable when something other than the service manager asked the nats-server to shut down.

JetStream

Remember that in order to enable JetStream and all the functionalities that use it you need to enable it on at least one of your servers

Command Line

Enable JetStream by specifying the -js flag when starting the NATS server.

$ nats-server -js

Configuration File

You can also enable JetStream through a configuration file. By default, the JetStream subsytem will store data in the /tmp directory. Here's a minimal file that will store data in a local "nats" directory, suitable for development and local testing.

nats-server -c js.conf

# js.conf
jetstream {
   store_dir=nats
}

Normally JetStream will be run in clustered mode and will replicate data, so the best place to store JetStream data would be locally on a fast SSD. One should specifically avoid NAS or NFS storage for JetStream. More information on containerized NATS is available here.

Windows Service

The NATS server supports running as a Windows service. In fact, this is the recommended way of running NATS on Windows. There is currently no installer; users should use sc.exe to install the service:

sc.exe create nats-server binPath= "%NATS_PATH%\nats-server.exe [nats-server flags]"
sc.exe start nats-server

The above will create and start a nats-server service. Note the nats-server flags should be provided when creating the service. This allows for the running multiple NATS server configurations on a single Windows server by using a 1:1 service instance per installed NATS server service. Once the service is running, it can be controlled using sc.exe or nats-server.exe --signal:

REM Reload server configuration
nats-server.exe --signal reload

REM Reopen log file for log rotation
nats-server.exe --signal reopen

REM Stop the server
nats-server.exe --signal stop

The above commands will default to controlling the nats-server service. If the service is another name, it can be specified:

nats-server.exe --signal stop=<service name>

For a complete list of signals, see process signaling.

Permissions

The default user in the above example will be System, which has local administrator permissions and write access to almost all files on disk.

If you change the service user, e.g. to the more restricted NetworkService, make sure permissions have been set. The server at a minimum will need read access to the config file and when using Jetstream, write access to the JetStream store directory.

sc config "nats-server" obj= "NT AUTHORITY\NetworkService" password= ""

Nats-server will write log entries to the default console when no log file is configured. Console logging is not permitted for all users (e.g. not for NetworkService).

提示

To ease debugging, it is recommended to run a NATS service with an explicit log file and carefully check write permissions for the configured user.

sc.exe create nats-server binPath= "%NATS_PATH%\nats-server.exe --log C:\temp\nats-server.log [other flags]"

Windows Service Specific Settings

Windows Service Specific Settings

NATS_STARTUP_DELAY environment variable

The Windows service system requires communication with programs that run as Windows services. One important signal from the program is the initial "ready" signal, where the program informs Windows that it is running as expected.

By default nats-server allows itself up to 10 seconds to send this signal. If the server is not ready after this time, the server will signal a failure to start. This delay can be adjusted by setting the NATS_STARTUP_DELAY environment variable to a suitable duration (e.g. "20s" for 20 seconds, "1m" for one minute).

Please Note

• For the environment variable NATS_STARTUP_DELAY to be accessible from the NATS service, it is recommended to set it as a SYSTEM variable.
• NATS_STARTUP_DELAY=30s will make the NATS server wait up to 30s, but will report the service as RUNNING as soon as the server is ready to accept connections. To test the extended time startup timeout you may need to slow down server startup, e.g. by using a very large stream (10s of GB) or placing Jetstream storage on a slow network device.

This adjustment can be necessary in cases where NATS is correctly running from command line, but takes longer than 10s to recover JetStream stream state and connect to its cluster peers.

Flags

The NATS server has many flags to customize its behavior without having to write a configuration file.

The configuration flags revolve around:

• Server Options
• Logging
• Authorization
• TLS Security
• Clustering
• Information

Server Options

JetStream Options

Authentication Options

The following options control straightforward authentication:

See token authentication, and username/password for more information.

Logging Options

The following flags are available on the server to configure logging:

You can read more about logging configuration here.

TLS Options

You can read more about tls configuration here.

Cluster Options

The following flags are available on the server to configure clustering:

You can read more about clustering configuration here.

Common Options

Environmental considerations

Networking

Load balancers

It is possible to deploy a load balancer between the client applications and the cluster servers (or even between servers in a cluster or between clusters in a super-cluster), but you don't need to: NATS already has its own mechanisms to balance the connections between the seeds in the connection URL (including the clients randomizing the returned DNS A records) and to automatically re-establish dropped connections. If you have a cluster with 3 seed nodes you often get more network throughput than going through a load balancer (cloud provider load balancers can be woefully under-powered, not to mention it costs you more money as the load balancer is typically billed by the amount of data going through it). Finally, if you want to use TLS for authentication you do not want the load balancer to be the TLS termination point.

注意

If you decide to use load balancers with NATS you need to understand the client and cluster connection and auto-discovery behavior.

Client connections and routes are permanent. Therefore a load balancer will not distribute messages from one client connections between servers, or worse (which is common when deploying on auto-config environments), you end up with redundant, but disconnected (non-clustered) servers to which the clients will randomly connect. Advertising needs to be switched off or configured to point to the load balancer address through the advertise server config option.

Load balancers can also cause issues through improperly configured idle detection, protocol problems due to packet inspection, and ephemeral port problems at high scale.

If routes or gateway connections go through load balancers, you could very well have the kind of problems mentioned above, which could results in JetStream lost quorum periods and create undue re-synchronization and protocol overhead traffic.

There are some sensible use cases for running client connections through load balancers. If the balanced NATS servers are setup as clusters (or superclusters), which provide transparent message routing, the load balancer can simplify the client config by presenting a single port of entry, or automatically connect to the geographically closed cluster node.

Virtualization, Containerization

NATS is 'cloud native' and expected to be deployed in virtual environments and/or containers.

However, when it comes to ensuring the highest possible level of performance that NATS can provide it is good to keep a few things in mind.

Think of Core NATS servers as a software equivalent of network switches. Enable JetStream, and they also become a new kind of DB server as well.

What you need to remember is that when selecting the instance types and storage options for your NATS server host instances that in public clouds: you get what you pay for.

For example non network optimized instances may give you 10 Gb/s of network bandwidth... but only for some period of time (like 30 minutes), after which the available bandwidth may drop down dramatically (like to 5 Gb/s) for another period of time. So select network optimized instances types instead if you always need the advertised bandwidth.

It's the same when it comes to storage options: local SSDs instance types can provide the best latency, while using a network attached block storage, e.g. Elastic Block Storage from AWS), can provide the highest overall throughput. When using EBS again you get what you pay for: general purpose storage type may give you a certain number of IOPS, but you can sustain those rates only for some period of time after which the number can drop down dramatically. So select IO optimized storage types if you want to continuously sustain the same max number of IOPS (e.g. AWS).

Resource Limits

Be careful when setting resource limits for the nats-server containers. The nats-server processes use resources in proportion to the load traffic generated by all the client applications, if the NATS (and JetStream) usage is high or in bursts (nats-server is very fast and can process sharp bursts in traffic), then you will need to set the container resource limits accordingly, or the container orchestration system will kill the server's container. The nats-server will automatically detect number of available cores, but it will try to use all host memory, not the resource limits set for the container, unless you specify GOMEMLIMIT. GOMEMLIMIT is an available option in official NATS helm charts.

NATS and Docker

NATS Server Containerization

The NATS server is provided as a Docker image on Docker Hub that you can run using the Docker daemon. The NATS server Docker image is extremely lightweight, coming in under 10 MB in size.

Synadia actively maintains and supports the NATS server Docker image.

Nightly

The nightly build container can be found here

Usage

To use the Docker container image, install Docker and pull the public image:

docker pull nats

Run the NATS server image:

docker run nats

By default the NATS server exposes multiple ports:

• 4222 is for clients.
• 8222 is an HTTP management port for information reporting.
• 6222 is a routing port for clustering.

The default ports may be customized by providing either a -p or -P option on the docker run command line.

The following steps illustrate how to run a server with the ports exposed on a docker network.

First, create the docker network 'nats:'

docker network create nats

Then start the server:

docker run --name nats --network nats --rm -p 4222:4222 -p 8222:8222 nats --http_port 8222

Creating a NATS Cluster

First, run a server with the ports exposed on the 'nats' docker network:

docker run --name nats --network nats --rm -p 4222:4222 -p 8222:8222 nats --http_port 8222 --cluster_name NATS --cluster nats://0.0.0.0:6222

[1] 2021/09/28 09:21:56.554756 [INF] Starting nats-server
[1] 2021/09/28 09:21:56.554864 [INF]   Version:  2.6.1
[1] 2021/09/28 09:21:56.554878 [INF]   Git:      [c91f0fe]
[1] 2021/09/28 09:21:56.554894 [INF]   Name:     NDIQLLD2UGGPSAEYBKHW3S2JB2DXIAFHMIWWRUBAX7FC4RTQX4ET2JNQ
[1] 2021/09/28 09:21:56.555001 [INF]   ID:       NDIQLLD2UGGPSAEYBKHW3S2JB2DXIAFHMIWWRUBAX7FC4RTQX4ET2JNQ
[1] 2021/09/28 09:21:56.557658 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2021/09/28 09:21:56.557967 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2021/09/28 09:21:56.559224 [INF] Server is ready
[1] 2021/09/28 09:21:56.559375 [INF] Cluster name is NATS
[1] 2021/09/28 09:21:56.559433 [INF] Listening for route connections on 0.0.0.0:6222

Next, start additional servers pointing them to the seed server to cause them to form a cluster:

docker run --name nats-1 --network nats --rm nats --cluster_name NATS --cluster nats://0.0.0.0:6222 --routes=nats://ruser:T0pS3cr3t@nats:6222
docker run --name nats-2 --network nats --rm nats --cluster_name NATS --cluster nats://0.0.0.0:6222 --routes=nats://ruser:T0pS3cr3t@nats:6222

NOTE Since the Docker image protects routes using credentials we need to provide them above. Extracted from Docker image configuration

To verify the routes are connected, you can make a request to the monitoring endpoint on /routez and confirm that there are now 2 routes:

curl http://127.0.0.1:8222/routez

{
  "server_id": "NDIQLLD2UGGPSAEYBKHW3S2JB2DXIAFHMIWWRUBAX7FC4RTQX4ET2JNQ",
  "now": "2021-09-28T09:22:15.8019785Z",
  "num_routes": 2,
  "routes": [
    {
      "rid": 5,
      "remote_id": "NBRAUY3YSVFYU7BFWI2YF5VPQFGO2XCKKAHYZ7ETCMGB3SQY3FDFTYOQ",
      "did_solicit": false,
      "is_configured": false,
      "ip": "172.18.0.3",
      "port": 59092,
      "pending_size": 0,
      "rtt": "1.2505ms",
      "in_msgs": 4,
      "out_msgs": 3,
      "in_bytes": 2714,
      "out_bytes": 1943,
      "subscriptions": 35
    },
    {
      "rid": 6,
      "remote_id": "NA5STTST5GYFCD22M2I3VDJ57LQKOU35ZVWKQY3O5QRFGOPC3RFDIDVJ",
      "did_solicit": false,
      "is_configured": false,
      "ip": "172.18.0.4",
      "port": 47424,
      "pending_size": 0,
      "rtt": "1.2008ms",
      "in_msgs": 4,
      "out_msgs": 1,
      "in_bytes": 2930,
      "out_bytes": 833,
      "subscriptions": 35
    }
  ]
}

Creating a NATS Cluster with Docker Compose

It is also straightforward to create a cluster using Docker Compose. Below is a simple example that uses a network named 'nats' to create a full mesh cluster.

version: "3.5"
services:
  nats:
    image: nats
    ports:
      - "8222:8222"
    command: "--cluster_name NATS --cluster nats://0.0.0.0:6222 --http_port 8222 "
    networks: ["nats"]
  nats-1:
    image: nats
    command: "--cluster_name NATS --cluster nats://0.0.0.0:6222 --routes=nats://ruser:T0pS3cr3t@nats:6222"
    networks: ["nats"]
    depends_on: ["nats"]
  nats-2:
    image: nats
    command: "--cluster_name NATS --cluster nats://0.0.0.0:6222 --routes=nats://ruser:T0pS3cr3t@nats:6222"
    networks: ["nats"]
    depends_on: ["nats"]

networks:
  nats:
    name: nats

Now we use Docker Compose to create the cluster that will be using the 'nats' network:

docker-compose -f nats-cluster.yaml up

[+] Running 3/3
 ⠿ Container xxx_nats_1    Created
 ⠿ Container xxx_nats-1_1  Created
 ⠿ Container xxx_nats-2_1  Created
Attaching to nats-1_1, nats-2_1, nats_1
nats_1    | [1] 2021/09/28 10:42:36.742844 [INF] Starting nats-server
nats_1    | [1] 2021/09/28 10:42:36.742898 [INF]   Version:  2.6.1
nats_1    | [1] 2021/09/28 10:42:36.742913 [INF]   Git:      [c91f0fe]
nats_1    | [1] 2021/09/28 10:42:36.742929 [INF]   Name:     NCZIIQ6QT4KT5K5WBP7H2RRBM4MSYD4C2TVSRZOZN57EHX6VTF4EWXAU
nats_1    | [1] 2021/09/28 10:42:36.742954 [INF]   ID:       NCZIIQ6QT4KT5K5WBP7H2RRBM4MSYD4C2TVSRZOZN57EHX6VTF4EWXAU
nats_1    | [1] 2021/09/28 10:42:36.745289 [INF] Starting http monitor on 0.0.0.0:8222
nats_1    | [1] 2021/09/28 10:42:36.745737 [INF] Listening for client connections on 0.0.0.0:4222
nats_1    | [1] 2021/09/28 10:42:36.750381 [INF] Server is ready
nats_1    | [1] 2021/09/28 10:42:36.750669 [INF] Cluster name is NATS
nats_1    | [1] 2021/09/28 10:42:36.751444 [INF] Listening for route connections on 0.0.0.0:6222
nats-1_1  | [1] 2021/09/28 10:42:37.709888 [INF] Starting nats-server
nats-1_1  | [1] 2021/09/28 10:42:37.709977 [INF]   Version:  2.6.1
nats-1_1  | [1] 2021/09/28 10:42:37.709999 [INF]   Git:      [c91f0fe]
nats-1_1  | [1] 2021/09/28 10:42:37.710023 [INF]   Name:     NBHTXXY3HYZVPXITYQ73BSDA5CQZINTKYRM23XFI46RWWTTUP5TAXQMB
nats-1_1  | [1] 2021/09/28 10:42:37.710042 [INF]   ID:       NBHTXXY3HYZVPXITYQ73BSDA5CQZINTKYRM23XFI46RWWTTUP5TAXQMB
nats-1_1  | [1] 2021/09/28 10:42:37.711646 [INF] Listening for client connections on 0.0.0.0:4222
nats-1_1  | [1] 2021/09/28 10:42:37.712197 [INF] Server is ready
nats-1_1  | [1] 2021/09/28 10:42:37.712376 [INF] Cluster name is NATS
nats-1_1  | [1] 2021/09/28 10:42:37.712469 [INF] Listening for route connections on 0.0.0.0:6222
nats_1    | [1] 2021/09/28 10:42:37.718918 [INF] 172.18.0.4:52950 - rid:4 - Route connection created
nats-1_1  | [1] 2021/09/28 10:42:37.719906 [INF] 172.18.0.3:6222 - rid:4 - Route connection created
nats-2_1  | [1] 2021/09/28 10:42:37.731357 [INF] Starting nats-server
nats-2_1  | [1] 2021/09/28 10:42:37.731518 [INF]   Version:  2.6.1
nats-2_1  | [1] 2021/09/28 10:42:37.731531 [INF]   Git:      [c91f0fe]
nats-2_1  | [1] 2021/09/28 10:42:37.731543 [INF]   Name:     NCG6UQ2N3IHE6OS76TL46RNZBAPHNUCQSA64FDFHG5US2LLJOQLD5ZK2
nats-2_1  | [1] 2021/09/28 10:42:37.731554 [INF]   ID:       NCG6UQ2N3IHE6OS76TL46RNZBAPHNUCQSA64FDFHG5US2LLJOQLD5ZK2
nats-2_1  | [1] 2021/09/28 10:42:37.732893 [INF] Listening for client connections on 0.0.0.0:4222
nats-2_1  | [1] 2021/09/28 10:42:37.733431 [INF] Server is ready
nats-2_1  | [1] 2021/09/28 10:42:37.733491 [INF] Cluster name is NATS
nats-2_1  | [1] 2021/09/28 10:42:37.733835 [INF] Listening for route connections on 0.0.0.0:6222
nats_1    | [1] 2021/09/28 10:42:37.740860 [INF] 172.18.0.5:54616 - rid:5 - Route connection created
nats-2_1  | [1] 2021/09/28 10:42:37.741557 [INF] 172.18.0.3:6222 - rid:4 - Route connection created
nats-1_1  | [1] 2021/09/28 10:42:37.743981 [INF] 172.18.0.5:6222 - rid:5 - Route connection created
nats-2_1  | [1] 2021/09/28 10:42:37.744332 [INF] 172.18.0.4:40250 - rid:5 - Route connection created

Testing the Clusters

Now, the following should work: make a subscription on one of the nodes and publish it from another node. You should be able to receive the message without problems.

docker run --network nats --rm -it natsio/nats-box

Inside the container:

nats sub -s nats://nats:4222 hello &
nats pub -s "nats://nats-1:4222" hello first
nats pub -s "nats://nats-2:4222" hello second

Stopping the seed node, which received the subscription, should trigger an automatic failover to the other nodes:

docker-compose -f nats-cluster.yaml stop nats

Output extract:

... 
16e55f1c4f3c:~# 10:47:28 Disconnected due to: EOF, will attempt reconnect
10:47:28 Disconnected due to: EOF, will attempt reconnect
10:47:28 Reconnected [nats://172.18.0.4:4222]

Publishing again will continue to work after the reconnection:

nats pub -s "nats://nats-1:4222" hello again
nats pub -s "nats://nats-2:4222" hello again

Tutorial

See the NATS Docker tutorial for more instructions on using the NATS server Docker image.

Tutorial

In this tutorial you run the NATS server Docker image. The Docker image provides an instance of the NATS Server. Synadia actively maintains and supports the nats-server Docker image. The NATS image is only 6 MB in size.

1. Set up Docker.

See Get Started with Docker for guidance.

The easiest way to run Docker is to use the Docker Toolbox.

2. Run the nats-server Docker image.

docker run -p 4222:4222 -p 8222:8222 -p 6222:6222 --name nats-server -ti nats:latest

3. Verify that the NATS server is running.

You should see the following:

Unable to find image 'nats:latest' locally
latest: Pulling from library/nats
2d3d00b0941f: Pull complete 
24bc6bd33ea7: Pull complete 
Digest: sha256:47b825feb34e545317c4ad122bd1a752a3172bbbc72104fc7fb5e57cf90f79e4
Status: Downloaded newer image for nats:latest

Followed by this, indicating that the NATS server is running:

[1] 2019/06/01 18:34:19.605144 [INF] Starting nats-server version 2.0.0
[1] 2019/06/01 18:34:19.605191 [INF] Starting http monitor on 0.0.0.0:8222
[1] 2019/06/01 18:34:19.605286 [INF] Listening for client connections on 0.0.0.0:4222
[1] 2019/06/01 18:34:19.605312 [INF] Server is ready
[1] 2019/06/01 18:34:19.608756 [INF] Listening for route connections on 0.0.0.0:6222

Notice how quickly the NATS server Docker image downloads. It is a mere 6 MB in size.

4. Test the NATS server to verify it is running.

An easy way to test the client connection port is through using telnet.

telnet localhost 4222

Expected result:

Trying ::1...
Connected to localhost.
Escape character is '^]'.
INFO {"server_id":"NDP7NP2P2KADDDUUBUDG6VSSWKCW4IC5BQHAYVMLVAJEGZITE5XP7O5J","version":"2.0.0","proto":1,"go":"go1.11.10","host":"0.0.0.0","port":4222,"max_payload":1048576,"client_id":13249}

You can also test the monitoring endpoint, viewing http://localhost:8222 with a browser.

Docker Swarm

Step 1:

Create an overlay network for the cluster (in this example, nats-cluster-example), and instantiate an initial NATS server.

First create an overlay network:

docker network create --driver overlay nats-cluster-example

Next instantiate an initial "seed" server for a NATS cluster listening for other servers to join route to it on port 6222:

docker service create --network nats-cluster-example --name nats-cluster-node-1 nats:1.0.0 -cluster nats://0.0.0.0:6222 -DV

Step 2:

The 2nd step is to create another service which connects to the NATS server within the overlay network. Note that we connect to to the server at nats-cluster-node-1:

docker service create --name ruby-nats --network nats-cluster-example wallyqs/ruby-nats:ruby-2.3.1-nats-v0.8.0 -e '
  NATS.on_error do |e|
    puts "ERROR: #{e}"
  end
  NATS.start(:servers => ["nats://nats-cluster-node-1:4222"]) do |nc|
    inbox = NATS.create_inbox
    puts "[#{Time.now}] Connected to NATS at #{nc.connected_server}, inbox: #{inbox}"

    nc.subscribe(inbox) do |msg, reply|
      puts "[#{Time.now}] Received reply - #{msg}"
    end

    nc.subscribe("hello") do |msg, reply|
      next if reply == inbox
      puts "[#{Time.now}] Received greeting - #{msg} - #{reply}"
      nc.publish(reply, "world")
    end

    EM.add_periodic_timer(1) do
      puts "[#{Time.now}] Saying hi (servers in pool: #{nc.server_pool}"
      nc.publish("hello", "hi", inbox)
    end
  end'

Step 3:

Now you can add more nodes to the Swarm cluster via more docker services, referencing the seed server in the -routes parameter:

docker service create --network nats-cluster-example --name nats-cluster-node-2 nats:1.0.0 -cluster nats://0.0.0.0:6222 -routes nats://nats-cluster-node-1:6222 -DV

In this case, nats-cluster-node-1 is seeding the rest of the cluster through the autodiscovery feature. Now NATS servers nats-cluster-node-1 and nats-cluster-node-2 are clustered together.

Add in more replicas of the subscriber:

docker service scale ruby-nats=3

Then confirm the distribution on the Docker Swarm cluster:

docker service ps ruby-nats

ID                         NAME         IMAGE                                     NODE    DESIRED STATE  CURRENT STATE          ERROR
25skxso8honyhuznu15e4989m  ruby-nats.1  wallyqs/ruby-nats:ruby-2.3.1-nats-v0.8.0  node-1  Running        Running 2 minutes ago  
0017lut0u3wj153yvp0uxr8yo  ruby-nats.2  wallyqs/ruby-nats:ruby-2.3.1-nats-v0.8.0  node-1  Running        Running 2 minutes ago  
2sxl8rw6vm99x622efbdmkb96  ruby-nats.3  wallyqs/ruby-nats:ruby-2.3.1-nats-v0.8.0  node-2  Running        Running 2 minutes ago

The sample output after adding more NATS server nodes to the cluster, is below - and notice that the client is dynamically aware of more nodes being part of the cluster via auto discovery!

[2016-08-15 12:51:52 +0000] Saying hi (servers in pool: [{:uri=>#<URI::Generic nats://10.0.1.3:4222>, :was_connected=>true, :reconnect_attempts=>0}]
[2016-08-15 12:51:53 +0000] Saying hi (servers in pool: [{:uri=>#<URI::Generic nats://10.0.1.3:4222>, :was_connected=>true, :reconnect_attempts=>0}]
[2016-08-15 12:51:54 +0000] Saying hi (servers in pool: [{:uri=>#<URI::Generic nats://10.0.1.3:4222>, :was_connected=>true, :reconnect_attempts=>0}]
[2016-08-15 12:51:55 +0000] Saying hi (servers in pool: [{:uri=>#<URI::Generic nats://10.0.1.3:4222>, :was_connected=>true, :reconnect_attempts=>0}, {:uri=>#<URI::Generic nats://10.0.1.7:4222>, :reconnect_attempts=>0}, {:uri=>#<URI::Generic nats://10.0.1.6:4222>, :reconnect_attempts=>0}]

Sample output after adding more workers which can reply back (since ignoring own responses):

[2016-08-15 16:06:26 +0000] Received reply - world
[2016-08-15 16:06:26 +0000] Received reply - world
[2016-08-15 16:06:27 +0000] Received greeting - hi - _INBOX.b8d8c01753d78e562e4dc561f1
[2016-08-15 16:06:27 +0000] Received greeting - hi - _INBOX.4c35d18701979f8c8ed7e5f6ea

And so forth...

From here you can experiment adding to the NATS cluster by simply adding servers with new service names, that route to the seed server nats-cluster-node-1. As you've seen above, clients will automatically be updated to know that new servers are available in the cluster.

docker service create --network nats-cluster-example --name nats-cluster-node-3 nats:1.0.0 -cluster nats://0.0.0.0:6222 -routes nats://nats-cluster-node-1:6222 -DV

Python and NGS Running in Docker

Start a lightweight Docker container:

docker run --entrypoint /bin/bash -it python:3.8-slim-buster

Or you can also mount local creds via a volume:

docker run --entrypoint /bin/bash -v $HOME/.nkeys/creds/synadia/NGS/:/creds -it python:3.8-slim-buster

Install nats.py and dependencies to install nkeys:

apt-get update && apt-get install -y build-essential curl
pip install asyncio-nats-client[nkeys]

Get the Python examples using curl:

curl -o nats-pub.py -O -L https://raw.githubusercontent.com/nats-io/nats.py/master/examples/nats-pub/__main__.py
curl -o nats-sub.py -O -L https://raw.githubusercontent.com/nats-io/nats.py/master/examples/nats-sub/__main__.py

Create a subscription that lingers:

python nats-sub.py --creds /creds/NGS.creds  -s tls://connect.ngs.global:4222 hello &

Publish a message:

python nats-pub.py --creds /creds/NGS.creds  -s tls://connect.ngs.global:4222 hello -d world

JetStream

This mini-tutorial shows how to run a NATS server with JetStream enabled in a local Docker container. This enables quick and consequence-free experimentation with the many features of JetStream.

Using the official nats image, start a server. The -js option is passed to the server to enable JetStream. The -p option forwards your local 4222 port to the server inside the container, 4222 is the default client connection port.

docker run -p 4222:4222 nats -js

To persist JetStream data to a volume, you can use the -v option in combination with -sd:

docker run -p 4222:4222 -v nats:/data nats -js -sd /data

With the server running, use nats bench to create a stream and publish some messages to it.

nats bench -s localhost:4222 benchsubject --js --pub 1 --msgs=100000

JetStream persists the messages (on disk by default). Now consume them with:

nats bench -s localhost:4222 benchsubject --js --sub 3 --msgs=100000

You can use nats to inspect various aspects of the stream, for example:

nats -s localhost:4222 stream list
╭────────────────────────────────────────────────────────────────────────────────────╮
│                                       Streams                                      │
├─────────────┬─────────────┬─────────────────────┬──────────┬────────┬──────────────┤
│ Name        │ Description │ Created             │ Messages │ Size   │ Last Message │
├─────────────┼─────────────┼─────────────────────┼──────────┼────────┼──────────────┤
│ benchstream │             │ 2024-06-07 20:26:38 │ 100,000  │ 16 MiB │ 35s          │
╰─────────────┴─────────────┴─────────────────────┴──────────┴────────┴──────────────╯

Related and useful:

• Official Docker image for the NATS server on GitHub and issues
• nats images on DockerHub
• nats CLI tool and nats bench
• Administer JetStream

NGS Leaf Nodes

This mini-tutorial shows how to run 2 NATS server in local Docker containers interconnected via Synadia Cloud Platform. NGS is a global managed NATS network of NATS, and the local containers will connect to it as leaf nodes.

Start by creating a free account on https://cloud.synadia.com/.

Once you are logged in, go into the default account (you can manage multiple isolated NGS account within your Synadia Cloud account).

In Settings > Limits, increase Leaf Nodes to 2. Save the configuration change. (Your free account comes with up to 2 leaf connection, but the account is configured to use at most 1 initially).

Now navigate to the Users section of your default account and create 2 users, red and blue. (Users are another way you can isolate parts of your systems customizing permissions, access to data, limits and more)

For each of the two users, select Get Connected and Download Credentials.

You should now have 2 files on your computer: default-red.creds and default-blue.creds.

Create a minimal NATS Server configuration file leafnode.conf, it will work for both leaf nodes:

leafnodes {
    remotes = [
        {
          url: "tls://connect.ngs.global"
          credentials: "ngs.creds"
        },
    ]
}

Let's start the first leafnode (for user red) with:

docker run  -p 4222:4222 -v leafnode.conf:/leafnode.conf -v /etc/ssl/cert.pem:/etc/ssl/cert.pem -v default-red.creds:/ngs.creds  nats:latest -c /leafnode.conf

-p 4222:4222 maps the server port 4222 inside the container to your local port 4222. -v leafnode.conf:/leafnode.conf mounts the configuration file created above at location /leafnode.conf in the container. -v /etc/ssl/cert.pem:/etc/ssl/cert.pem installs root certificates in the container, since the nats image does not bundle them, and they are required to verify the TLS certificate presented by NGS. -v default-red.creds:/ngs.creds installs the credentials for user red at location /ngs.creds inside the container. -c /leafnode.conf are arguments passed to the container entry point (nats-server).

Launching the container, you should see the NATS server starting successfully:

[1] 2024/06/14 18:03:51.810719 [INF] Server is ready
[1] 2024/06/14 18:03:52.075951 [INF] 34.159.142.0:7422 - lid:5 - Leafnode connection created for account: $G
[1] 2024/06/14 18:03:52.331354 [INF] 34.159.142.0:7422 - lid:5 - JetStream using domains: local "", remote "ngs"

Now start the second leaf nodes with two minor tweaks to the command:

docker run  -p 4333:4222 -v leafnode.conf:/leafnode.conf -v /etc/ssl/cert.pem:/etc/ssl/cert.pem -v default-blue.creds:/ngs.creds  nats:latest -c /leafnode.conf

Notice we bind to local port 4333 (since 4222) is busy, and we mount blue credentials.

Congratulations, you have 2 leaf nodes connected to the NGS global network. Despite this being a global shared environment, your account is completely isolated from the rest of the traffic, and vice versa.

Now let's make 2 clients connected to the 2 leaf nodes talk to each other.

Let us start a simple service on the Leafnode of user red:

nats -s localhost:4222 reply docker-leaf-test "At {{Time}}, I received your request: {{Request}}"

Using the LeafNode run by user blue, let's send a request:

$ nats -s localhost:4333 request docker-leaf-test "Hello World"

At 8:15PM, I received your request: Hello World

Congratulations, you just connected 2 Leaf nodes to the global NGS network and used them to send a request and receive a response.

Your messages were routed transparently with millions of others, but they were not visible to anyone outside of your Synadia Cloud account.

Related and useful:

• Official Docker image for the NATS server on GitHub and issues
• nats images on DockerHub
• nats CLI tool and nats bench
• Leaf Nodes configuration

NATS and Kubernetes

The recommended way to deploy NATS on Kubernetes is using Helm with the official NATS Helm Chart.

Helm repo

To register the NATS Helm chart run:

helm repo add nats https://nats-io.github.io/k8s/helm/charts/

Config values

The default configuration values of the chart will deploy a single NATS server as a StatefulSet and a single replica nats-box Deployment.

The ArtifactHub page provides the list of Helm configuration values and examples for the current release.

For tracking the development version, refer to the source repo.

Once the desired configuration is created, install the chart:

helm install nats nats/nats

Validate connectivity

Once the pods are up, validate by accessing the nats-box container and running a CLI command.

kubectl exec -it deployment/nats-box -- nats pub test hi

The output should indicate a successful publish to NATS.

16:17:00 Published 2 bytes to "test"

Commercial Options

Synadia offers Deploy for Kubernetes, a self-service, bring-your-own Kubernetes deployment option that includes NATS and additional components.

NATS Server Clients

A NATS client is an application making a connection to one of the nats servers pointed to by its connection URL, and uses a credential file to authenticate and indicate its authorization to the server and the whole NATS infrastructure.

The nats-server doesn't come bundled with any clients, but its companion is the nats CLI tool that you should install (even if you don't intend to run your own servers) as it is the best tool to use to test, monitor, manage and generally interact with a NATS infrastructure (regardless of that infrastructure being an isolated local server, a leaf node server, a cluster or even a global super-cluster).

Other NATS client tools to know about are the nsc CLI tool (to manage accounts attributes and user JWT tokens) and the 'nk' tool (and library) to manage Nkeys.

Also, most client libraries come with sample programs that allow you to publish, subscribe, send requests and reply messages.

Embedding NATS

If your application is in Go, and if it fits your use case and deployment scenarios, you can even embed a NATS server inside your application.

Embedding NATS in Go

Installing the nats CLI Tool

Please refer to the installation section in the readme.

Testing your setup

Open a terminal and start a nats-server:

nats-server

[45695] 2021/09/29 02:22:53.570667 [INF] Starting nats-server
[45695] 2021/09/29 02:22:53.570796 [INF]   Version:  2.6.1
[45695] 2021/09/29 02:22:53.570799 [INF]   Git:      [not set]
[45695] 2021/09/29 02:22:53.570804 [INF]   Name:     NAAACXGWSD6ZW5KVHOTSGGPU2JCMZUDSMY5GVZZP27DMRPWYINC2X6ZI
[45695] 2021/09/29 02:22:53.570807 [INF]   ID:       NAAACXGWSD6ZW5KVHOTSGGPU2JCMZUDSMY5GVZZP27DMRPWYINC2X6ZI
[45695] 2021/09/29 02:22:53.571747 [INF] Listening for client connections on 0.0.0.0:4222
[45695] 2021/09/29 02:22:53.572051 [INF] Server is ready

On another terminal session first check the connection to the server

nats server check connection -s nats://0.0.0.0:4222

OK Connection OK:connected to nats://127.0.0.1:4222 in 790.28µs OK:rtt time 69.896µs OK:round trip took 0.000102s | connect_time=0.0008s;0.5000;1.0000 rtt=0.0001s;0.5000;1.0000 request_time=0.0001s;0.5000;1.0000

Next, start a subscriber using the nats CLI tool:

nats subscribe ">" -s nats://0.0.0.0:4222

Note that when the client connected, the server didn't log anything interesting because server output is relatively quiet unless something interesting happens.

To make the server output more lively, you can specify the -V flag to enable logging of server protocol tracing messages. Go ahead and <ctrl>+c the process running the server, and restart the server with the -V flag:

nats-server -V

[45703] 2021/09/29 02:23:05.189377 [INF] Starting nats-server
[45703] 2021/09/29 02:23:05.189489 [INF]   Version:  2.6.1
[45703] 2021/09/29 02:23:05.189493 [INF]   Git:      [not set]
[45703] 2021/09/29 02:23:05.189497 [INF]   Name:     NAIBOVQLOZSDIUFQYZOQUGV3PNZUT66D4WF5MKS2G7N423UGJDH2DFWG
[45703] 2021/09/29 02:23:05.189500 [INF]   ID:       NAIBOVQLOZSDIUFQYZOQUGV3PNZUT66D4WF5MKS2G7N423UGJDH2DFWG
[45703] 2021/09/29 02:23:05.190236 [INF] Listening for client connections on 0.0.0.0:4222
[45703] 2021/09/29 02:23:05.190504 [INF] Server is ready
[45703] 2021/09/29 02:23:07.111053 [TRC] 127.0.0.1:51653 - cid:4 - <<- [CONNECT {"verbose":false,"pedantic":false,"tls_required":false,"name":"NATS CLI Version 0.0.26","lang":"go","version":"1.12.0","protocol":1,"echo":true,"headers":true,"no_responders":true}]
[45703] 2021/09/29 02:23:07.111282 [TRC] 127.0.0.1:51653 - cid:4 - "v1.12.0:go:NATS CLI Version 0.0.26" - <<- [PING]
[45703] 2021/09/29 02:23:07.111301 [TRC] 127.0.0.1:51653 - cid:4 - "v1.12.0:go:NATS CLI Version 0.0.26" - ->> [PONG]
[45703] 2021/09/29 02:23:07.111632 [TRC] 127.0.0.1:51653 - cid:4 - "v1.12.0:go:NATS CLI Version 0.0.26" - <<- [SUB >  1]
[45703] 2021/09/29 02:23:07.111679 [TRC] 127.0.0.1:51653 - cid:4 - "v1.12.0:go:NATS CLI Version 0.0.26" - <<- [PING]
[45703] 2021/09/29 02:23:07.111689 [TRC] 127.0.0.1:51653 - cid:4 - "v1.12.0:go:NATS CLI Version 0.0.26" - ->> [PONG]

If you had created a subscriber, you should notice output on the subscriber telling you that it disconnected, and reconnected. The server output above is more interesting. You can see the subscriber send a CONNECT protocol message and a PING which was responded to by the server with a PONG.

You can learn more about the NATS protocol here, but more interesting than the protocol description is an interactive demo.

On a third terminal, publish your first message:

nats pub hello world -s nats://0.0.0.0:4222

On the subscriber window you should see:

[#1] Received on "hello"
world

Testing Against a Remote Server

If the NATS server were running in a different machine or a different port, you'd have to specify that to the client by specifying a NATS URL (either in a nats context or using the -s flag).

NATS URLs

NATS URLs take the form of: nats://<server>:<port> and tls://<server>:<port>. URLs with a tls protocol sport a secured TLS connection.

If you are connecting to a cluster you can specify more than one URL (comma separated). e.g. nats://localhost:4222,nats://localhost:5222,nats://localhost:6222 if you are running a test cluster of 3 nats servers on your local machine, listening at ports 4222, 5222, and 6222 respectively.

Example

nats sub -s nats://server:port ">"

If you want to try on a remote server, the NATS team maintains a demo server you can reach at demo.nats.io.

nats sub -s nats://demo.nats.io ">"

Configuring NATS Server

While the NATS server has many flags that allow for simple testing of features from command line. The standard way of configuring the NATS server product is through a configuration file. We use a simple configuration format that combines the best of traditional formats and newer styles such as JSON and YAML.

nats-server -config my-server.conf

The NATS configuration supports the following syntax:

• Lines can be commented with # and //
• Values can be assigned to properties with delimiters:
  • Equals sign: foo = 2
  • Colon: foo: 2
  • Whitespace: foo 2
• Arrays are enclosed in brackets: ["a", "b", "c"]
• Maps are enclosed in braces: {foo: 2}
• Maps can be assigned with no delimiter accounts { SYS {...}, cloud-user {...} }
• Semicolons can be optionally used as terminators host: 127.0.0.1; port: 4222;

The NATS configuration file is parsed with UTF-8 encoding.

提示

We strongly recommend using only ASCII for names and values, limiting the use of Unicode, no ASCII text to comments.

Note

The NATS configuration in the file can also be rendered as a JSON object (with comments!), but to combine it with variables the variables still have to be unquoted.

提示

JSON config files should be limited machine generated configuration files

Strings and Numbers

The configuration parser is very forgiving, as you have seen:

• values can be a primitive, or a list, or a map
• strings and numbers typically do the right thing
• numbers support units such as, 1K for 1000, 1KB for 1024

String values that start with a digit can create issues. To force such values as strings, quote them.

BAD Config:

listen: 127.0.0.1:4222
authorization: {
    # Bad - Number parsing error
    token: 3secret
}

Fixed Config:

listen: 127.0.0.1:4222
authorization: {
    # Good
    token: "3secret"
}

Variables

Server configurations can specify variables. Variables allow you to reference a value from one or more sections in the configuration.

Variables syntax:

• Are block-scoped
• Are referenced with a $ prefix. Variables in quotes blocks are ignored. For example, a usage like foo = "$VAR1" will result in foo being the literal string "$VAR1".
• Variables MUST be used to be recognized as such. The config parser will distinguish unknown field from variable by finding a reference to the variable.
• Variable reference which are not defined will be resolved from environment variables.

Variable resolution sequence:

• Look for variable in same scope
• Look for variable in parent scopes
• Look for variable in enviroment variables
• If not found stop server startup with the error below

nats-server: variable reference for 'PORT' on line 5 can not be found

注意

If the environment variable value begins with a number you may have trouble resolving it depending on the server version you are running.

# Define a variable in the config
TOKEN: "secret"

# Reference the variable
authorization {
    token: $TOKEN
}

# Define a variable in the config
# But TOKEN is never used resulting in a config parsing error
TOKEN: "secret"

# Reference the variable
authorization {
    token: "another secret"
}

unknown field "TOKEN"

A similar configuration, but this time, the variable is resolved from the environment:

export TOKEN="hello"
nats-server -c /config/file

# TOKEN is defined in the environment
authorization {
    token: $TOKEN
}

Include Directive

The include directive allows you to split a server configuration into several files. This is useful for separating configuration into chunks that you can easily reuse between different servers.

Includes must use relative paths, and are relative to the main configuration (the one specified via the -c option):

server.conf:

listen: 127.0.0.1:4222
include ./auth.conf

Note that include is not followed by = or :, as it is a directive.

auth.conf:

authorization: {
    token: "f0oBar"
}

> nats-server -c server.conf

Configuration Reloading

The config file is being read by the server on startup and is not re-scanned for changes and not locked.

A server can reload most configuration changes without requiring a server restart or clients to disconnect by sending the nats-server a signal:

nats-server --signal reload

As of NATS v2.10.0, a reload signal can be sent on a NATS service using a system account user, where <server-id> is the unique ID of the server be targeted.

nats --user sys --password sys request '$SYS.REQ.SERVER.<server-id>.RELOAD' ""

Configuration Properties

Config files have the following structure (in no specific order). All blocks and properties are optional (except host and port).

Please see sections below for links to detailed explanations of each configuration block

#General settings
host: 0.0.0.0
port: 4222

# Various server level options
# ...

# The following sections are maps with a set of (nested) properties

jetstream {
    # JetStream storage location, limits and encryption
	store_dir: nats
}

tls { 
    # Configuration map for tls parameters used for client connections, 
    # routes and https monitoring connections.
}

gateway {
    # Configuration map for gateway. Gateways are used to connected clusters.
}

leafnodes {
    # Configuration map for leafnodes. LeafNodes are lightweight clusters.
}

mqtt {
    # Configuration map for mqtt. Allow clients to connect via mqtt protocol.
} 

websocket {
    # Configuration map for websocket. Allow clients to connect via websockets.
} 

accounts {
    # List of accounts and user within accounts
    # User may have an authorization and authentication section
}

authorization { 
    # User may have an authorization and authentication section
    # This section is only useful when no accounts are defined
}

mappings {
    # Subject mappings for default account
    # When accounts are defined this section must be in the account map
}

resolver {
    # Pointer to external Authentication/Authorization resolver
    # There are multiple possible resolver type explained in their own chapters of this docuemntaion
    # memory, nats-base, url ... more may be added in the future
    # This parameter can be a value `MEMORY` for simple configuration
    # or a map of properties for connecting to the resolver
}

resolver_tls {
    # TLS configuration for an URL based resolver
}

resolver_preload {
    # List of JWT tokens to be loaded at server start.
}

Connectivity

Clustering

Subject Mappings

Note that each accounts forms its own subject namespace. Therefore the mappings section can appear on the server level (applying to the default account) or on the account level.

host: 0.0.0.0
port:4222

mappings: {
	foo: bar
}

accounts: {
    accountA: { 
	mappings: {
	    orders.acme.*: orders.$1
	}
        users: [
            {user: admin, password: admin},
            {user: user, password: user}
           ]
    },
}

Connection Timeouts

Limits

JetStream Server settings

You can enable JetStream in the server's configuration by simply adding a jetstream {} map. By default, the JetStream subsystem will store data in the /tmp directory, but you can specify the directory to use via the store_dir, as well as the limits for JetStream storage (a value of 0 means no limit).

Normally JetStream will be run in clustered mode and will replicate data, so the best place to store JetStream data would be locally on a fast SSD. One should specifically avoid NAS or NFS storage for JetStream.

注意

Note that each JetStream enabled server MUST use its own individual storage directory. JetStream replicates data between cluster nodes (up to 5 replicas), achieving redundancy and availability through this.

JetStream does not implement standby and fault tolerance through a shared file system. If a standby server shares a storage directory with an active server, you must make sure only one is active at any time. Access conflicts are not detected. We do not recommend such a setup.

Here's an example minimal file that will store data in a local "nats" directory with some limits.

$ nats-server -c js.conf

jetstream {
  store_dir: nats

  # 1GB
  max_memory_store: 1073741824

  # 10GB
  max_file_store: 10737418240
}

Global JetStream options (server level)

JetStream Account Settings

A JetStream section may also appear in accounts. JetStream is disabled by default. The minimal configuration will enable JetStream.

accounts {
  A {
      jetstream {
      }
    } 
}

Tracing open telemetry style

Enables tracing for messages marked with the traceparent header. See also: Tracing

accounts {
  A {
      msg_trace {
        dest: "trace-ot.accountA.samples"
        sampling: 5
      }
    } 
}

JetStream TPM encryption

jetstream {
  store_dir: nats
  max_file_store: 10G
  tpm {
          keys_file: "keys"
          encryption_password: "pwd"
  }
}

JetStream Server Limits

jetstream {
  store_dir: nats
  max_file_store: 10G
  limits {
      max_ack_pending: 10000
      duplicate_window: 600s
  }
}

Authentication and Authorization

Centralized Authentication and Authorization

A default NATS server will have no authentication or authorization enabled. This is useful for development and simple embedded use cases only. The default account is $G.

Once at least one user is configured in the authorization or accounts sections, the default $G account and no-authentication user are disabled. You can restore no authentication access by setting the no_auth_user.

Decentralized Authentication and Authorization

The Configuration options here refer to JWT based authentication and authorization.

Runtime Configuration

Cluster Configuration, Monitoring and Tracing

Configuring JetStream

Enabling JetStream for a nats-server

To enable JetStream in a server we have to configure it at the top level first:

jetstream: enabled

You can also use the -js, --jetstream and -sd, --store_dir <dir> flags from the command line

Multi-tenancy & Resource Mgmt

JetStream is compatible with NATS 2.0 Multi-Tenancy using Accounts. A JetStream enabled server supports creating fully isolated JetStream environments for different accounts.

JetStream environments in leaf nodes should be isolated in their own JetStream domain - Leaf nodes

This will dynamically determine the available resources. It's recommended that you set specific limits though:

jetstream {
    store_dir: /data/jetstream
    max_mem: 1G
    max_file: 100G
    domain: acme
}

Setting account resource limits

At this point JetStream will be enabled and if you have a server that does not have accounts enabled, all users in the server would have access to JetStream

jetstream {
    store_dir: /data/jetstream
    max_mem: 1G
    max_file: 100G
}

accounts {
    HR: {
        jetstream: enabled
    }
}

Here the HR account would have access to all the resources configured on the server, we can restrict it:

jetstream {
    store_dir: /data/jetstream
    max_mem: 1G
    max_file: 100G
}

accounts {
    HR: {
        jetstream {
            max_mem: 512M
            max_file: 1G
            max_streams: 10
            max_consumers: 100
        }
    }
}

Now the HR account is limited in various dimensions.

If you try to configure JetStream for an account without enabling it globally you'll get a warning and the account designated as System cannot have JetStream enabled.

Setting JetStream API and Max HA assets limits

Since version v2.10.21, the NATS JetStream API has a limit of 10K inflight requests after which it will start to drop requests in order to protect from memory buildup and to avoid overwhelming the JetStream service. Sometimes it might be necessary to reduce the limit further in order to reduce the possibility of an increase in JetStream traffic impacting the service. Another important limit is max_ha_assets which would constrain the maximum number of supported R3 or R5 streams and consumers per server:

Example:

jetstream {
  request_queue_limit: 1000
    limits {
      max_ha_assets = 2000
    }
  }

When the request limit is reached, all pending requests are dropped, therefore it may be necessary in some situations to reduce the limit further in order to lessen the impact on applications. In the logs the following would appear reporting that requests have been dropped:

[WRN] JetStream API queue limit reached, dropping 1000 requests

For an application, this will mean that those operations will error and will have to be retried. This will also emit an advisory under the subject $JS.EVENT.ADVISORY.API.LIMIT_REACHED whenever it occurs.

Operator mode account resources limits using the nsc CLI tool

If your setup is in operator mode, JetStream specific account configuration can be stored in account JWT. The earlier account named HR can be configured as follows:

nsc add account --name HR
nsc edit account --name HR --js-mem-storage 1G --js-disk-storage 512M  --js-streams 10 --js-consumer 100

Configuration Management

In many cases managing the configuration in your application code is the best model, many teams though wish to pre-create Streams and Consumers.

We support a number of tools to assist with this:

• CLI with Configuration Files
• Terraform
• GitHub Actions
• Kubernetes JetStream Controller

NATS Admin CLI

nats Admin CLI

The nats CLI can be used to manage Streams and Consumers easily using it's --config flag, for example:

Add a new Stream

This creates a new Stream based on orders.json. The orders.json file can be extracted from an existing stream using nats stream info ORDERS -j | jq .config

nats str add ORDERS --config orders.json

Edit an existing Stream

This edits an existing stream ensuring it complies with the configuration in orders.json

nats str edit ORDERS --config orders.json

Add a New Consumer

This creates a new Consumer based on orders_new.json. The orders_new.json file can be extracted from an existing stream using nats con info ORDERS NEW -j | jq .config

nats con add ORDERS NEW --config orders_new.json

Terraform

Terraform is a Cloud configuration tool from Hashicorp. We maintain a Provider for Terraform called terraform-provider-jetstream that can maintain JetStream using Terraform.

Find it in the Terraform registry.

Setup

In your project you can configure the Provider like this:

provider "jetstream" {
  servers = "connect.ngs.global"
  credentials = "ngs_jetstream_admin.creds"
}

Sample code below that creates the ORDERS example. Review the Project README for full details.

resource "jetstream_stream" "ORDERS" {
  name     = "ORDERS"
  subjects = ["ORDERS.*"]
  storage  = "file"
  max_age  = 60 * 60 * 24 * 365
}

resource "jetstream_consumer" "ORDERS_NEW" {
  stream_id      = jetstream_stream.ORDERS.id
  durable_name   = "NEW"
  deliver_all    = true
  filter_subject = "ORDERS.received"
  sample_freq    = 100
}

resource "jetstream_consumer" "ORDERS_DISPATCH" {
  stream_id      = jetstream_stream.ORDERS.id
  durable_name   = "DISPATCH"
  deliver_all    = true
  filter_subject = "ORDERS.processed"
  sample_freq    = 100
}

resource "jetstream_consumer" "ORDERS_MONITOR" {
  stream_id        = jetstream_stream.ORDERS.id
  durable_name     = "MONITOR"
  deliver_last     = true
  ack_policy       = "none"
  delivery_subject = "monitor.ORDERS"
}

output "ORDERS_SUBJECTS" {
  value = jetstream_stream.ORDERS.subjects
}

GitHub Actions

We have a pack of GitHub Actions that let you manage an already running JetStream Server, useful for managing releases or standing up test infrastructure.

Full details and examples are in the jetstream-gh-actions repository, here's an example.

on: push
name: orders
jobs:

  # First we delete the ORDERS stream and consumer if they already exist
  clean_orders:
    runs-on: ubuntu-latest
    steps:
      - name: orders_stream
        uses: nats-io/jetstream-gh-action/delete/stream@master
        with:
          missing_ok: 1
          stream: ORDERS
          server: js.example.net

  # Now we create the Stream and Consumers using the same configuration files the 
  # nats CLI utility would use as shown above
  create_orders:
    runs-on: ubuntu-latest
    needs: clean_orders
    steps:
      - uses: actions/checkout@master
      - name: orders_stream
        uses: nats-io/jetstream-gh-action/create/stream@master
        with:
          config: ORDERS.json
          server: js.example.net
      - name: orders_new_consumer
        uses: nats-io/jetstream-gh-action/create/consumer@master
        with:
          config: ORDERS_NEW.json
          stream: ORDERS
          server: js.example.net

  # We publish a message to a specific Subject, perhaps some consumer is 
  # waiting there for it to kick off tests
  publish_message:
    runs-on: ubuntu-latest
    needs: create_orders
    steps:
      - uses: actions/checkout@master
      - name: orders_new_consumer
        uses: nats-io/jetstream-gh-action@master
        with:
          subject: ORDERS.deployment
          message: Published new deployment via "${{ github.event_name }}" in "${{ github.repository }}"
          server: js.example.net

Kubernetes Controller

The JetStream controllers allow you to manage NATS JetStream Streams and Consumers via K8S CRDs. You can find more info on how to deploy and usage here. Below you can find an example of how to create a stream and a couple of consumers:

---
apiVersion: jetstream.nats.io/v1beta1
kind: Stream
metadata:
  name: mystream
spec:
  name: mystream
  subjects: ["orders.*"]
  storage: memory
  maxAge: 1h
---
apiVersion: jetstream.nats.io/v1beta1
kind: Consumer
metadata:
  name: my-push-consumer
spec:
  streamName: mystream
  durableName: my-push-consumer
  deliverSubject: my-push-consumer.orders
  deliverPolicy: last
  ackPolicy: none
  replayPolicy: instant
---
apiVersion: jetstream.nats.io/v1beta1
kind: Consumer
metadata:
  name: my-pull-consumer
spec:
  streamName: mystream
  durableName: my-pull-consumer
  deliverPolicy: all
  filterSubject: orders.received
  maxDeliver: 20
  ackPolicy: explicit

Once the CRDs are installed you can use kubectl to manage the streams and consumers as follows:

$ kubectl get streams
NAME       STATE     STREAM NAME   SUBJECTS
mystream   Created   mystream      [orders.*]

$ kubectl get consumers
NAME               STATE     STREAM     CONSUMER           ACK POLICY
my-pull-consumer   Created   mystream   my-pull-consumer   explicit
my-push-consumer   Created   mystream   my-push-consumer   none

# If you end up in an Errored state, run kubectl describe for more info.
#     kubectl describe streams mystream
#     kubectl describe consumers my-pull-consumer

Clustering

NATS Server Clustering

NATS supports running each server in clustered mode. You can cluster servers together for high volume messaging systems and resiliency and high availability.

NATS servers achieve this by gossiping about and connecting to, all of the servers they know, thus dynamically forming a full mesh. Once clients connect or re-connect to a particular server, they are informed about current cluster members. Because of this behavior, a cluster can grow, shrink and self heal. The full mesh does not necessarily have to be explicitly configured either.

Note that NATS clustered servers have a forwarding limit of one hop. This means that each nats-server instance will only forward messages that it has received from a client to the immediately adjacent nats-server instances to which it has routes. Messages received from a route will only be distributed to local clients.

For the cluster to successfully form a full mesh and NATS to function as intended and described throughout the documentation - temporary errors permitting - it is necessary that servers can connect to each other and that clients can connect to each server in the cluster.

Cluster URLs

In addition to a port to listen for clients, nats-server listens on a "cluster" URL (the -cluster option). Additional nats-server servers can then add that URL to their -routes argument to join the cluster. These options can also be specified in a config file, but only the command-line version is shown in this overview for simplicity.

Running a Simple Cluster

Here is a simple cluster running on the same machine:

Server A - the 'seed server'

nats-server -p 4222 -cluster nats://localhost:4248 --cluster_name test-cluster

Server B

nats-server -p 5222 -cluster nats://localhost:5248 -routes nats://localhost:4248 --cluster_name test-cluster

Check the output of the server for the selected client and route ports.

Server C

nats-server -p 6222 -cluster nats://localhost:6248 -routes nats://localhost:4248 --cluster_name test-cluster

Check the output of the server for the selected client and route ports.

Each server has a client and cluster port specified. Servers with the routes option establish a route to the seed server. Because the clustering protocol gossips members of the cluster, all servers are able to discover other server in the cluster. When a server is discovered, the discovering server will automatically attempt to connect to it in order to form a full mesh. Typically only one instance of the server will run per machine, so you can reuse the client port (4222) and the cluster port (4248), and simply the route to the host/port of the seed server.

Similarly, clients connecting to any server in the cluster will discover other servers in the cluster. If the connection to the server is interrupted, the client will attempt to connect to all other known servers.

There is no explicit configuration for seed server. They simply serve as the starting point for server discovery by other members of the cluster as well as clients. As such these are the servers that clients have in their list of connect urls and cluster members have in their list of routes. They reduce configuration as not every server needs to be in these lists. But the ability for other server and clients to successfully connect depends on seed server running. If multiple seed server are used, they make use of the routes option as well, so they can establish routes to one another.

Command Line Options

The following cluster options are supported:

--routes [rurl-1, rurl-2]     Routes to solicit and connect
--cluster nats://host:port    Cluster URL for solicited routes

When a NATS server routes to a specified URL, it will advertise its own cluster URL to all other servers in the route effectively creating a routing mesh to all other servers.

Note: when using the -routes option, you must also specify a -cluster option.

Clustering can also be configured using the server config file.

Three Server Cluster Example

The following example demonstrates how to run a cluster of 3 servers on the same host. We will start with the seed server and use the -D command line parameter to produce debug information.

nats-server -p 4222 -cluster nats://localhost:4248 -D

Alternatively, you could use a configuration file, let's call it seed.conf, with a content similar to this:

# Cluster Seed Node

listen: 127.0.0.1:4222
http: 8222

cluster {
  listen: 127.0.0.1:4248
}

And start the server like this:

nats-server -config ./seed.conf -D

This will produce an output similar to:

[83329] 2020/02/12 16:04:52.369039 [INF] Starting nats-server version 2.1.4
[83329] 2020/02/12 16:04:52.369130 [DBG] Go build version go1.13.6
[83329] 2020/02/12 16:04:52.369133 [INF] Git commit [not set]
[83329] 2020/02/12 16:04:52.369360 [INF] Starting http monitor on 127.0.0.1:8222
[83329] 2020/02/12 16:04:52.369436 [INF] Listening for client connections on 127.0.0.1:4222
[83329] 2020/02/12 16:04:52.369441 [INF] Server id is NDSGCS74MG5ZUMBOVWOUJ5S3HIOW
[83329] 2020/02/12 16:04:52.369443 [INF] Server is ready
[83329] 2020/02/12 16:04:52.369534 [INF] Listening for route connections on 127.0.0.1:4248

It is also possible to specify the hostname and port independently. At the minimum, the port is required. If you leave the hostname off it will bind to all the interfaces ('0.0.0.0').

cluster {
  host: 127.0.0.1
  port: 4248
}

Now let's start two more servers, each one connecting to the seed server.

nats-server -p 5222 -cluster nats://localhost:5248 -routes nats://localhost:4248 -D

When running on the same host, we need to pick different ports for the client connections -p, and for the port used to accept other routes -cluster. Note that -routes points to the -cluster address of the seed server (localhost:4248).

Here is the log produced. See how it connects and registers a route to the seed server (...GzM).

[83330] 2020/02/12 16:05:09.661047 [INF] Starting nats-server version 2.1.4
[83330] 2020/02/12 16:05:09.661123 [DBG] Go build version go1.13.6
[83330] 2020/02/12 16:05:09.661125 [INF] Git commit [not set]
[83330] 2020/02/12 16:05:09.661341 [INF] Listening for client connections on 0.0.0.0:5222
[83330] 2020/02/12 16:05:09.661347 [INF] Server id is NAABC2CKRVPZBIECMLZZA6L3PK
[83330] 2020/02/12 16:05:09.661349 [INF] Server is ready
[83330] 2020/02/12 16:05:09.662429 [INF] Listening for route connections on localhost:5248
[83330] 2020/02/12 16:05:09.662676 [DBG] Trying to connect to route on localhost:4248
[83330] 2020/02/12 16:05:09.663308 [DBG] 127.0.0.1:4248 - rid:1 - Route connect msg sent
[83330] 2020/02/12 16:05:09.663370 [INF] 127.0.0.1:4248 - rid:1 - Route connection created
[83330] 2020/02/12 16:05:09.663537 [DBG] 127.0.0.1:4248 - rid:1 - Registering remote route "NDSGCS74MG5ZUMBOVWOUJ5S3HIOW"
[83330] 2020/02/12 16:05:09.663549 [DBG] 127.0.0.1:4248 - rid:1 - Sent local subscriptions to route

From the seed's server log, we see that the route is indeed accepted:

[83329] 2020/02/12 16:05:09.663386 [INF] 127.0.0.1:62941 - rid:1 - Route connection created
[83329] 2020/02/12 16:05:09.663665 [DBG] 127.0.0.1:62941 - rid:1 - Registering remote route "NAABC2CKRVPZBIECMLZZA6L3PK"
[83329] 2020/02/12 16:05:09.663681 [DBG] 127.0.0.1:62941 - rid:1 - Sent local subscriptions to route

Finally, let's start the third server:

nats-server -p 6222 -cluster nats://localhost:6248 -routes nats://localhost:4248 -D

Again, notice that we use a different client port and cluster address, but still point to the same seed server at the address nats://localhost:4248:

[83331] 2020/02/12 16:05:12.838022 [INF] Listening for client connections on 0.0.0.0:6222
[83331] 2020/02/12 16:05:12.838029 [INF] Server id is NBE7SLUDLFIMHS2U6347N3DQEJ
[83331] 2020/02/12 16:05:12.838031 [INF] Server is ready
...
[83331] 2020/02/12 16:05:12.839203 [INF] Listening for route connections on localhost:6248
[83331] 2020/02/12 16:05:12.839453 [DBG] Trying to connect to route on localhost:4248
[83331] 2020/02/12 16:05:12.840112 [DBG] 127.0.0.1:4248 - rid:1 - Route connect msg sent
[83331] 2020/02/12 16:05:12.840198 [INF] 127.0.0.1:4248 - rid:1 - Route connection created
[83331] 2020/02/12 16:05:12.840324 [DBG] 127.0.0.1:4248 - rid:1 - Registering remote route "NDSGCS74MG5ZUMBOVWOUJ5S3HIOW"
[83331] 2020/02/12 16:05:12.840342 [DBG] 127.0.0.1:4248 - rid:1 - Sent local subscriptions to route
[83331] 2020/02/12 16:05:12.840717 [INF] 127.0.0.1:62946 - rid:2 - Route connection created
[83331] 2020/02/12 16:05:12.840906 [DBG] 127.0.0.1:62946 - rid:2 - Registering remote route "NAABC2CKRVPZBIECMLZZA6L3PK"
[83331] 2020/02/12 16:05:12.840915 [DBG] 127.0.0.1:62946 - rid:2 - Sent local subscriptions to route

First a route is created to the seed server (...IOW) and after that, a route from ...3PK - which is the ID of the second server - is accepted.

The log from the seed server shows that it accepted the route from the third server:

[83329] 2020/02/12 16:05:12.840111 [INF] 127.0.0.1:62945 - rid:2 - Route connection created
[83329] 2020/02/12 16:05:12.840350 [DBG] 127.0.0.1:62945 - rid:2 - Registering remote route "NBE7SLUDLFIMHS2U6347N3DQEJ"
[83329] 2020/02/12 16:05:12.840363 [DBG] 127.0.0.1:62945 - rid:2 - Sent local subscriptions to route

And the log from the second server shows that it connected to the third.

[83330] 2020/02/12 16:05:12.840529 [DBG] Trying to connect to route on 127.0.0.1:6248
[83330] 2020/02/12 16:05:12.840684 [DBG] 127.0.0.1:6248 - rid:2 - Route connect msg sent
[83330] 2020/02/12 16:05:12.840695 [INF] 127.0.0.1:6248 - rid:2 - Route connection created
[83330] 2020/02/12 16:05:12.840814 [DBG] 127.0.0.1:6248 - rid:2 - Registering remote route "NBE7SLUDLFIMHS2U6347N3DQEJ"
[83330] 2020/02/12 16:05:12.840827 [DBG] 127.0.0.1:6248 - rid:2 - Sent local subscriptions to route

At this point, there is a full mesh cluster of NATS servers.

Testing the Cluster

Now, the following should work: make a subscription to the first server (port 4222). Then publish to each server (ports 4222, 5222, 6222). You should be able to receive messages without problems.

Testing server A

nats sub -s "nats://127.0.0.1:4222" hello &
nats pub -s "nats://127.0.0.1:4222" hello world_4222

23:34:45 Subscribing on hello
23:34:45 Published 10 bytes to "hello"

[#1] Received on "hello"
world_4222

Testing server B

nats pub -s "nats://127.0.0.1:5222" hello world_5222

[#2] Received on "hello"
23:36:09 Published 10 bytes to "hello"
world_5222

Testing server C

nats pub -s "nats://127.0.0.1:6222" hello world_6222

23:38:40 Published 10 bytes to "hello"
[#3] Received on "hello"
world_6222

Testing using seed (i.e. A, B and C) server URLs

nats pub -s "nats://127.0.0.1:4222,nats://127.0.0.1:5222,nats://127.0.0.1:6222" hello whole_world

[#4] Received on "hello"
23:39:16 Published 11 bytes to "hello"
whole_world

Clustering Configuration

The cluster configuration map has the following configuration options:

cluster {
  name: example

  # host/port for inbound route connections from other server
  listen: localhost:4244

  # Authorization for route connections
  # Other server can connect if they supply the credentials listed here
  # This server will connect to discovered routes using this user
  authorization {
    user: route_user
    password: pwd
    timeout: 0.5
  }

  # This server establishes routes with these server.
  # This server solicits new routes and Routes are actively solicited and connected to from this server.
  # Other servers can connect to us if they supply the correct credentials
  # in their routes definitions from above.
  routes = [
    nats://route_user:pwd@127.0.0.1:4245
    nats://route_user:pwd@127.0.0.1:4246
  ]
}

v2 Routes

Introduced in NATS v2.10.0

Connection pooling

Before the v2.10.0 release, two servers in a cluster had only one connection to transmit all messages for all accounts, which could lead to slow down and increased memory usage when data was not transmitted fast enough.

The v2.10.0 release introduces the ability to have multiple route connections between two servers. By default, without any configuration change, clustering two v2.10.0+ servers together will create 3 route connections. This can of course be configured to a different value by explicitly configuring the pool size.

cluster {
  pool_size: 3
}

Pool size requirements

Each of those connections will handle a specific subset of the accounts, and the assignment of an account to a specific connection index in the pool is the same in any server in the cluster. It is required that each server in the cluster have the same pool size value, otherwise, clustering will fail to be established with an error similar to:

[ERR] 127.0.0.1:6222 - rid:6 - Mismatch route pool size: 3 vs 4

Handling loss of connection

In the event that a given connection of the pool breaks, automatic reconnection occurs as usual. However, while the disconnection is happening, traffic for accounts handled by that connection is stopped (that is, traffic is not routed through other connections), the same way that it was when there was a single route connection.

Configuration reload

Note that although the cluster's pool_size configuration parameter can be changed through a configuration reload, connections between servers will likely break since there will be a mismatch between servers. It is possible though to do a rolling reload by setting the same value on all servers. Client and other connections (including dedicated account routes - see next section) will not be closed.

Monitoring

When monitoring is enabled, you will see that the /routez page now has more connections than before, which is expected.

Disabling connection pooling

It is possible to disable route connection pooling by setting the pool_size configuration parameter to the value -1. When that is the case, a server will behave as a server pre-v2.10.0 and create a single route connection between its peers.

Note that in that mode, no accounts list can be defined (see "Accounts Pinning" below).

Account pinning

In addition to connection pooling, the release v2.10.0 has introduced the ability to configure a list of accounts that will have a dedicated route connection.

cluster {
  accounts: [acc1, acc2]
}

Note that by default, the server will create a dedicated route for the system account (no specific configuration is needed).

Having a dedicated route improves performance and reduces latency, but another benefit is that since the route is dedicated to an account, the account name does not need to be added to the message route protocols. Since account names can be quite long, this reduces the number of bytes that need to be transmitted in all other cases.

Handling loss of connection

In the event that an account route connection breaks, automatic reconnection occurs as usual. However, while the disconnection is happening, traffic for this account is stopped.

Configuration reload

The accounts list can be modified and a configuration signal be sent to the server. All servers need to have the same list, however, it is possible to perform a rolling configuration reload.

For instance, adding an account to the list of server A and issuing a configuration reload will not produce an error, even though the other server in the cluster does not have that account in the list yet. A dedicated connection will not yet be established, but traffic for this account in the pooled connection currently handling it will stop. When the configuration reload happens on the other server, a dedicated connection will then be established and this account’s traffic will resume.

When removing an account from the list and issuing a configuration reload, the connection for this account will be closed, and traffic for this account will stop. Other server(s) that still have this account configured with a dedicated connection will fail to reconnect. When they are also sent the configuration reload (with updated accounts configuration), the account traffic will now be handled by a connection in the pool.

Note that configuration reload of changes in the accounts list do not affect existing pool connections, and therefore should not affect traffic for other accounts.

Monitoring

When monitoring is enabled, the route connection's information now has a new field called account that displays the name of the account this route is for.

Disabling connection pooling

As indicated in the connection pooling section, if pool_size is set to -1, the accounts list cannot be configured nor would be in use.

Connecting to an older server

Although it is recommended that all servers part of the same cluster be at the same version number, a v2.10.0 server will be able to connect to an older server and in that case create a single route to that server. This allows for an easy deployment of a v2.10.0 release into an existing cluster running an older release.

Compression

Release v2.10.0 introduces the ability to configure compression between servers having route connections. The current compression algorithm used is S2, an extension of Snappy. By default, routes do not use compression and it needs to be explicitly enabled.

cluster {
  compression: {
    mode: accept
  }
}

Compression Modes

There are several modes of compression and there is no requirement to have the same mode between routed servers.

• off - Explicitly disables compression for any route between the server and a peer.
• accept (default) - Does not initiate compression, but will accept the compression mode of the peer it connects to.
• s2_fast - Applies compression, but optimizes for speed over compression ratio.
• s2_better - Applies compression, providing a balance of speed and compression ratio.
• s2_best - Applies compression and optimizes for compression ratio over speed.
• s2_auto - Choose the appropriate s2_* mode relative to the round-trip time (RTT) measured between the server and the peer. See rtt_thresholds below.

Round-trip time thresholds

When s2_auto compression is used, it relies on a rtt_thresholds option, which is a list of three latency thresholds that dictate increasing or decreasing the compression mode.

cluster {
  compression: {
    mode: s2_auto
    rtt_thresholds: [10ms, 50ms, 100ms]
  }
}

The default rtt_thresholds value is [10ms, 50ms, 100ms]. The way to read this is that if the RTT is under 10ms, no compression is applied. Once 10ms is reached, s2_fast is applied and so on with the remaining two thresholds for s2_better and s2_best.

Configuration reload

The compression configuration can be changed through configuration reload. If the value is changed from off to anything else, then connections are closed and recreated, same as if the compression mode was set to something and disabled by setting it to off.

For all other compression modes, the mode is changed dynamically without a need to close the route connections.

Monitoring

When monitoring is enabled, the route connection's information now has a new field called compression that displays the current compression mode. It can be off or any other mode described above. Note that s2_auto is not displayed, instead, what will be displayed is the current mode, say s2_best or s2_uncompressed.

If connected to an older server, the compression field will display not supported.

Connecting to an older server

It is possible to have a v2.10.0+ server, with a compression mode configured, connect to an older server that does not support compression. The connection will simply not use compression.

JetStream Clustering

Clustering in JetStream is required for a highly available and scalable system. Behind clustering is RAFT. There's no need to understand RAFT in depth to use clustering, but knowing a little explains some of the requirements behind setting up JetStream clusters.

RAFT

JetStream uses a NATS optimized RAFT algorithm for clustering. Typically RAFT generates a lot of traffic, but the NATS server optimizes this by combining the data plane for replicating messages with the messages RAFT would normally use to ensure consensus. Each server participating requires an unique server_name (only applies within the same domain).

RAFT Groups

The RAFT groups include API handlers, streams, consumers, and an internal algorithm designates which servers handle which streams and consumers.

The RAFT algorithm has a few requirements:

• A log to persist state
• A quorum for consensus

The Quorum

In order to ensure data consistency across complete restarts, a quorum of servers is required. A quorum is ½ cluster size + 1. This is the minimum number of nodes to ensure at least one node has the most recent data and state after a catastrophic failure. So for a cluster size of 3, you’ll need at least two JetStream enabled NATS servers available to store new messages. For a cluster size of 5, you’ll need at least 3 NATS servers, and so forth.

RAFT Groups

Meta Group - all servers join the Meta Group and the JetStream API is managed by this group. A leader is elected and this owns the API and takes care of server placement.

Stream Group - each Stream creates a RAFT group, this group synchronizes state and data between its members. The elected leader handles ACKs and so forth, if there is no leader the stream will not accept messages.

Consumer Group - each Consumer creates a RAFT group, this group synchronizes consumer state between its members. The group will live on the machines where the Stream Group is and handle consumption ACKs etc. Each Consumer will have their own group.

Cluster Size

Generally, we recommend 3 or 5 JetStream enabled servers in a NATS cluster. This balances scalability with a tolerance for failure. For example, if 5 servers are JetStream enabled you would want two servers in one “zone”, two servers in another, and the remaining server in a third. This means you can lose any one “zone” at any time and continue operating.

Mixing JetStream enabled servers with standard NATS servers

This is possible and even recommended in some cases. By mixing server types you can dedicate certain machines optimized for storage for Jetstream and others optimized solely for compute for standard NATS servers, reducing operational expense. With the right configuration, the standard servers would handle non-persistent NATS traffic and the JetStream enabled servers would handle JetStream traffic.

Configuration

To configure JetStream clusters, just configure clusters as you normally would by specifying a cluster block in the configuration. Any JetStream enabled servers in the list of clusters will automatically chatter and set themselves up. Unlike core NATS clustering though, each JetStream node must specify a server name and cluster name.

Below are explicitly listed server configuration for a three-node cluster across three machines, n1-c1, n2-c1, and n3-c1.

Server password configuration

A user and password under the system account ($SYS) should be configured. The following configuration uses a bcrypted password: a very long s3cr3t! password.

Server 1 (host_a)

server_name=n1-c1
listen=4222

accounts {
  $SYS {
    users = [
      { user: "admin",
        pass: "$2a$11$DRh4C0KNbNnD8K/hb/buWe1zPxEHrLEiDmuq1Mi0rRJiH/W25Qidm"
      }
    ]
  }
}

jetstream {
   store_dir=/nats/storage
}

cluster {
  name: C1
  listen: 0.0.0.0:6222
  routes: [
    nats://host_b:6222
    nats://host_c:6222
  ]
}

Server 2 (host_b)

server_name=n2-c1
listen=4222

accounts {
  $SYS {
    users = [
      { user: "admin",
        pass: "$2a$11$DRh4C0KNbNnD8K/hb/buWe1zPxEHrLEiDmuq1Mi0rRJiH/W25Qidm"
      }
    ]
  }
}

jetstream {
   store_dir=/nats/storage
}

cluster {
  name: C1
  listen: 0.0.0.0:6222
  routes: [
    nats://host_a:6222
    nats://host_c:6222
  ]
}

Server 3 (host_c)

server_name=n3-c1
listen=4222

accounts {
  $SYS {
    users = [
      { user: "admin",
        pass: "$2a$11$DRh4C0KNbNnD8K/hb/buWe1zPxEHrLEiDmuq1Mi0rRJiH/W25Qidm"
      }
    ]
  }
}

jetstream {
   store_dir=/nats/storage
}

cluster {
  name: C1
  listen: 0.0.0.0:6222
  routes: [
    nats://host_a:6222
    nats://host_b:6222
  ]
}

Add nodes as necessary. Choose a data directory that makes sense for your environment, ideally a fast SSD, and launch each server. After two servers are running you'll be ready to use JetStream.

Administration

Once a JetStream cluster is operating interactions with the CLI and with nats CLI is the same as before. For these examples, lets assume we have a 5 server cluster, n1-n5 in a cluster named C1.

Account Level

Within an account there are operations and reports that show where users data is placed and which allow them some basic interactions with the RAFT system.

Creating clustered streams

When adding a stream using the nats CLI the number of replicas will be asked, when you choose a number more than 1, (we suggest 1, 3 or 5), the data will be stored on multiple nodes in your cluster using the RAFT protocol as above.

nats stream add ORDERS --replicas 3

Example output extract:

....
Information for Stream ORDERS created 2021-02-05T12:07:34+01:00
....
Configuration:
....
             Replicas: 3

Cluster Information:

                 Name: C1
               Leader: n1-c1
              Replica: n4-c1, current, seen 0.07s ago
              Replica: n3-c1, current, seen 0.07s ago

Above you can see that the cluster information will be reported in all cases where Stream info is shown such as after add or using nats stream info.

Here we have a stream in the NATS cluster C1, its current leader is a node n1-c1 and it has 2 followers - n4-c1 and n3-c1.

The current indicates that followers are up to date and have all the messages, here both cluster peers were seen very recently.

The replica count can be edited once configured.

Viewing Stream Placement and Stats

Users can get overall statistics about their streams and also where these streams are placed:

nats stream report

Obtaining Stream stats
+----------+-----------+----------+--------+---------+------+---------+----------------------+
| Stream   | Consumers | Messages | Bytes  | Storage | Lost | Deleted | Cluster              |
+----------+-----------+----------+--------+---------+------+---------+----------------------+
| ORDERS   | 4         | 0        | 0 B    | File    | 0    | 0       | n1-c1*, n2-c1, n3-c1 |
| ORDERS_3 | 4         | 0        | 0 B    | File    | 0    | 0       | n1-c1*, n2-c1, n3-c1 |
| ORDERS_4 | 4         | 0        | 0 B    | File    | 0    | 0       | n1-c1*, n2-c1, n3-c1 |
| ORDERS_5 | 4         | 0        | 0 B    | File    | 0    | 0       | n1-c1, n2-c1, n3-c1* |
| ORDERS_2 | 4         | 1,385    | 13 MiB | File    | 0    | 1       | n1-c1, n2-c1, n3-c1* |
| ORDERS_0 | 4         | 1,561    | 14 MiB | File    | 0    | 0       | n1-c1, n2-c1*, n3-c1 |
+----------+-----------+----------+--------+---------+------+---------+----------------------+

Forcing Stream and Consumer leader election

Every RAFT group has a leader that's elected by the group when needed. Generally there is no reason to interfere with this process, but you might want to trigger a leader change at a convenient time. Leader elections will represent short interruptions to the stream so if you know you will work on a node later it might be worth moving leadership away from it ahead of time.

Moving leadership away from a node does not remove it from the cluster and does not prevent it from becoming a leader again, this is merely a triggered leader election.

nats stream cluster step-down ORDERS

14:32:17 Requesting leader step down of "n1-c1" in a 3 peer RAFT group
14:32:18 New leader elected "n4-c1"

Information for Stream ORDERS created 2021-02-05T12:07:34+01:00
...
Cluster Information:

                 Name: c1
               Leader: n4-c1
              Replica: n1-c1, current, seen 0.12s ago
              Replica: n3-c1, current, seen 0.12s ago

The same is true for consumers, nats consumer cluster step-down ORDERS NEW.

System Level

Systems users can view state of the Meta Group - but not individual Stream or Consumers.

Viewing the cluster state

We have a high level report of cluster state:

nats server report jetstream --user admin --password s3cr3t!

+--------------------------------------------------------------------------------------------------+
|                                        JetStream Summary                                         |
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
| Server | Cluster | Streams | Consumers | Messages | Bytes  | Memory | File   | API Req | API Err |
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
| n3-c2  | c2      | 0       | 0         | 0        | 0 B    | 0 B    | 0 B    | 1       | 0       |
| n3-c1  | c1      | 6       | 24        | 2,946    | 27 MiB | 0 B    | 27 MiB | 3       | 0       |
| n2-c2  | c2      | 0       | 0         | 0        | 0 B    | 0 B    | 0 B    | 3       | 0       |
| n1-c2  | c2      | 0       | 0         | 0        | 0 B    | 0 B    | 0 B    | 14      | 2       |
| n2-c1  | c1      | 6       | 24        | 2,946    | 27 MiB | 0 B    | 27 MiB | 15      | 0       |
| n1-c1* | c1      | 6       | 24        | 2,946    | 27 MiB | 0 B    | 27 MiB | 31      | 0       |
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
|        |         | 18      | 72        | 8,838    | 80 MiB | 0 B    | 80 MiB | 67      | 2       |
+--------+---------+---------+-----------+----------+--------+--------+--------+---------+---------+
+---------------------------------------------------+
|            RAFT Meta Group Information            |
+-------+--------+---------+---------+--------+-----+
| Name  | Leader | Current | Offline | Active | Lag |
+-------+--------+---------+---------+--------+-----+
| n1-c1 | yes    | true    | false   | 0.00s  | 0   |
| n1-c2 |        | true    | false   | 0.05s  | 0   |
| n2-c1 |        | false   | true    | 9.00s  | 2   |
| n2-c2 |        | true    | false   | 0.05s  | 0   |
| n3-c1 |        | true    | false   | 0.05s  | 0   |
| n3-c2 |        | true    | false   | 0.05s  | 0   |
+-------+--------+---------+---------+--------+-----+

This is a full cluster wide report, the report can be limited to a specific account using --account.

Here we see the distribution of streams, messages, api calls etc by across 2 super clusters and an overview of the RAFT meta group.

In the Meta Group report the server n2-c1 is not current and has not been seen for 9 seconds, it's also behind by 2 raft operations.

This report is built using raw data that can be obtained from the monitor port on the /jsz url, or over nats using:

nats server req jetstream --user admin --password s3cr3t! --help

usage: nats server request jetstream [<flags>] [<wait>]

Show JetStream details

Flags:
  -h, --help                    Show context-sensitive help (also try --help-long and --help-man).
      --version                 Show application version.
  -s, --server=NATS_URL         NATS server urls
      --user=NATS_USER          Username or Token
      --password=NATS_PASSWORD  Password
      --creds=NATS_CREDS        User credentials
      --nkey=NATS_NKEY          User NKEY
      --tlscert=NATS_CERT       TLS public certificate
      --tlskey=NATS_KEY         TLS private key
      --tlsca=NATS_CA           TLS certificate authority chain
      --timeout=NATS_TIMEOUT    Time to wait on responses from NATS
      --js-api-prefix=PREFIX    Subject prefix for access to JetStream API
      --js-event-prefix=PREFIX  Subject prefix for access to JetStream Advisories
      --js-domain=DOMAIN        JetStream domain to access
      --context=CONTEXT         Configuration context
      --trace                   Trace API interactions
      --limit=2048              Limit the responses to a certain amount of records
      --offset=0                Start at a certain record
      --name=NAME               Limit to servers matching a server name
      --host=HOST               Limit to servers matching a server host name
      --cluster=CLUSTER         Limit to servers matching a cluster name
      --tags=TAGS ...           Limit to servers with these configured tags
      --account=ACCOUNT         Show statistics scoped to a specific account
      --accounts                Include details about accounts
      --streams                 Include details about Streams
      --consumer                Include details about Consumers
      --config                  Include details about configuration
      --leader                  Request a response from the Meta-group leader only
      --all                     Include accounts, streams, consumers and configuration

Args:
  [<wait>]  Wait for a certain number of responses

nats server req jetstream --user admin --password s3cr3t! --leader

This will produce a wealth of raw information about the current state of your cluster - here requesting it from the leader only.

Forcing Meta Group leader election

Similar to Streams and Consumers above the Meta Group allows leader stand down. The Meta Group is cluster wide and spans all accounts, therefore to manage the meta group you have to use a SYSTEM user.

nats server cluster step-down --user admin --password s3cr3t!

17:44:24 Current leader: n2-c2
17:44:24 New leader: n1-c2

Evicting a peer

Generally when shutting down NATS, including using Lame Duck Mode, the cluster will notice this and continue to function.

There might be a case though where you know a node will never return, and you want to signal to JetStream that the node will not return. A peer-remove will remove that node from the Stream in question and all its Consumers.

After the node is removed the cluster will notice that the replica count of a stream is not honored anymore and will immediately pick a new node and start replicating data to it. The new node will be selected using the same placement rules as the existing stream.

nats server cluster peer-remove n4-c1 --user admin --password s3cr3t!

? Really remove offline peer n4-c1 (y/N)

警告

Peer-removing nodes from the cluster is a destructive operation, and decreases the size of the cluster. A server that is to be peer-removed should ideally already be offline. It can be performed with nodes that are still online, but in this case, JetStream will be disabled on those nodes. The server should be shut down and not restarted, or if it is, JetStream should be disabled. The server may not come back under the same server_name if it was peer-removed and its disk wiped. This means the configured server_name will need to be changed to a new value before restarting.

Alternatively, if you're intending for the node to remain and only move a stream off of a specific node, you can peer-remove a node on the stream-level.

nats stream cluster peer-remove ORDERS

? Select a Peer n4-c1
14:38:50 Removing peer "n4-c1"
14:38:50 Requested removal of peer "n4-c1"

At this point the stream and all consumers will have removed n4-c1 from the group. A new node will be selected, and data will be replicated to it. In this case n2-c1 is selected as a new peer.

$ nats stream info ORDERS

....
Cluster Information:

                 Name: c1
               Leader: n3-c1
              Replica: n1-c1, current, seen 0.02s ago
              Replica: n2-c1, outdated, seen 0.42s ago

We can see a new replica was picked, the stream is back to replication level of 3 and n4-c1 is not active any more in this Stream or any of its Consumers.

Troubleshooting

Diagnosing problems in NATS JetStream clusters requires:

• knowledge of JetStream concepts
• knowledge of the NATS Command Line Interface (CLI)

The following tips and commands (while not an exhaustive list) can be useful when diagnosing problems in NATS JetStream clusters:

Troubleshooting tips

1. Look at nats-server logs. By default, only warning and error logs are produced, but debug and trace logs can be turned on from the command line using -D and -DV, respectively. Alternatively, enabling debug or trace in the server config.
2. Make sure that in the NATS JetStream configuration, at least one system user is configured in this section: { $SYS { users } }.

nats account commands

Basic nats server commands

nats server report commands

nats server request commands

nats server cluster commands

Experimental commands

Further troubleshooting references

• Testing your setup

Super-cluster with Gateways

Gateways

Gateways enable connecting one or more clusters together into a full mesh; they allow the formation of superclusters from smaller clusters. Cluster and Gateway protocols listen on different ports. Clustering is used for adjacent servers; gateways are for joining clusters together.

Gateway configuration is similar to clustering:

• gateways have a dedicated port where they listen for gateway requests
• gateways gossip gateway nodes and remote discovered gateways

Unlike clusters, gateways:

• have names, specifying the cluster they are in
• don't form a full mesh between gateway nodes, but form a full mesh between clusters instead
• are bound by uni-directional connections
• don't gossip gateway nodes to clients

Gateways exist to:

• reduce the number of connections required between servers
• optimize the interest graph propagation

If gateways are to be used in a cluster, all servers of this cluster need to have a gateway configuration with the same name.
Furthermore, every gateway node needs to be able to connect to any other gateway node and vice versa. Everything else is considered a misconfiguration.

Gateway Connections

A nats-server in a gateway role will specify a port where it will accept gateway connections. If the configuration specifies other external gateways, the gateway will create one outbound gateway connection for each gateway in its configuration. It will also gossip other gateways it knows or discovers. Fewer external gateways mean less configuration. Yet, the ability to discover more gateways and gateway nodes depends on these servers running. This is similar to seed server in cluster. It is recommended to have all seed server of a cluster listed in the gateways section.

If the local cluster has three gateway nodes, this means there will be three outbound connections from the local cluster to each external gateway cluster.

In the example above cluster A has configured gateway connections for B (solid lines). B has discovered gateway connections to A (dotted lines). Note that the number of outgoing connections always matches the number of gateways with the same name.

In this second example, again configured connections are shown with solid lines and discovered gateway connections are shown using dotted lines. Gateways A and C were both discovered via gossiping; B discovered A and A discovered C.

A key point in the description above is that each node in the cluster will make a connection to a single node in every remote cluster — a difference from the clustering protocol, where every node is directly connected to all other nodes.

For those mathematically inclined, cluster connections are N(N-1)/2 where N is the number of nodes in the cluster. On gateway configurations, outbound connections are the summation of Ni(M-1) where Ni is the number of nodes in a gateway i, and M is the total number of gateways. Inbound connections are the summation of U-Ni where U is the sum of all gateway nodes in all gateways, and N is the number of nodes in a gateway i. It works out that both inbound and outbound connection counts are the same.

The number of connections required to join clusters using clustering vs. gateways is apparent very quickly. For 3 clusters, with N nodes:

A cluster section is not needed for gateways, they work with single server as well. Yet, they start to be useful when participating cluster consist of more than one server and they reduce the number of connections.

Interest Propagation

Messages from clients directly connected to a gateway node will be sent along outgoing gateway connections according to the following two interest propagation mechanisms:

• Interest-only Mode
• Queue Subscriptions

Local interest permitting, the receiving gateway node sends the messages directly to its subscribing clients as well as the server within the cluster.

Interest-only Mode

Gateway A sends messages only for subjects that gateway B has explicitly expressed interest in. As subscriptions come and go on B, B will update its subject interest with A.

Queue Subscriptions

When a queue subscriber creates a new subscription, the gateway propagates the subscription interest to other gateways. The subscription interest is only propagated once per Account and subject. When the last queue subscriber is gone, the cluster interest is removed.

Queue subscriptions work on Interest-only Mode to honor NATS' queue semantics across the Super Cluster. For each queue group, a message is only delivered to a single queue subscriber. If the same queue group exists in multiple clusters, the server will pick one member from the queue group in its cluster, only sending to a different cluster if there is no interest in its cluster. In other words, a server will always try to serve local queue subscribers first and only failover when a local queue subscriber is not found. The server will pick the cluster with the lowest RTT.

Gateway Configuration

The Gateway Configuration document describes all the options available to gateways.

Configuration

The gateway configuration block is similar to a cluster block:

gateway {
    name: "A"
    listen: "localhost:7222"
    authorization {
        user: gwu
        password: gwp
    }

    gateways: [
        {name: "A", url: "nats://gwu:gwp@localhost:7222"},
        {name: "B", url: "nats://gwu:gwp@localhost:7333"},
        {name: "C", url: "nats://gwu:gwp@localhost:7444"},
    ]
}

One difference is that instead of routes you specify gateways. As expected self-gateway connections are ignored, so you can share gateway configurations with minimal fuss.

Starting a server:

nats-server -c A.conf

[85803] 2019/05/07 10:50:55.902474 [INF] Starting nats-server version 2.0.0
[85803] 2019/05/07 10:50:55.903669 [INF] Gateway name is A
[85803] 2019/05/07 10:50:55.903684 [INF] Listening for gateways connections on localhost:7222
[85803] 2019/05/07 10:50:55.903696 [INF] Address for gateway "A" is localhost:7222
[85803] 2019/05/07 10:50:55.903909 [INF] Listening for client connections on 0.0.0.0:4222
[85803] 2019/05/07 10:50:55.903914 [INF] Server id is NBHUDBF3TVJSWCDPG2HSKI4I2SBSPDTNYEXEMOFAZUZYXVA2IYRUGPZU
[85803] 2019/05/07 10:50:55.903917 [INF] Server is ready
[85803] 2019/05/07 10:50:56.830669 [INF] 127.0.0.1:50892 - gid:2 - Processing inbound gateway connection
[85803] 2019/05/07 10:50:56.830673 [INF] 127.0.0.1:50891 - gid:1 - Processing inbound gateway connection
[85803] 2019/05/07 10:50:56.831079 [INF] 127.0.0.1:50892 - gid:2 - Inbound gateway connection from "C" (NBHWDFO3KHANNI6UCEUL27VNWL7NWD2MC4BI4L2C7VVLFBSMZ3CRD7HE) registered
[85803] 2019/05/07 10:50:56.831211 [INF] 127.0.0.1:50891 - gid:1 - Inbound gateway connection from "B" (ND2UJB3GFUHXOQ2UUMZQGOCL4QVR2LRJODPZH7MIPGLWCQRARJBU27C3) registered
[85803] 2019/05/07 10:50:56.906103 [INF] Connecting to explicit gateway "B" (localhost:7333) at 127.0.0.1:7333
[85803] 2019/05/07 10:50:56.906104 [INF] Connecting to explicit gateway "C" (localhost:7444) at 127.0.0.1:7444
[85803] 2019/05/07 10:50:56.906404 [INF] 127.0.0.1:7333 - gid:3 - Creating outbound gateway connection to "B"
[85803] 2019/05/07 10:50:56.906444 [INF] 127.0.0.1:7444 - gid:4 - Creating outbound gateway connection to "C"
[85803] 2019/05/07 10:50:56.906647 [INF] 127.0.0.1:7444 - gid:4 - Outbound gateway connection to "C" (NBHWDFO3KHANNI6UCEUL27VNWL7NWD2MC4BI4L2C7VVLFBSMZ3CRD7HE) registered
[85803] 2019/05/07 10:50:56.906772 [INF] 127.0.0.1:7333 - gid:3 - Outbound gateway connection to "B" (ND2UJB3GFUHXOQ2UUMZQGOCL4QVR2LRJODPZH7MIPGLWCQRARJBU27C3) registered

Once all the gateways are up, these clusters of one will forward messages as expected:

nats sub -s localhost:4333 ">"

On a different session...

nats pub -s localhost:4444 foo bar

The subscriber should print

[#1] Received on [foo] : 'bar'

Gateway Configuration Block

Gateway Entry

The gateways configuration block is a list of gateway entries with the following properties:

By using urls and an array, you can specify a list of endpoints that form part of a cluster as below. A NATS Server will pick one of those addresses randomly and only establish a single outbound gateway connection to one of the members from another cluster:

gateway {
    name: "DC-A"
    listen: "localhost:7222"

    gateways: [
        {name: "DC-A", urls: ["nats://localhost:7222", "nats://localhost:7223", "nats://localhost:7224"]},
        {name: "DC-B", urls: ["nats://localhost:7332", "nats://localhost:7333", "nats://localhost:7334"]},
        {name: "DC-C", urls: ["nats://localhost:7442", "nats://localhost:7333", "nats://localhost:7335"]}
    ]
}

TLS Entry - Certificate rotation

In addition to the normal TLS configuration advice, bear in mind that TLS keys and certificates for multiple clusters, or servers in different locations, rarely rotate at the exact same time and that Certificate Authorities do roll between multiple Intermediate certificates. Make sure that the ca_file contains both old and new root or intermediate CAs.

If using a certificate bundle which accompanied the issuance of a certificate then the CA in that bundle will typically be for just that certificate. Using only that CA as the CA for gateway authentication is ill-advised. You should ensure that you allow for rolling between Certificate Authorities, even if only between multiple CAs from the same organization entity, and use a separate certificate bundle for verification of peers. This way when DC-B rolls before DC-A, it will not be cut off from your supercluster.

To apply changes (even when only changing the content of those files) the server configuration needs to be reloaded. Configuration reloading

Gateway behind Load Balancer

When running in a private network ( such as VPC in the cloud ), you'd probably want to setup a Load Balancer to expose the gateway port. In this case, it's important to set the advertise value of each NATS server to the Load Balancer hostname and port serving them. Otherwise, they will advertise their own private IP and port, and can generate failures and downtime if any reconnection is required.

Leaf Nodes

A Leaf Node extends an existing NATS system of any size, optionally bridging both operator and security domains. A leafnode server will transparently route messages as needed from local clients to one or more remote NATS system(s) and vice versa. The leaf node authenticates and authorizes clients using a local policy. Messages are allowed to flow to the cluster or into the leaf node based on leaf node connection permissions of either.

Leaf nodes are useful in IoT and edge scenarios and when the local server traffic should be low RTT and local unless routed to the super cluster. NATS' queue semantics are honored across leaf connections by serving local queue consumers first.

• Clients to leaf nodes authenticate locally (or just connect if authentication is not required)
• Traffic between the leaf node and the cluster assumes the restrictions of the user configuration used to create the leaf connection.
  • Subjects that the user is allowed to publish are exported to the cluster.
  • Subjects the user is allowed to subscribe to, are imported into the leaf node.

Unlike cluster or gateway nodes, leaf nodes do not need to be reachable themselves and can be used to explicitly configure any acyclic graph topologies.

If a leaf node connects to a cluster, it is recommended to configure it with knowledge of all seed servers and have each seed server accept connections from leaf nodes. Should the remote cluster's configuration change, the discovery protocol will gossip peers capable of accepting leaf connections. A leaf node can have multiple remotes, each connecting to a different cluster. Each URL in a remote needs to point to the same cluster. If one node in a cluster is configured as leaf node, all nodes need to. Likewise, if one server in a cluster accepts leaf node connections, all servers need to.

Leaf Nodes are an important component as a way to bridge traffic between local NATS servers you control and servers that are managed by a third-party. Synadia's NGS allows accounts to use leaf nodes, but gain accessibility to the global network to inexpensively connect geographically distributed servers or small clusters.

LeafNode Configuration Options

LeafNode Configuration Tutorial

The main server is just a standard NATS server. Clients to the main cluster are just using token authentication, but any kind of authentication can be used. The server allows leaf node connections at port 7422 (default port):

leafnodes {
    port: 7422
}
accounts: {
    app: { 
        users: [
            {user: appuser, password: s3cr3t},
            {user: leafuser, password: s3cr3t}
           ]
    }  
}

Start the server:

nats-server -c /tmp/server.conf

Output extract

...
[5774] 2019/12/09 11:11:23.064276 [INF] Listening for leafnode connections on 0.0.0.0:7422
...

We create a replier on the server to listen for requests on 'q', which it will aptly respond with '42':

nats reply -s nats://appuser:s3cr3t@localhost q 42

The leaf node, allows local clients to connect to through port 4111, and doesn't require any kind of authentication. The configuration specifies where the remote cluster is located, and specifies how to connect to it (just a simple token in this case):

listen: "127.0.0.1:4111"
leafnodes {
    remotes = [
        {
          url: "nats://leafuser:s3cr3t@localhost"
        },
    ]
}

In the case where the remote leaf connection is connecting with tls:

listen: "127.0.0.1:4111"
leafnodes {
    remotes = [
        {
          url: "tls://leafuser:s3cr3t@localhost"
        },
    ]
}

Note the leaf node configuration lists a number of remotes. The url specifies the port on the server where leaf node connections are allowed.

Start the leaf node server:

nats-server -c /tmp/leaf.conf

Output extract

....
[3704] 2019/12/09 09:55:31.548308 [INF] Listening for client connections on 127.0.0.1:4111
...
[3704] 2019/12/09 09:55:31.549404 [INF] Connected leafnode to "localhost"

Connect a client to the leaf server and make a request to 'q':

nats req -s nats://127.0.0.1:4111 q ""

Published [q] : ''
Received  [_INBOX.Ua82OJamRdWof5FBoiKaRm.gZhJP6RU] : '42'

Leaf Node Example Using a Remote Global Service

In this example, we connect a leaf node to Synadia's NGS. Leaf nodes are supported on free developer and paid accounts. To use NGS, ensure that you've signed up and have an account loaded on your local system. It takes less than 30 seconds to grab yourself a free account to follow along if you don't have one already!

The nsc tool can operate with many accounts and operators, so it's essential to make sure you're working with the right operator and account. You can set the account using the nsc tool like below. The DELETE_ME account is used as an example, which is registered with NGS as a free account.

❯ nsc env -a DELETE_ME
❯ nsc describe account
+--------------------------------------------------------------------------------------+
|                                   Account Details                                    |
+---------------------------+----------------------------------------------------------+
| Name                      | DELETE_ME                                                |
| Account ID                | ABF3NX7FJLDCUO5QXBH56PV6EU4PR5HFCUJBXAG57AKSDUBTGORDFOLI |
| Issuer ID                 | ODSKBNDIT3LTZWFSRAWOBXSBZ7VZCDQVU6TBJX3TQGYXUWRU46ANJJS4 |
| Issued                    | 2023-03-02 18:18:42 UTC                                  |
| Expires                   |                                                          |
+---------------------------+----------------------------------------------------------+
| Max Connections           | 10                                                       |
| Max Leaf Node Connections | Not Allowed                                              |
| Max Data                  | 1.0 GB (1000000000 bytes)                                |
| Max Exports               | 2                                                        |
| Max Imports               | 7                                                        |
| Max Msg Payload           | 1.0 kB (1000 bytes)                                      |
| Max Subscriptions         | 10                                                       |
| Exports Allows Wildcards  | True                                                     |
| Disallow Bearer Token     | False                                                    |
| Response Permissions      | Not Set                                                  |
+---------------------------+----------------------------------------------------------+
| Jetstream                 | Disabled                                                 |
+---------------------------+----------------------------------------------------------+
| Exports                   | None                                                     |
+---------------------------+----------------------------------------------------------+

The nsc tool is aware of the account, so let's proceed to create a user for our example.

nsc add user leaftestuser

[ OK ] generated and stored user key "UB5QBEU4LU7OR26JEYSG27HH265QVUFGXYVBRD7SVKQJMEFSZTGFU62F"
[ OK ] generated user creds file "~/.nkeys/creds/synadia/leaftest/leaftestuser.creds"
[ OK ] added user "leaftestuser" to account "leaftest"

Let's craft a leaf node connection much like we did earlier:

leafnodes {
    remotes = [
        {
          url: "tls://connect.ngs.global"
          credentials: "/Users/alberto/.nkeys/creds/synadia/leaftest/leaftestuser.creds"
        },
    ]
}

The default port for leaf nodes is 7422, so we don't have to specify it.

Let's start the leaf server:

nats-server -c /tmp/ngs_leaf.conf

...
[4985] 2023/03/03 10:55:51.577569 [INF] Listening for client connections on 0.0.0.0:4222
...
[4985] 2023/03/03 10:55:51.918781 [INF] Connected leafnode to "connect.ngs.global"

Again, let's connect a replier, but this time to Synadia's NGS. NSC connects specifying the credentials file:

nsc reply q 42

And now let's make the request from the local host:

nats-req q ""

Published [q] : ''
Received  [_INBOX.hgG0zVcVcyr4G5KBwOuyJw.uUYkEyKr] : '42'

Leaf Authorization

In some cases you may want to restrict what messages can be exported from the leaf node or imported from the leaf connection. You can specify restrictions by limiting what the leaf connection client can publish and subscribe to. See NATS Authorization for how you can do this.

TLS-first Handshake

As of NATS v2.10.0

Leafnode connections follow the model where when a TCP connection is created to the server, the server will immediately send an INFO protocol message in clear text. This INFO protocol provides metadata, including whether the server requires a secure connection.

Some environments prefer not to want a server that is configured to accept TLS connections for leafnodes having any traffic sent in clear text. It was possible to by-pass this using a websocket connection. However, if websocket is not desired, the accepting and remote servers can be configured to perform a TLS handshake before sending the INFO protocol message.

Configuration

leafnodes Configuration Block

The leaf node configuration block is used to configure incoming as well as outgoing leaf node connections. Most properties are for the configuration of incoming connections. The properties remotes and reconnect are for outgoing connections.

TLS Block

As of NATS v2.10.0

The tls block for leafnodes configuration has an additional field enabling a TLS-first handshake between the remote and the accepting server.

Accepting side

Since Leafnodes can connect to a variety of servers, the ability to indicate if the TLS handshake should be done first is configured in 2 places. The accepting side is in the tls block of the leafnodes block.

leafnodes {
  port: 7422
  tls {
    handshake_first: true
    # other TLS fields...
  }
}

With the above configuration, an older server, or a server that does not have the remote configuration also configured with handshake_first: true, will fail to create a leafnode connection because the accepting-side server will initiate the TLS handshake while the soliciting side will wait for the INFO protocol to be received.

Remote side

To indicate that a leafnode connection should perform the TLS handshake first, it needs to be configured in the remote configuration:

leafnodes {
  remotes [
    {
      urls: ["tls://example:7422"]
      tls: {
         handshake_first: true
         # other TLS fields...
      }
    }
  ]
}

If the remote is configured as such but the server it is connecting to does not have handshake_first: true configured, the connection will fail since the solicit side is performing a TLS handshake but will receive an INFO protocol in clear.

Authorization Block

提示

A leaf node can authenticate against any user account on the hub (the incoming side of the connection), including those defined in the accounts themselves. This authorization block is therefore optional if appropriate account users already exist.

Whether configuring users in the account or in this dedicated authorization block is more convenient will depend on your deployment style.

Users Block

Here are some examples of using basic user/password authentication for leaf nodes (note while this is using accounts it is not using JWTs)

Singleton mode:

leafnodes {
  port: ...
  authorization {
    user: leaf
    password: secret
    account: TheAccount
  }
}

With above configuration, if a soliciting server creates a Leafnode connection with url: nats://leaf:secret@host:port, then the accepting server will bind the leafnode connection to the account "TheAccount". This account needs to exist otherwise the connection will be rejected.

Multi-users mode:

leafnodes {
  port: ...
  authorization {
    users = [
      {user: leaf1, password: secret, account: account1}
      {user: leaf2, password: secret, account: account2}
    ]
  }
}

With the above, if a server connects using leaf1:secret@host:port, then the accepting server will bind the connection to account account1. If using leaf2 user, then the accepting server will bind to connection to account2.

If username/password (either singleton or multi-users) is defined, then the connecting server MUST provide the proper credentials otherwise the connection will be rejected.

If no username/password is provided, it is still possible to provide the account the connection should be associated with:

leafnodes {
  port: ...
  authorization {
    account: TheAccount
  }
}

With the above, a connection without credentials will be bound to the account "TheAccount".

If other form of credentials are used (jwt, nkey or other), then the server will attempt to authenticate and if successful associate to the account for that specific user. If the user authentication fails (wrong password, no such user, etc..) the connection will be also rejected.

LeafNode remotes Entry Block

Signature Handler

As of NATS Server v.2.9.0, for users embedding the NATS Server, it is possible to replace the use of the credentials file by a signature callback which will sign the nonce and provide the JWT in the CONNECT protocol. The RemoteLeafOpts has a new field:

SignatureCB  SignatureHandler

The callback definition is:

// SignatureHandler is used to sign a nonce from the server while
// authenticating with Nkeys. The callback should sign the nonce and
// return the JWT and the raw signature.
type SignatureHandler func([]byte) (string, []byte, error)

And example of how to use it can be found here

Connecting using WebSocket protocol

Since NATS 2.2.0, Leaf nodes support outbound WebSocket connections by specifying ws as the scheme component of the remote server URLs:

leafnodes {
  remotes [
    {urls: ["ws://hostname1:443", "ws://hostname2:443"]}
  ]
}

Note that if a URL has the ws scheme, all URLs the list must be ws. You cannot mix and match. Therefore this would be considered an invalid configuration:

  remotes [
    # Invalid configuration that will prevent the server from starting
    {urls: ["ws://hostname1:443", "nats://hostname2:7422"]}
  ]

Note that the decision to make a TLS connection is not based on wss:// (as opposed to ws://) but instead in the presence of a TLS configuration in the leafnodes{} or the specific remote configuration block.

To configure Websocket in the remote server, check the Websocket section.

tls Configuration Block

JetStream on Leaf Nodes

If you want to see a demonstration of the full range of this functionality, check out our video

One of the use cases for a NATS server configured as a leaf node is to provide a local NATS network even when the connection to a hub or the cloud is down. To support such a disconnected use case with JetStream, independent JetStream islands are also supported and available through the same NATS network.

The general issue with multiple, independent JetStreams, accessible from the same client is that you need to be able to tell them apart. As an example, consider a leaf node with a non-clustered JetStream in each server. You connect to one of them, but which JetStream responds when you use the JetStream API $JS.API.> ?

To disambiguate between servers, the option domain was added to the JetStream configuration block. When using it, follow these rules: Every server in a cluster and super cluster needs to have the same domain name. This means that domain names can only change between two servers if they are connected via a leaf node connection. As a result of this the JetStream API $JS.API.> will also be available under a disambiguated name $JS.<domain>.API.>. Needless to say, domain names need to be unique.

There are reasons to connect system accounts on either end of your leaf node connection. You probably don't want to connect your cloud and edge device system accounts, but you might connect them when the only reason keeping you from using a super cluster are firewall rules.

The benefits are:

• Monitoring of all connected nats-servers
• nats-account-resolver working on the entire network
• extended JetStream cluster

When domain is set, JetStream-related traffic on the system account is suppressed. This is what causes JetStream not to be extended.

In addition, traffic on $JS.API.> is also suppressed. This causes clients to use the local JetStream that is available in the nats-servers they are connected to. To communicate with another JetStream, that JetStream's unique domain specific prefix $JS.<domain>.API needs to be specified.

Please be aware that each domain is an independent name space. Meaning, inside the same account it is legal to use the same stream name in different domains.

Furthermore, regular message flow is not restricted. Thus, if the same subject is subscribed to by different streams in the same account in different domains, as long as the underlying leaf node was connected at the time, each stream will store the message. This can be resolved by using the same account but use different subjects in each domain or use different accounts in each domain or isolate accounts used in leaf nodes.

Known issue: if you have more than one JetStream enabled leaf node in a different cluster, the cluster you connect to also needs JetStream enabled and a domain set.

Known issue: when you intend to extend a central JetStream, by not supplying a domain name in leaf nodes, that central JetStream needs to be in clustered mode.

Configuration

Below is the config needed to connect two JetStream enabled servers via a leaf node connection. In the example, the system accounts are connected for demonstration purposes (you do not have to do that).

accounts.conf Imported by Both Servers

accounts {
    SYS: {
        users: [{user: admin, password: admin}]
    },
    ACC: {
        users: [{user: acc, password: acc}],
        jetstream: enabled
    }
}
system_account: SYS

hub.conf

To be started with nats-server -c hub.conf:

port: 4222
server_name: hub-server
jetstream {
    store_dir="./store_server"
    domain=hub
}
leafnodes {
    port: 7422
}
include ./accounts.conf

leaf.conf

To be started with nats-server -c leaf.conf:

port: 4111
server_name: leaf-server
jetstream {
    store_dir="./store_leaf"
    domain=leaf
}
leafnodes {
    remotes = [
        {
            urls: ["nats://admin:admin@0.0.0.0:7422"]
            account: "SYS"
        },
        {
            urls: ["nats://acc:acc@0.0.0.0:7422"]
            account: "ACC"
        }
    ]
}
include ./accounts.conf

Usage

Because the system account is connected, you can obtain the JetStream server report from both servers.

nats  --server nats://admin:admin@localhost:4222 server report jetstream

╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                                JetStream Summary                                                │
├─────────────┬─────────────┬────────┬─────────┬───────────┬──────────┬───────┬────────┬──────┬─────────┬─────────┤
│ Server      │ Cluster     │ Domain │ Streams │ Consumers │ Messages │ Bytes │ Memory │ File │ API Req │ API Err │
├─────────────┼─────────────┼────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│ leaf-server │ leaf-server │ leaf   │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
│ hub-server  │             │ hub    │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
├─────────────┼─────────────┼────────┼─────────┼───────────┼──────────┼───────┼────────┼──────┼─────────┼─────────┤
│             │             │        │ 0       │ 0         │ 0        │ 0 B   │ 0 B    │ 0 B  │ 0       │ 0       │
╰─────────────┴─────────────┴────────┴─────────┴───────────┴──────────┴───────┴────────┴──────┴─────────┴─────────╯

Create a stream named test subscribing to subject test in the JetStream domain, the program is connected to. As a result, this stream will be created in the domain hub which is the domain of the server listening on localhost:4222.

nats  --server nats://acc:acc@localhost:4222 stream add

? Stream Name test
? Subjects to consume test
? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Per Subject Messages Limit -1
? Message size limit -1
? Maximum message age limit -1
? Maximum individual message size -1
? Duplicate tracking time window 2m
? Replicas 1
Stream test was created

Information for Stream test created 2021-06-28T12:52:29-04:00

Configuration:

             Subjects: test
     Acknowledgements: true
            Retention: File - Limits
             Replicas: 1
       Discard Policy: Old
     Duplicate Window: 2m0s
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: 0.00s
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited


State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 0
              LastSeq: 0
     Active Consumers: 0

To create a stream in a different domain while connected somewhere else, just provide the js-domain argument. While connected to the same server as before, now the stream is created in leaf.

nats  --server nats://acc:acc@localhost:4222 stream add --js-domain leaf

? Stream Name test
? Subjects to consume test
? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Per Subject Messages Limit -1
? Message size limit -1
? Maximum message age limit -1
? Maximum individual message size -1
? Duplicate tracking time window 2m
? Replicas 1
Stream test was created

Information for Stream test created 2021-06-28T12:59:18-04:00

Configuration:

             Subjects: test
     Acknowledgements: true
            Retention: File - Limits
             Replicas: 1
       Discard Policy: Old
     Duplicate Window: 2m0s
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: 0.00s
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited


State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 0
              LastSeq: 0
     Active Consumers: 0

Publish a message so there is something to retrieve.

nats  --server nats://acc:acc@localhost:4222 pub test "hello world"

Because both streams subscribe to the same subject, each one now reports one message. This is done to demonstrate the issue. If you want to avoid that, you need to either use different subjects, different accounts, or one isolated account.

nats  --server nats://acc:acc@localhost:4222 stream report --js-domain leaf

Obtaining Stream stats

╭─────────────────────────────────────────────────────────────────────────────╮
│                                Stream Report                                │
├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────┤
│ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │
├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────┤
│ test   │ File    │ 0         │ 1        │ 45 B  │ 0    │ 0       │          │
╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────╯

nats  --server nats://acc:acc@localhost:4222 stream report --js-domain hub

Output

Obtaining Stream stats

╭─────────────────────────────────────────────────────────────────────────────╮
│                                Stream Report                                │
├────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────┤
│ Stream │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │
├────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────┤
│ test   │ File    │ 0         │ 1        │ 45 B  │ 0    │ 0       │          │
╰────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────╯

Copying across domains via source or mirror

In order to copy a stream from one domain into another, specify the JetStream domain when creating a mirror. If you want to connect a leaf to the hub and get commands, even when the leaf node connection is offline, mirroring a stream located in the hub is the way to go.

nats  --server nats://acc:acc@localhost:4222 stream add --js-domain hub --mirror test

? Stream Name backup-test-leaf
? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Message size limit -1
? Maximum message age limit -1
? Maximum individual message size -1
? Replicas 1
? Adjust mirror start No
? Import mirror from a different JetStream domain Yes
? Foreign JetStream domain name leaf
? Delivery prefix
Stream backup-test-leaf was created

Information for Stream backup-test-leaf created 2021-06-28T14:00:43-04:00

Configuration:

     Acknowledgements: true
            Retention: File - Limits
             Replicas: 1
       Discard Policy: Old
     Duplicate Window: 2m0s
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: 0.00s
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited
               Mirror: test, API Prefix: $JS.leaf.API, Delivery Prefix:


State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 0
              LastSeq: 0
     Active Consumers: 0

Similarly, if you want to aggregate streams located in any number of leaf nodes use source. If the streams located in each leaf are used for the same reasons, it is recommended to aggregate them in the hub for processing via source.

nats  --server nats://acc:acc@localhost:4222 stream add --js-domain hub --source test

? Stream Name aggregate-test-leaf
? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Message size limit -1
? Maximum message age limit -1
? Maximum individual message size -1
? Duplicate tracking time window 2m
? Replicas 1
? Adjust source "test" start No
? Import "test" from a different JetStream domain Yes
? test Source foreign JetStream domain name leaf
? test Source foreign JetStream domain delivery prefix
Stream aggregate-test-leaf was created

Information for Stream aggregate-test-leaf created 2021-06-28T14:02:36-04:00

Configuration:

     Acknowledgements: true
            Retention: File - Limits
             Replicas: 1
       Discard Policy: Old
     Duplicate Window: 2m0s
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: 0.00s
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited
              Sources: test, API Prefix: $JS.leaf.API, Delivery Prefix:


State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 0
              LastSeq: 0
     Active Consumers: 0

source as well as mirror take a copy of the messages. Once copied, accessing the data is independent of the leaf node connection being online. Copying this way also avoids having to run a dedicated program of your own. This is the recommended way to exchange persistent data across domains.

nats  --server nats://acc:acc@localhost:4222 stream report --js-domain hub

Obtaining Stream stats

╭──────────────────────────────────────────────────────────────────────────────────────────╮
│                                      Stream Report                                       │
├─────────────────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────┤
│ Stream              │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │
├─────────────────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────┤
│ backup-test-leaf    │ File    │ 0         │ 1        │ 45 B  │ 0    │ 0       │          │
│ test                │ File    │ 0         │ 1        │ 45 B  │ 0    │ 0       │          │
│ aggregate-test-leaf │ File    │ 0         │ 1        │ 98 B  │ 0    │ 0       │          │
╰─────────────────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────╯

╭────────────────────────────────────────────────────────────────────────────────────╮
│                                 Replication Report                                 │
├─────────────────────┬────────┬──────────────┬───────────────┬────────┬─────┬───────┤
│ Stream              │ Kind   │ API Prefix   │ Source Stream │ Active │ Lag │ Error │
├─────────────────────┼────────┼──────────────┼───────────────┼────────┼─────┼───────┤
│ backup-test-leaf    │ Mirror │ $JS.leaf.API │ test          │ 0.21s  │ 0   │       │
│ aggregate-test-leaf │ Source │ $JS.leaf.API │ test          │ 1.23s  │ 0   │       │
╰─────────────────────┴────────┴──────────────┴───────────────┴────────┴─────┴───────╯

Cross account & domain import

All of the above happened in the same account. To share domain access across accounts the account.conf from above needs to be modified and the server restarted or reloaded. This example exports the consumer and FC API as well as a delivery subject which is used by the internal push consumer created by source and mirror.

In support of another example on how to share a durable pull consumer for client access across domains and accounts, the NEXT and ACK API are exported as well.

Known issue: Currently, across accounts, push consumer are not supported.

On import, the JetStream API prefix $JS.hub.API is renamed to JS.test@hub.API. This is to, once more, disambiguate which JetStream a client in the importing account might want to interact with. When using domains, the general recommendation is to export the domain specific API $JS.<domain>.API as this allows you to pin the export to a particular domain.

Furthermore, the delivery subject is extended on import. This is to allow for easier export into multiple accounts.

This example also exports the absolute minimum necessary. It is possible to give access to the entire consumer API $JS.hub.API.CONSUMER.> or the entire API in a domain $JS.hub.API.> or the entire API $JS.API.> wherever the importing client connects.

accounts {
    SYS: {
        users: [{user: admin, password: admin}]
    },
    ACC: {
        users: [{user: acc, password: acc}],
        jetstream: enabled
        exports: [
            # minimum export needed to allow source/mirror to create a consumer on the fly
            {service: "$JS.hub.API.CONSUMER.CREATE.*", response_type: "stream"}
            # minimum export needed for push consumer. This includes source and mirror!
            {stream: "deliver.acc.hub.>"}
            # minimum export needed for durable pull consumer `dur` in stream `aggregate-test-leaf`. (clients only - source and mirror do not use this)
            {service: "$JS.hub.API.CONSUMER.MSG.NEXT.aggregate-test-leaf.dur", response_type: "stream"}
            # minimum export needed to ack messages for durable consumer `dur` in stream `aggregate-test-leaf`. (clients only - source and mirror do not use this)
            {service: "$JS.ACK.aggregate-test-leaf.dur.>"}
            # minimum export needed for flow control of source/mirror
            {service: "$JS.FC.aggregate-test-leaf.dur.>"}
        ]
    }
    IMPORT_MIRROR: {
        users: [{user: import_mirror, password: import_mirror}],
        jetstream: enabled
        imports: [
            {service: {account: ACC, subject: "$JS.hub.API.CONSUMER.CREATE.*"}, to: "JS.acc@hub.API.CONSUMER.CREATE.*" }
            {service: {account: ACC, subject: "$JS.FC.aggregate-test-leaf.dur.>"}}
            {stream: {account: ACC, subject: deliver.acc.hub.import_mirror.>}}
        ]
    }
    # As of now, cross account, only pull consumer are supported.
    IMPORT_CLIENT: {
        users: [{user: import_client, password: import_client}],
        jetstream: enabled
        imports: [
            {service: {account: ACC, subject: "$JS.hub.API.CONSUMER.MSG.NEXT.aggregate-test-leaf.dur"}, to: "JS.acc@hub.API.CONSUMER.MSG.NEXT.aggregate-test-leaf.dur" }
            {service: {account: ACC, subject: "$JS.ACK.aggregate-test-leaf.dur.>"}}
        ]
    }
}
system_account: SYS

Copying via source and mirror

Once the servers have been restarted or reloaded, a mirror can be created as follows (same applies to source): On import from a different account the renamed prefix JS.acc@hub.API is provided. In addition, the delivery subject name is extended to also include the importing domain and stream. This makes it unique to that particular import. If every delivery prefix follows the pattern <static type>.<exporting account>.<exporting domain>.<importing account>.<importing domain>.<importing stream name> overlaps caused by multiple imports are avoided.

nats  --server nats://import_mirror:import_mirror@localhost:4222 stream add --js-domain hub --mirror aggregate-test-leaf

? Stream Name aggregate-test-leaf-from-acc
? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Message size limit -1
? Maximum message age limit -1
? Maximum individual message size -1
? Replicas 1
? Adjust mirror start No
? Import mirror from a different JetStream domain No
? Import mirror from a different account Yes
? Foreign account API prefix JS.acc@hub.API
? Foreign account delivery prefix deliver.acc.hub.import_mirror.hub.aggregate-test-leaf-from-acc
Stream aggregate-test-leaf-from-acc was created

Information for Stream aggregate-test-leaf-from-acc created 2021-06-28T16:59:15-04:00

Configuration:

     Acknowledgements: true
            Retention: File - Limits
             Replicas: 1
       Discard Policy: Old
     Duplicate Window: 2m0s
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: 0.00s
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited
               Mirror: aggregate-test-leaf, API Prefix: JS.acc@hub.API, Delivery Prefix: deliver.acc.hub.import_mirror.hub.aggregate-test-leaf-from-acc


State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 0
              LastSeq: 0
     Active Consumers: 0

A subsequent check shows that the one message stored in the stream aggregate in account ACC got copied to the new stream in the account IMPORTER.

nats  --server nats://import_mirror:import_mirror@localhost:4222 stream report --js-domain hub

Obtaining Stream stats

╭───────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                           Stream Report                                           │
├──────────────────────────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────┤
│ Stream                       │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │
├──────────────────────────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────┤
│ aggregate-test-leaf-from-acc │ File    │ 0         │ 1        │ 98 B  │ 0    │ 0       │          │
╰──────────────────────────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────╯

╭─────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                                         Replication Report                                          │
├──────────────────────────────┬────────┬────────────────┬─────────────────────┬────────┬─────┬───────┤
│ Stream                       │ Kind   │ API Prefix     │ Source Stream       │ Active │ Lag │ Error │
├──────────────────────────────┼────────┼────────────────┼─────────────────────┼────────┼─────┼───────┤
│ aggregate-test-leaf-from-acc │ Mirror │ JS.acc@hub.API │ aggregate-test-leaf │ 0.59s  │ 0   │       │
╰──────────────────────────────┴────────┴────────────────┴─────────────────────┴────────┴─────┴───────╯

Direct access of a durable pull consumer

The modified accounts.conf also includes a separate import for an existing pull consumer. Let's create a consumer by the name dur in the stream aggregate-test-leaf in the account acc.

nats  --server nats://acc:acc@localhost:4222 consumer add  --js-domain hub

? Consumer name dur
? Delivery target (empty for Pull Consumers)
? Start policy (all, new, last, 1h, msg sequence) all
? Replay policy instant
? Filter Stream by subject (blank for all)
? Maximum Allowed Deliveries -1
? Maximum Acknowledgements Pending 0
? Select a Stream aggregate-test-leaf
Information for Consumer aggregate-test-leaf > dur created 2021-06-28T17:16:51-04:00

Configuration:

        Durable Name: dur
           Pull Mode: true
         Deliver All: true
          Ack Policy: Explicit
            Ack Wait: 30s
       Replay Policy: Instant
     Max Ack Pending: 20,000
   Max Waiting Pulls: 512

State:

   Last Delivered Message: Consumer sequence: 0 Stream sequence: 0
     Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0
         Outstanding Acks: 0 out of maximum 20000
     Redelivered Messages: 0
     Unprocessed Messages: 1
            Waiting Pulls: 0 of maximum 512

nats  --server nats://acc:acc@localhost:4222 stream report --js-domain hub

Obtaining Stream stats

╭──────────────────────────────────────────────────────────────────────────────────────────╮
│                                      Stream Report                                       │
├─────────────────────┬─────────┬───────────┬──────────┬───────┬──────┬─────────┬──────────┤
│ Stream              │ Storage │ Consumers │ Messages │ Bytes │ Lost │ Deleted │ Replicas │
├─────────────────────┼─────────┼───────────┼──────────┼───────┼──────┼─────────┼──────────┤
│ backup-test-leaf    │ File    │ 0         │ 1        │ 45 B  │ 0    │ 0       │          │
│ test                │ File    │ 0         │ 1        │ 45 B  │ 0    │ 0       │          │
│ aggregate-test-leaf │ File    │ 1         │ 1        │ 98 B  │ 0    │ 0       │          │
╰─────────────────────┴─────────┴───────────┴──────────┴───────┴──────┴─────────┴──────────╯

╭────────────────────────────────────────────────────────────────────────────────────╮
│                                 Replication Report                                 │
├─────────────────────┬────────┬──────────────┬───────────────┬────────┬─────┬───────┤
│ Stream              │ Kind   │ API Prefix   │ Source Stream │ Active │ Lag │ Error │
├─────────────────────┼────────┼──────────────┼───────────────┼────────┼─────┼───────┤
│ backup-test-leaf    │ Mirror │ $JS.leaf.API │ test          │ 1.85s  │ 0   │       │
│ aggregate-test-leaf │ Source │ $JS.leaf.API │ test          │ 1.85s  │ 0   │       │
╰─────────────────────┴────────┴──────────────┴───────────────┴────────┴─────┴───────╯

nats  --server nats://acc:acc@localhost:4222 consumer report --js-domain hub

Output

? Select a Stream aggregate-test-leaf
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                          Consumer report for aggregate-test-leaf with 1 consumers                           │
├──────────┬──────┬────────────┬──────────┬─────────────┬─────────────┬─────────────┬───────────┬─────────────┤
│ Consumer │ Mode │ Ack Policy │ Ack Wait │ Ack Pending │ Redelivered │ Unprocessed │ Ack Floor │ Cluster     │
├──────────┼──────┼────────────┼──────────┼─────────────┼─────────────┼─────────────┼───────────┼─────────────┤
│ dur      │ Pull │ Explicit   │ 30.00s   │ 0           │ 0           │ 1 / 100%    │ 0         │ hub-server* │
╰──────────┴──────┴────────────┴──────────┴─────────────┴─────────────┴─────────────┴───────────┴─────────────╯

To retrieve the messages stored in the domain hub using nats while connected to the leaf node, provide the correct stream and durable name as well as the API prefix JS.acc@hub.API

nats --server nats://import_client:import_client@localhost:4111 consumer next aggregate-test-leaf dur --js-api-prefix JS.acc@hub.API

[17:44:16] subj: test / tries: 1 / cons seq: 1 / str seq: 1 / pending: 0

Headers:

  Nats-Stream-Source: test:mSx7q4yJ 1

Data:


hello world

Acknowledged message

nats  --server nats://acc:acc@localhost:4222 consumer report --js-domain hub

? Select a Stream aggregate-test-leaf
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│                          Consumer report for aggregate-test-leaf with 1 consumers                           │
├──────────┬──────┬────────────┬──────────┬─────────────┬─────────────┬─────────────┬───────────┬─────────────┤
│ Consumer │ Mode │ Ack Policy │ Ack Wait │ Ack Pending │ Redelivered │ Unprocessed │ Ack Floor │ Cluster     │
├──────────┼──────┼────────────┼──────────┼─────────────┼─────────────┼─────────────┼───────────┼─────────────┤
│ dur      │ Pull │ Explicit   │ 30.00s   │ 0           │ 0           │ 0           │ 1         │ hub-server* │
╰──────────┴──────┴────────────┴──────────┴─────────────┴─────────────┴─────────────┴───────────┴─────────────╯

This works similarly when writing your own client. To avoid waiting for the ack timeout, a new message is sent on test from where it is copied into aggregate-test-leaf.

nats  --server nats://acc:acc@localhost:4222 pub test "hello world 2"

The client is connected to the leaf node and receives the message just sent.

./main nats://import_client:import_client@localhost:4111

starting
&{Sequence:{Consumer:3 Stream:3} NumDelivered:1 NumPending:0 Timestamp:2021-06-28 17:51:05.186878 -0400 EDT Stream:aggregate-test-leaf Consumer:dur}
hello world 2
nats: timeout
^Cnats: timeout

There the API prefix is communicated with setting the option nats.APIPrefix("JS.acc@hub.API") when obtaining the JetStream object. Because the API access is limited, the subscribe call provides the option nats.Bind("aggregate-test-leaf", "dur") which prevents calls to infer the stream and durable name.

package main

import (
    "fmt"
    "os"
    "os/signal"
    "syscall"
    "time"

    "github.com/nats-io/nats.go"
)

func main() {
    nc, err := nats.Connect(os.Args[1], nats.Name("JS test"))
    defer nc.Close()
    if err != nil {
        fmt.Printf("nats connect: %v\n", err)
        return
    }
    js, err := nc.JetStream(nats.APIPrefix("JS.acc@hub.API"))
    if err != nil {
        fmt.Printf("JetStream: %v\n", err)
        if js == nil {
            return
        }
    }
    s, err := js.PullSubscribe("", "dur", nats.Bind("aggregate-test-leaf", "dur"))
    if err != nil {
        fmt.Printf("PullSubscribe: %v\n", err)
        return
    }

    shutdown := make(chan os.Signal, 1)
    signal.Notify(shutdown, os.Interrupt, syscall.SIGTERM)

    fmt.Printf("starting\n")
    for {
        select {
        case <-shutdown:
            return
        default:
            if m, err := s.Fetch(1, nats.MaxWait(time.Second)); err != nil {
                fmt.Println(err)
            } else {

                if meta, err := m[0].Metadata(); err == nil {
                    fmt.Printf("%+v\n", meta)
                }
                fmt.Println(string(m[0].Data))

                if err := m[0].Ack(); err != nil {
                    fmt.Printf("ack error: %+v\n", err)
                }
            }
        }
    }
}

A push subscriber will need a similar setup. It will require the ACK subject. However, instead of exporting/importing the NEXT subject, the delivery subject shown for source/mirror needs to be used.

See Also

JetStream Leaf Nodes demo and its companion video recording.

Securing NATS

The NATS server provides several forms of security:

• Connections can be encrypted with TLS
• Client connections can require authentication
• Clients can require authorization for subjects they publish or subscribe to

Enabling TLS

The NATS server uses modern TLS semantics to encrypt client, route, and monitoring connections. Check here for pitfalls.

Server configuration revolves around a tls map, which has the following properties:

The simplest configuration:

tls: {
  cert_file: "./server-cert.pem"
  key_file: "./server-key.pem"
}

Or by using server options:

nats-server --tls --tlscert=./server-cert.pem --tlskey=./server-key.pem

[21417] 2019/05/16 11:21:19.801539 [INF] Starting nats-server version 2.0.0
[21417] 2019/05/16 11:21:19.801621 [INF] Git commit [not set]
[21417] 2019/05/16 11:21:19.801777 [INF] Listening for client connections on 0.0.0.0:4222
[21417] 2019/05/16 11:21:19.801782 [INF] TLS required for client connections
[21417] 2019/05/16 11:21:19.801785 [INF] Server id is ND6ZZDQQDGKYQGDD6QN2Y26YEGLTH6BMMOJZ2XJB2VASPVII3XD6RFOQ
[21417] 2019/05/16 11:21:19.801787 [INF] Server is ready

Notice that the log indicates that the client connections will be required to use TLS. If you run the server in Debug mode with -D or -DV, the logs will show the cipher suite selection for each connected client:

[22242] 2019/05/16 11:22:20.216322 [DBG] 127.0.0.1:51383 - cid:1 - Client connection created
[22242] 2019/05/16 11:22:20.216539 [DBG] 127.0.0.1:51383 - cid:1 - Starting TLS client connection handshake
[22242] 2019/05/16 11:22:20.367275 [DBG] 127.0.0.1:51383 - cid:1 - TLS handshake complete
[22242] 2019/05/16 11:22:20.367291 [DBG] 127.0.0.1:51383 - cid:1 - TLS version 1.2, cipher suite TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256

When a tls section is specified at the root of the configuration, it also affects the monitoring port if https_port option is specified. Other sections such as cluster can specify a tls block.

Certificate rotation

Bear in mind that TLS keys and certificates for multiple clusters, or servers in different locations, rarely rotate at the exact same time and that Certificate Authorities do roll between multiple Intermediate certificates. Make sure that the ca_file contains both old and new root or intermediate CAs.

To apply changes to certificate files (even when only changing the content of thos files) the server configuration needs to be reloaded. Configuration reloading

TLS-first Handshake

As of NATS v2.10.4

Client connections follow the model where, when a TCP connection is created to the server, the server will immediately send an INFO protocol message in clear text. This INFO protocol provides metadata, including whether the server requires a secure connection.

Some environments prefer having clients' TLS connections be initiated right away, that is, not having any traffic sent in clear text. It was possible to by-pass this using a websocket connection. However, if a websocket connection is not desired, the server can be configured to perform a TLS handshake before sending the INFO protocol message.

Only clients that implement an equivalent option would be able to connect if the server runs with this option enabled.

The configuration would look something like this:

tls: {
  cert_file: "./server-cert.pem"
  key_file: "./server-key.pem"
  handshake_first: true
}

However, the parameter can be set to auto or a Golang time duration (e.g. 250ms) to fallback to the original behavior. This is intended for deployments where it is known that not all clients have been upgraded to a client library providing the TLS-first handshake option.

After the delay has elapsed without receiving the TLS handshake from the client, the server reverts to sending the INFO protocol so that older clients can connect. Clients that do connect with the "TLS first" option will be marked as such in the monitoring's Connz page/result. It will allow the administrator to keep track of applications still needing to upgrade.

The configuration would be similar to:

tls: {
  cert_file: "./server-cert.pem"
  key_file: "./server-key.pem"
  handshake_first: auto
}

With the above value, the fallback delay used by the server is 50 milliseconds.

The duration can be explicitly set, say 300 milliseconds:

tls {
    cert_file: ...
    key_file: ...

    handshake_first: "300ms"
}

It is understood that any configuration other than "true" will result in the server sending the INFO protocol after the elapsed amount of time without the client initiating the TLS handshake. Therefore, for administrators who do not want any data transmitted in plain text, the value must be set to "true" only. It will require applications to be updated to a library that provides the option, which may or may not be readily available.

TLS Timeout

The timeout setting enables you to control the amount of time that a client is allowed to upgrade its connection to tls. If your clients are experiencing disconnects during TLS handshake, you'll want to increase the value, however, if you do be aware that an extended timeout exposes your server to attacks where a client doesn't upgrade to TLS and thus consumes resources. Conversely, if you reduce the TLS timeout too much, you are likely to experience handshake errors.

tls: {
  cert_file: "./server-cert.pem"
  key_file: "./server-key.pem"
  # clients will fail to connect (value is too low)
  timeout: 0.0001
}

TLS Connection Rate Limit

As of NATS v2.7.0

TLS handshakes are expensive, and a burst of reconnecting clients can keep the server busy negotiating new sessions instead of serving traffic. The connection_rate_limit setting caps the number of new TLS client connections the server will accept in any one second window. Connections above the cap are closed and the client receives the error Connection throttling is active. Please try again later.. The count of rejected connections is logged once per second so it is visible to operators.

tls: {
  cert_file: "./server-cert.pem"
  key_file: "./server-key.pem"
  connection_rate_limit: 1000
}

The limit is a single counter for the whole server, not per-client or per-account. It only applies to the client listener, so cluster, gateway, leaf node, websocket and MQTT listeners are not affected by it. A value of 0, the default, disables the limit.

Certificate Authorities

The ca_file file should contain one or more Certificate Authorities in PEM format, in a bundle. This is a common format.

When a certificate is issued, it is often accompanied by a copy of the intermediate certificate used to issue it. This is useful for validating that certificate. It is not necessarily a good choice as the only CA suitable for use in verifying other certificates a server may see.

Do consider though that organizations issuing certificates will change the intermediate they use. For instance, a CA might issue intermediates in pairs, with an active and a standby, and reserve the right to switch to the standby without notice. You probably would want to trust both of those for the ca_file directive, to be prepared for such a day, and then after the first CA has been compromised you can remove it. This way the roll from one CA to another will not break your NATS server deployment.

Self Signed Certificates for Testing

Explaining Public key infrastructure, Certificate Authorities (CA) and x509 certificates fall well outside the scope of this document. So does an explanation on how to obtain a properly trusted certificates.

If anybody outside your organization needs to connect, get certs from a public certificate authority. Think carefully about revocation and cycling times, as well as automation, when picking a CA. If arbitrary applications inside your organization need to connect, use a cert from your in-house CA. If only resources inside a specific environment need to connect, that environment might have its own dedicated automatic CA, eg in Kubernetes clusters, so use that.

Only for testing purposes does it make sense to generate self-signed certificates, even your own CA. This is a short guide on how to do just that and what to watch out for.

DO NOT USE these certificates in production!!!

Problems With Self Signed Certificates

The issues and pitfalls listed here are not limited to self-signed certificates. You will most likely encounter them first in DEV environments when using those.

Client advertise not matching TLS names

NATS cluster advertises host:port of all nodes in a cluster to the connecting client. When connecting to a server via TLS the server name (or IP) is validated against the certificate presented by the server.

When using TLS, it is important to control the hostname that clients will use when discovering the server. By default, the cluster will advertise an IP address, which may result in a failed TLS hostname verification with an IP SANs error.

Set avertise or cluster_advertise in the cluster section to advertise verifiable server names. See cluster_config.md

Missing in Relevant Trust Stores

As they should, these are not trusted by the system your server or clients are running on.

One option is to specify the CA in every client you are using. In case you make use of verify, verify_and_map or verify_cert_and_check_known_urls you need to specify ca_file in the server. If you are having a more complex setup involving cluster, gateways or leaf nodes, ca_file needs to be present in tls maps used to connect to the server with self-signed certificates. While this works for server and libraries from the NATS ecosystem, you will experience issues when connecting with other tools such as your Browser.

Another option is to configure your system's trust store to include self-signed certificate(s). Which trust store needs to be configured depends on what you are testing.

• This may be your OS for server and certain clients.
• The runtime environment for other clients like Java, Python or Node.js.
• Your browser for monitoring endpoints and websockets.

Please check your system's documentation on how to trust a particular self-signed certificate.

Missing Subject Alternative Name

Another common problem is failed identity validation. The IP or DNS name to connect to needs to match a Subject Alternative Name (SAN) inside the certificate. Meaning, if a client/browser/server connect via tls to 127.0.0.1, the server needs to present a certificate with a SAN containing the IP 127.0.0.1 or the connection will be closed with a handshake error.

When verify_cert_and_check_known_urls is specified, Subject Alternative Name (SAN) DNS records are necessary. In order to successfully connect there must be an overlap between the DNS records provided as part of the certificate and the urls configured. If you dynamically grow your cluster and use a new certificate, this route or gateway the server connects to will have to be reconfigured to include an url for the new server. Only then can the new server connect. If the DNS record is a wildcard, matching according to rfc6125 will be performed. Using certificates with a wildcard Subject Alternative Name (SAN) and configuration with url(s) that would match are a way to keep the flexibility of dynamic cluster growth without configuration changes in other clusters.

Wrong Key Usage

When generating your certificate you need to make sure to include the right purpose for which you want to use the certificate. This is encoded in key usage and extended key usage. The necessary values for key usage depend on the ciphers used. Digital Signature and Key Encipherment are an interoperable choice.

With respect to NATS the relevant values for extended key usage are:

• TLS WWW server authentication - To authenticate as server for incoming connections. A NATS server will need a certificate containing this.
• TLS WWW client authentication - To authenticate as client for outgoing connections. Only needed when connecting to a server where verify, verify_and_map or verify_cert_and_check_known_urls are specified. In these cases, a NATS client will need a certificate with this value.
  • Leaf node connections can be configured with verify as well. Then the connecting NATS server will have to present a certificate with this value too. Certificates containing both values are an option.
  • Cluster connections always have verify enabled. Which server acts as client and server comes down to timing and therefore can't be individually configured. Certificates containing both values are a must.
  • Gateway connections always have verify enabled. Unlike cluster outgoing connections can specify a separate cert. Certificates containing both values are an option that reduce configuration.

Note that it's common practice for non-web protocols to use the TLS WWW authentication fields, as a matter of history those have become embedded as generic options.

Creating Self Signed Certificates for Testing

The simplest way to generate a CA as well as client and server certificates is mkcert. This zero config tool generates and installs the CA into your local system trust store(s) and makes providing SAN straight forward. Check its documentation for installation and your system's trust store. Here is a simple example:

Generate a CA as well as a certificate, valid for server authentication by localhost and the IP ::1(-cert-file and -key-file overwrite default file names). Then start a NATS server using the generated certificate.

mkcert -install
mkcert -cert-file server-cert.pem -key-file server-key.pem localhost ::1
nats-server --tls --tlscert=server-cert.pem --tlskey=server-key.pem -ms 8222

Now you should be able to access the monitoring endpoint https://localhost:8222 with your browser. https://127.0.0.1:8222 however should result in an error as 127.0.0.1 is not listed as SAN. You will not be able to establish a connection from another computer either. For that to work you have to provide appropriate DNS and/or IP SAN(s)

To generate certificates that work with verify and cluster/gateway/leaf_nodes provide the -client option. It will cause the appropriate key usage for client authentication to be added. This example also adds a SAN email for usage as user name in verify_and_map.

mkcert -client -cert-file client-cert.pem -key-file client-key.pem localhost ::1 email@localhost

Please note:

• That client refers to connecting process, not necessarily a NATS client.
• mkcert -client will generate a certificate with key usage suitable for client and server authentication.

Examples in this document make use of the certificates generated so far. To simplify examples using the CA certificate, copy rootCA.pem into the same folder where the certificates were generated. To obtain the CA certificate's location use this command:

mkcert -CAROOT

Once you are done testing, remove the CA from your local system trust store(s).

mkcert -uninstall

Alternatively, you can also use openssl to generate certificates. This tool allows a lot more customization of the generated certificates. It is more complex and does not manage installation into the system trust store(s).

However, for inspecting certificates it is quite handy. To inspect the certificates from the above example execute these commands:

openssl x509 -noout -text -in server-cert.pem
openssl x509 -noout -text -in client-cert.pem

TLS-Terminating Reverse Proxies

Using a TLS-terminating reverse proxy with NATS requires some specific configuration on the server. In a typical proxy scenario, the client to proxy communication is secured and the proxy to server is insecure. This causes a "mismatch" because the server appears to be insecure but the client is told to connect securely. To fix this, the server must be configured as "tls available". This is done via an empty tls block and the allow_non_tls flag.

tls {}
allow_non_tls: true

Once this is configured, your client can connect to the proxy with normal (language specific) tls configuration. Please make sure you are using the appropriate version of your language specific client.

nats.js

See: https://github.com/nats-io/nats.js/issues/369

nats.rs

See: https://github.com/nats-io/nats.rs/blob/main/async-nats/src/connector.rs

nats.net (v1)

See: https://github.com/nats-io/nats.net.v1/tree/main/src/Samples/TlsVariationsExample

Authentication

NATS authentication is multi-level. All of the security modes have an accounts level with users belonging to those accounts. The decentralized JWT Authentication also has an operator to which the accounts belong.

Each account has its own independent subject namespace: a message published on subject 'foo' in one account will not be seen by subscribers to 'foo' in other accounts. Accounts can however define exports and imports of subject(s) streams as well as expose request-reply services between accounts. Users within an account will share the same subject namespace but can be restricted to only be able to publish-subscribe to specific subjects.

Authentication Methods

The NATS server provides various ways of authenticating clients:

• Token Authentication
• Plain Text Username/Password credentials
• Bcrypted Username/Password credentials
• TLS Certificate
• NKEY with Challenge
• Decentralized JWT Authentication/Authorization

Authentication deals with allowing a NATS client to connect to the server. Except for JWT authentication, authentication and authorization are configured in the authorization section of the configuration. With JWT authentication the account and user information are stored in the resolver rather than in the server configuration file.

Authorization Map

The authorization block provides authentication configuration as well as authorization:

User Configuration Map

A user configuration map specifies credentials and permissions options for a single user:

Tokens

Token authentication is a string that if provided by a client, allows it to connect. It is the most straightforward authentication provided by the NATS server.

To use token authentication, you can specify an authorization section with the token property set:

authorization {
    token: "s3cr3t"
}

Token authentication can be used in the authorization section for clients and clusters.

Or start the server with the --auth flag:

nats-server --auth s3cr3t

A client can easily connect by specifying the server URL:

nats sub -s nats://s3cr3t@localhost:4222 ">"

Bcrypted Tokens

Tokens can be bcrypted enabling an additional layer of security, as the clear-text version of the token would not be persisted on the server configuration file.

You can generate bcrypted tokens and passwords using the nats tool:

nats server passwd

? Enter password [? for help] **********************
? Reenter password [? for help] **********************

$2a$11$PWIFAL8RsWyGI3jVZtO9Nu8.6jOxzxfZo7c/W0eLk017hjgUKWrhy

Here's a simple configuration file:

authorization {
    token: "$2a$11$PWIFAL8RsWyGI3jVZtO9Nu8.6jOxzxfZo7c/W0eLk017hjgUKWrhy"
}

The client will still require the clear-text token to connect:

nats sub -s nats://dag0HTXl4RGg7dXdaJwbC8@localhost:4222 ">"

Username/Password

Plain Text Passwords

You can authenticate one or more clients using username and passwords; this enables you to have greater control over the management and issuance of credential secrets.

Single User

authorization: {
    user: a,
    password: b
}

You can also specify a single username/password by:

> nats-server --user a --pass b

Multiple users

authorization: {
    users: [
        {user: a, password: b},
        {user: b, password: a}
    ]
}

Bcrypted Passwords

Username/password also supports bcrypted passwords using the nats tool. Simply replace the clear text password with the bcrypted entries:

> nats server passwd
? Enter password [? for help] **********************
? Reenter password [? for help] **********************

$2a$11$V1qrpBt8/SLfEBr4NJq4T.2mg8chx8.MTblUiTBOLV3MKDeAy.f7u

And on the configuration file:

authorization: {
    users: [
        {user: a, password: "$2a$11$V1qrpBt8/SLfEBr4NJq4T.2mg8chx8.MTblUiTBOLV3MKDeAy.f7u"},
        ...
    ]
}

Reloading a Configuration

As you add/remove passwords from the server configuration file, you'll want your changes to take effect. To reload without restarting the server and disconnecting clients, do:

> nats-server --signal reload

TLS Authentication

The server can require TLS certificates from a client. When needed, you can use the certificates to:

• Validate the client certificate matches a known or trusted CA
• Extract information from a trusted certificate to provide authentication

Note: To simplify the common scenario of maintainers looking at the monitoring endpoint, verify and verify_and_map do not apply to the monitoring port.

The examples in the following sections make use of the certificates you generated locally.

Validating a Client Certificate

The server can verify a client certificate using a CA certificate. To require verification, add the option verify to the TLS configuration section as follows:

tls {
  cert_file: "server-cert.pem"
  key_file:  "server-key.pem"
  ca_file:   "rootCA.pem"
  verify:    true
}

Or via the command line:

nats-server --tlsverify --tlscert=server-cert.pem --tlskey=server-key.pem --tlscacert=rootCA.pem

This option verifies the client's certificate is signed by the CA specified in the ca_file option. When ca_file is not present it will default to CAs in the system trust store. It also makes sure that the client provides a certificate with the extended key usage TLS Web Client Authentication.

Mapping Client Certificates To A User

In addition to verifying that a specified CA issued a client certificate, you can use information encoded in the certificate to authenticate a client. The client wouldn't have to provide or track usernames or passwords.

To have TLS Mutual Authentication map certificate attributes to the user's identity use verify_and_map as shown as follows:

tls {
  cert_file: "server-cert.pem"
  key_file:  "server-key.pem"
  ca_file:   "rootCA.pem"
  # Require a client certificate and map user id from certificate
  verify_and_map: true
}

Note that verify was changed to verify_and_map.

When present, the server will check if a Subject Alternative Name (SAN) maps to a user. It will search all email addresses first, then all DNS names. If no user could be found, it will try the certificate subject.

Note: This mechanism will pick the user it finds first. There is no configuration to restrict this.

openssl x509 -noout -text -in  client-cert.pem

Certificate:
...
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                DNS:localhost, IP Address:0:0:0:0:0:0:0:1, email:email@localhost
            X509v3 Extended Key Usage:
                TLS Web Client Authentication
...

The configuration to authorize this user would be as follow:

authorization {
  users = [
    {user: "email@localhost"}
  ]
}

Use the RFC 2253 Distinguished Names syntax to specify a user corresponding to the certificate subject:

openssl x509 -noout -text -in client-cert.pem

Certificate:
    Data:
...
        Subject: O=mkcert development certificate, OU=testuser@MacBook-Pro.local (Test User)
...

Note that for this example to work you will have to modify the user to match what is in your certificates subject. In doing so, watch out for the order of attributes!

The configuration to authorize this user would be as follows:

authorization {
  users = [
    {user: "OU=testuser@MacBook-Pro.local (Test User),O=mkcert development certificate"}
  ]
}

TLS Timeout

TLS timeout is described here.

TLS Authentication in clusters

When setting up clusters, all servers in the cluster, if using TLS, will both verify the connecting endpoints and the server responses. So certificates are checked in both directions. Certificates can be configured only for the server's cluster identity, keeping client and server certificates separate from cluster formation.

TLS Mutual Authentication is the only way of securing routes.

cluster {
  listen: 127.0.0.1:4244

  tls {
    # Route cert
    cert_file: "./configs/certs/srva-cert.pem"
    # Private key
    key_file:  "./configs/certs/srva-key.pem"
    # Optional certificate authority verifying connected routes
    # Required when we have self-signed CA, etc.
    ca_file:   "./configs/certs/ca.pem"
  }
  # Routes are actively solicited and connected to from this server.
  # Other servers can connect to us if they supply the correct credentials
  # in their routes definitions from above.
  routes = [
    nats://127.0.0.1:4246
  ]
}

NKeys

NKeys are a new, highly secure public-key signature system based on Ed25519.

With NKeys the server can verify identities without ever storing or ever seeing private keys. The authentication system works by requiring a connecting client to provide its public key and digitally sign a challenge with its private key. The server generates a random challenge with every connection request, making it immune to playback attacks. The generated signature is validated against the provided public key, thus proving the identity of the client. If the public key is known to the server, authentication succeeds.

NKey is an excellent replacement for token authentication because a connecting client will have to prove it controls the private key for the authorized public key.

To generate nkeys, you'll need the nk tool.

Generating NKeys and Configuring the Server

To generate a User NKEY:

nk -gen user -pubout

SUACSSL3UAHUDXKFSNVUZRF5UHPMWZ6BFDTJ7M6USDXIEDNPPQYYYCU3VY
UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4

The first output line starts with the letter S for Seed. The second letter, U stands for User. Seeds are private keys; you should treat them as secrets and guard them with care.

The second line starts with the letter U for User and is a public key which can be safely shared.

To use nkey authentication, add a user, and set the nkey property to the public key of the user you want to authenticate:

authorization: {
  users: [
    { nkey: UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4 }
  ]
}

Note that the user section sets the nkey property (user/password/token properties are not needed). Add permission sections as required.

Client Configuration

Now that you have a user nkey, let's configure a client to use it for authentication. As an example, here are the connect options for the node client:

const NATS = require('nats');
const nkeys = require('ts-nkeys');

const nkey_seed = ‘SUACSSL3UAHUDXKFSNVUZRF5UHPMWZ6BFDTJ7M6USDXIEDNPPQYYYCU3VY’;
const nc = NATS.connect({
  port: PORT,
  nkey: 'UDXU4RCSJNZOIQHZNWXHXORDPRTGNJAHAHFRGZNEEJCPQTT2M7NLCNF4',
  sigCB: function (nonce) {
    // client loads seed safely from a file
    // or some constant like `nkey_seed` defined in
    // the program
    const sk = nkeys.fromSeed(Buffer.from(nkey_seed));
    return sk.sign(nonce);
   }
});
...

The client provides a function that it uses to parse the seed (the private key) and sign the connection challenge.

Authentication Timeout

You can specify a timeout to limit how long the server will wait for a client to authenticate.

If you don't specify a value, or if you specify the value "0", then the default will be 1 second more than the tls_timeout.

If you do specify an invalid value, it will use a default of 1 second.

If a client doesn't authenticate to the server within the specified time, the server disconnects the server to prevent abuses.

Timeouts are specified in seconds (and can be fractional). Unlike tls_timeout, you cannot use "human readable" values like 10s, you must specify a number, which will be interpreted as seconds. 10 will be 10 seconds, 3.5 will be 3 seconds and 500 milliseconds, etc.

As with TLS timeouts, long timeouts can be an opportunity for abuse. If setting the authentication timeout, it is important to note that it should be longer than the tls timeout option, as the authentication timeout includes the TLS upgrade time.

authorization: {
    timeout: 3
    users: [
        {user: a, password b},
        {user: b, password a}
    ]
}

Decentralized JWT Authentication/Authorization

With other authentication mechanisms, configuration for identifying a user and Account, is in the server configuration file. JWT authentication leverages JSON Web Tokens (JWT) to describe the various entities supported. When a client connects, servers verify the authenticity of the request using NKeys, download account information and validate a trust chain. Users are not directly tracked by the server, but rather verified as and belonging to an Account. This enables the management of users, without requiring server configuration updates.

Effectively, JWTs improve accounts and provide for a distributed configuration paradigm. Previously each user (or client) needed to be known and authorized a priori in the server’s configuration requiring an administrator to modify and update server configurations. These chores are eliminated. User creation can even be performed by different entities altogether.

Note: This scheme improves accounts. Functionalities like isolation or defining exports/imports between accounts remain! It moves configuration of accounts, exports/imports or users and their permissions away from the server into several trusted JSON Web Token (JWT) that are managed separately, therefore removing the need to configure these entities in each and every server. It furthermore adds functionalities like expiration and revocation fore decentralized account management

JSON Web Tokens

JSON Web Tokens (JWT) are an open and industry standard RFC7519 method for representing claims securely between two parties.

Claims are a fancy way of asserting information on a subject. In this context, a subject is the entity being described (not a messaging subject). Standard JWT claims are typically digitally signed and verified.

NATS further restricts JWTs by requiring that JWTs be:

• Digitally signed always and only using Ed25519.
• NATS adopts the convention that all Issuer and Subject fields in a JWT claim must be a public NKEY.
• Issuer and Subject must match specific roles depending on the claim NKeys.

NKey Roles

NKeys Roles are:

• Operators
• Accounts
• Users

Roles are hierarchical and form a chain of trust. Operators issue Accounts which in turn issue Users. Servers trust specific Operators. If an account is issued by an operator that is trusted, account users are trusted.

The Authentication Process

When a User connects to a server, it presents a JWT issued by its Account. The user proves its identity by signing a server-issued cryptographic challenge with its private key. The signature verification validates that the signature is attributable to the user's public key. Next, the server retrieves the associated account JWT that issued the user. It verifies the User issuer matches the referenced account. Finally, the server checks that a trusted Operator - one the server is configured with - issued the Account, completing the trust chain verification.

The Authorization Process

From an authorization point of view, the account provides information on messaging subjects that are imported from other accounts (including any ancillary related authorization) as well as messaging subjects exported to other accounts. Accounts can also bear limits, such as the maximum number of connections they may have. A user JWT can express restrictions on the messaging subjects to which it can publish or subscribe.

When a new user is added to an account, the account configuration need not change, as each user can and should have its own user JWT that can be verified by simply resolving its parent account.

JWTs and Privacy

One crucial detail to keep in mind is that while in other systems JWTs are used as sessions or proof of authentication, NATS JWTs are only used as configuration describing:

• the public ID of the entity
• the public ID of the entity that issued it
• capabilities of the entity

Authentication is a public key cryptographic process — a client signs a nonce proving identity while the trust chain and configuration provides the authorization.

The server is never aware of private keys but can verify that a signer or issuer indeed matches a specified or known public key.

Lastly, all NATS JWTs (Operators, Accounts, Users and others) are expected to be signed using the Ed25519 algorithm. If they are not, they are rejected by the system.

Decentralized Authentication and Authorization - Configuration and nsc

There is very little to configure on the nats-server to enable operator JWT security, once the servers have been initially configured the authentication and authorization tasks are typically done by using the nsc administration tool locally and synchronizing with the account resolvers built into the nats-server.

Configuration is broken up into separate steps. Depending on organizational needs these are performed by the same or different entities.

Practically, JWT configuration is done using the nsc tool. It can be set up to issue NKeys and corresponding JWTs for all nkey roles: Operator/Account/User (Example usage). Despite Account and User creation not happening in server configuration, this model is a centralized authentication and authorization setup.

Provided institutional trust, it is also possible to use nsc to import account or user public NKeys and issue corresponding JWTs. This way an operator can issue account JWTs and a separate entity can issue JWTs for user associated with it's account. Neither entity has to be aware of the other's private Nkey. This not only allows users to be configured some place other than servers, but also by different organizations altogether. Say administrators of a NATS installation controlling operators, issuing account JWTs to individual prod/dev teams managing their own user. This is a fully decentralized authorization setup!

With an Operator JWT in place, the server needs to be configured to trust it by specifying operator. Furthermore the server needs a way to obtain account JWTs. This done by either defaulting to the resolver specified in the operator jwt or by manually specifying the resolver. Depending on your configuration an account server needs to be in place

It is possible to mix JWT and NKEY/Account based Authentication/Authorization.

Managing JWT authentication

A lot more information is available in the In Depth Guide.

Account lookup using Resolver

The resolver configuration option is used in conjunction with NATS JWT Authentication and nsc. The resolver option specifies a URL where the nats-server can retrieve an account JWT. There are 3 resolver implementations:

• NATS Based Resolver which is the preferred option and should be your default selection
• MEMORY if you want to statically define the accounts in the server configuration
• URL if you want to build your own account service, typically in order to have some integration of NATS security with some external security system.

If the operator JWT specified in operator contains an account resolver URL, resolver only needs to be specified in order to overwrite that default.

NATS Based Resolver

The NATS based resolver is the preferred and easiest way to enable account lookup for the nats servers. It is built-in into nats-server and stores the account JWTs in a local (not shared) directory that the server has access to (i.e. you can't have more than one nats-servers using the same directory. All the servers in the cluster or super-cluster must be configured to use it, and they implement an 'eventually consistent' mechanism via NATS and the system account to synchronize (or lookup) the account data between themselves.

In order to avoid having to store all account JWT on every nats-server (i.e. if you have a lot of accounts), this resolver has two sub types full and cache.

In this mode of operation administrators typically use the nsc CLI tool to create/manage the JWTs locally, and use nsc push to push new JWTs to the nats-servers' built-in resolvers, nsc pull to refresh their local copy of account JWTs, and nsc revocations to revoke them.

Full

The Full resolver means that the nats-server stores all JWTs and exchanges them in an eventually consistent way with other resolvers of the same type.

resolver: {
    type: full
    # Directory in which account jwt will be stored
    dir: './jwt'
    # In order to support jwt deletion, set to true
    # If the resolver type is full delete will rename the jwt.
    # This is to allow manual restoration in case of inadvertent deletion.
    # To restore a jwt, remove the added suffix .delete and restart or send a reload signal.
    # To free up storage you must manually delete files with the suffix .delete.
    allow_delete: false
    # Interval at which a nats-server with a nats based account resolver will compare
    # it's state with one random nats based account resolver in the cluster and if needed,
    # exchange jwt and converge on the same set of jwt.
    interval: "2m"
    # limit on the number of jwt stored, will reject new jwt once limit is hit.
    limit: 1000
}

This resolver type also supports resolver_preload. When present, JWTs are listed and stored in the resolver. There, they may be subject to updates. Restarts of the nats-server will hold on to these more recent versions.

Not every server in a cluster needs to be set to full. You need enough to still serve your workload adequately, while some servers are offline.

Cache

The Cache resolver means that the nats-server only stores a subset of the JWTs and evicts others based on an LRU scheme. The cache relies on (a) full NATS-based resolver(s) to retrieve accounts not present in the cache. A cache resolver does NOT accept account push messages from nsc and therefore is not suitable for stand-alone operation without a full resolver present.

resolver: {
    type: cache
    # Directory in which account jwt will be store
    dir: "./"
    # limit on the number of jwt stored, will evict old jwt once limit is hit.
    limit: 1000
    # How long to hold on to a jwt before discarding it. 
    ttl: "2m"
}

NATS-Based Resolver - Integration

The NATS-based resolver utilizes the system account for lookup and upload of account JWTs. If your application requires tighter integration you can make use of these subjects for tighter integration.

To upload or update any generated account JWT without nsc, send it as a request to $SYS.REQ.CLAIMS.UPDATE. Each participating full NATS-based account resolver will respond with a message detailing success or failure.

To serve a requested account JWT yourself and essentially implement an account server, subscribe to $SYS.REQ.ACCOUNT.*.CLAIMS.LOOKUP and respond with the account JWT corresponding to the requested account id (wildcard).

Migrating account data

To migrate account data when you change from using the standalone (REST) account server to the built-in NATS account resolver (or between NATS environments, or account servers) you can use nsc:

1. Run nsc pull to make sure you have a copy of all the account data in the server in your local machine
2. Reconfigure your servers to use the nats resolver instead of the URL resolver
3. Modify the 'account server URL' setting in your operator to the nats URL from the old REST URL: i.e. just copy the nats URLs from the operator's 'service URLs' setting into the account server URLs. nsc edit operator --account-jwt-server-url <nats://...>
4. do nsc push -A to push your account data to the nats-servers using the built-in nats account resolver

You can also pass the account server URLs directly as a flag to the nsc pull and nsc push commands.

MEMORY

The MEMORY resolver is statically configured in the server's configuration file. You would use this mode if you would rather manage the account resolving 'by hand' through the nat-servers' configuration files. The memory resolver makes use of the resolver_preload directive, which specifies a map of public keys to account JWTs:

resolver: MEMORY
resolver_preload: {
ACSU3Q6LTLBVLGAQUONAGXJHVNWGSKKAUA7IY5TB4Z7PLEKSR5O6JTGR: eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5In0.eyJqdGkiOiJPRFhJSVI2Wlg1Q1AzMlFJTFczWFBENEtTSDYzUFNNSEZHUkpaT05DR1RLVVBISlRLQ0JBIiwiaWF0IjoxNTU2NjU1Njk0LCJpc3MiOiJPRFdaSjJLQVBGNzZXT1dNUENKRjZCWTRRSVBMVFVJWTRKSUJMVTRLM1lERzNHSElXQlZXQkhVWiIsIm5hbWUiOiJBIiwic3ViIjoiQUNTVTNRNkxUTEJWTEdBUVVPTkFHWEpIVk5XR1NLS0FVQTdJWTVUQjRaN1BMRUtTUjVPNkpUR1IiLCJ0eXBlIjoiYWNjb3VudCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJjb25uIjotMSwibGVhZiI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJ3aWxkY2FyZHMiOnRydWV9fX0._WW5C1triCh8a4jhyBxEZZP8RJ17pINS8qLzz-01o6zbz1uZfTOJGvwSTS6Yv2_849B9iUXSd-8kp1iMXHdoBA
}

The MEMORY resolver is recommended when the server has a small number of accounts that don't change very often.

For more information on how to configure a memory resolver, see this tutorial.

URL Resolver

NOTE: The standalone NATS Account JWT Server is now legacy, please use the NATS Based Resolver instead. However, the URL resolver option is still available in case you want to implement your own version of an account resolver

The URL resolver specifies a URL where the server can append an account public key to retrieve that account's JWT. Convention for standalone NATS Account JWT Servers is to serve JWTs at: http://localhost:9090/jwt/v1/accounts/. For such a configuration, you would specify the resolver as follows:

resolver: URL(http://localhost:9090/jwt/v1/accounts/)

Note that if you are not using a nats-account-server, the URL can be anything as long as by appending the public key for an account, the requested JWT is returned.

If the server used requires client authentication, or you want to specify which CA is trusted for the lookup of account information, specify resolver_tls. This tls configuration map lets you further restrict TLS to the resolver.

Memory Resolver Tutorial

The MEMORY resolver is a server built-in resolver for account JWTs. If there are a small number of accounts, or they do not change too often this can be a simpler configuration that does not require an external account resolver. Server configuration reload is supported, meaning the preloads can be updated in the server configuration and reloaded without a server restart.

The basic configuration for the server requires:

• The operator JWT
• resolver set to MEMORY
• resolver_preload set to an object where account public keys are mapped to account JWTs.

Create Required Entities

Let's create the setup:

nsc add operator -n memory

Generated operator key - private key stored "~/.nkeys/memory/memory.nk"
Success! - added operator "memory"

Add an account 'A'

nsc add account --name A

Generated account key - private key stored "~/.nkeys/memory/accounts/A/A.nk"
Success! - added account "A"

Describe the account

nsc describe account -W

╭──────────────────────────────────────────────────────────────────────────────────────╮
│                                   Account Details                                    │
├───────────────────────────┬──────────────────────────────────────────────────────────┤
│ Name                      │ A                                                        │
│ Account ID                │ ACSU3Q6LTLBVLGAQUONAGXJHVNWGSKKAUA7IY5TB4Z7PLEKSR5O6JTGR │
│ Issuer ID                 │ ODWZJ2KAPF76WOWMPCJF6BY4QIPLTUIY4JIBLU4K3YDG3GHIWBVWBHUZ │
│ Issued                    │ 2019-04-30 20:21:34 UTC                                  │
│ Expires                   │                                                          │
├───────────────────────────┼──────────────────────────────────────────────────────────┤
│ Max Connections           │ Unlimited                                                │
│ Max Leaf Node Connections │ Unlimited                                                │
│ Max Data                  │ Unlimited                                                │
│ Max Exports               │ Unlimited                                                │
│ Max Imports               │ Unlimited                                                │
│ Max Msg Payload           │ Unlimited                                                │
│ Max Subscriptions         │ Unlimited                                                │
│ Exports Allows Wildcards  │ True                                                     │
├───────────────────────────┼──────────────────────────────────────────────────────────┤
│ Imports                   │ None                                                     │
│ Exports                   │ None                                                     │
╰───────────────────────────┴──────────────────────────────────────────────────────────╯

Create a new user 'TA'

nsc add user --name TA

Generated user key - private key stored "~/.nkeys/memory/accounts/A/users/TA.nk"
Generated user creds file "~/.nkeys/memory/accounts/A/users/TA.creds"
Success! - added user "TA" to "A"

Create the Server Config

The nsc tool can generate a configuration file automatically. You provide a path to the server configuration. The nsc tool will generate the server config for you:

nsc generate config --mem-resolver --config-file /tmp/server.conf 

If you require additional settings, you may want to consider using include in your main configuration, to reference the generated files. Otherwise, you can start a server and reference the generated configuration:

nats-server -c /tmp/server.conf

You can then test it.

Manual Server Config

While generating a configuration file is easy, you may want to craft one by hand to know the details. With the entities created, and a standard location for the .nsc directory. You can reference the operator JWT and the account JWT in a server configuration or the JWT string directly. Remember that your configuration will be in $NSC_HOME/nats/<operator_name>/<operator_name>.jwt for the operator. The account JWT will be in $NSC_HOME/nats/<operator_name>/accounts/<account_name>/<account_name>.jwt

For the configuration you'll need:

• The path to the operator JWT
• A copy of the contents of the account JWT file

The format of the file is:

operator: <path to the operator jwt or jwt itself>
resolver: MEMORY
resolver_preload: {
    <public key for an account>: <contents of the account jwt>
    ### add as many accounts as you want
    ...
}

In this example this translates to:

operator: /Users/synadia/.nsc/nats/memory/memory.jwt
resolver: MEMORY
resolver_preload: {
ACSU3Q6LTLBVLGAQUONAGXJHVNWGSKKAUA7IY5TB4Z7PLEKSR5O6JTGR: eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5In0.eyJqdGkiOiJPRFhJSVI2Wlg1Q1AzMlFJTFczWFBENEtTSDYzUFNNSEZHUkpaT05DR1RLVVBISlRLQ0JBIiwiaWF0IjoxNTU2NjU1Njk0LCJpc3MiOiJPRFdaSjJLQVBGNzZXT1dNUENKRjZCWTRRSVBMVFVJWTRKSUJMVTRLM1lERzNHSElXQlZXQkhVWiIsIm5hbWUiOiJBIiwic3ViIjoiQUNTVTNRNkxUTEJWTEdBUVVPTkFHWEpIVk5XR1NLS0FVQTdJWTVUQjRaN1BMRUtTUjVPNkpUR1IiLCJ0eXBlIjoiYWNjb3VudCIsIm5hdHMiOnsibGltaXRzIjp7InN1YnMiOi0xLCJjb25uIjotMSwibGVhZiI6LTEsImltcG9ydHMiOi0xLCJleHBvcnRzIjotMSwiZGF0YSI6LTEsInBheWxvYWQiOi0xLCJ3aWxkY2FyZHMiOnRydWV9fX0._WW5C1triCh8a4jhyBxEZZP8RJ17pINS8qLzz-01o6zbz1uZfTOJGvwSTS6Yv2_849B9iUXSd-8kp1iMXHdoBA
}

Save the config at server.conf and start the server:

nats-server -c server.conf

You can then test it.

Testing the Configuration

To test the configuration, simply use one of the standard tools:

nats pub --creds ~/.nkeys/creds/memory/A/TA.creds hello world

Mixed Authentication/Authorization Setup

Mixing both nkeys static config and decentralized JWT Authentication/Authorization is possible but needs some preparation in order to be able to do it.

The way this can be done is by first preparing a basic trusted operator setup that could be used in the future, and then base from that configuration to create the NKEYS static config using the same shared public nkeys for the accounts and then use clustering routes to bridge the two different auth setups during the transition.

For example, creating the following initial setup using NSC:

        nsc add account --name SYS
        nsc add user    --name sys
        nsc add account --name A
        nsc add user -a A --name test
        nsc add account --name B
        nsc add user -a B --name test

This will then generate something like the following:

 nsc list accounts

╭─────────────────────────────────────────────────────────────────╮
│                            Accounts                             │
├──────┬──────────────────────────────────────────────────────────┤
│ Name │ Public Key                                               │
├──────┼──────────────────────────────────────────────────────────┤
│ A    │ ADFB2JXYTXOJEL6LNAXDREUGRX35BOLZI3B4PFFAC7IRPR3OA4QNKBN2 │
│ B    │ ACWOMQA7PZTKJSBTR7BF6TBK3D776734PWHWDKO7HFMQOM5BIOYPSYZZ │
│ SYS  │ ABKOWIYVTHNEK5HELPWLAT2CF2CUPELIK4SZH2VCJHLFU22B5U2IIZUO │
╰──────┴──────────────────────────────────────────────────────────╯

 nsc list users -a A

╭─────────────────────────────────────────────────────────────────╮
│                              Users                              │
├──────┬──────────────────────────────────────────────────────────┤
│ Name │ Public Key                                               │
├──────┼──────────────────────────────────────────────────────────┤
│ test │ UAPOK2P7EN3UFBL7SBJPQK3M3JMLALYRYKX5XWSVMVYK63ZMBHTOHVJR │
╰──────┴──────────────────────────────────────────────────────────╯

We could use this configuration as the initial starting configuration for an nkeys config now, where all the NKEYS users public nkeys are explicitly listed (centralized auth model).

port = 4222

cluster {
  port = 6222

  # We will bridge two different servers with different auth models via routes
  # routes [ nats://127.0.0.1:6223 ]
}

system_account = ABKOWIYVTHNEK5HELPWLAT2CF2CUPELIK4SZH2VCJHLFU22B5U2IIZUO

accounts {
  # Account A
  ADFB2JXYTXOJEL6LNAXDREUGRX35BOLZI3B4PFFAC7IRPR3OA4QNKBN2 {
    nkey: ADFB2JXYTXOJEL6LNAXDREUGRX35BOLZI3B4PFFAC7IRPR3OA4QNKBN2
    users = [
      {nkey: "UAPOK2P7EN3UFBL7SBJPQK3M3JMLALYRYKX5XWSVMVYK63ZMBHTOHVJR" }
    ]
  }

  # Account B
  ACWOMQA7PZTKJSBTR7BF6TBK3D776734PWHWDKO7HFMQOM5BIOYPSYZZ {
  }

  # Account SYS
  ABKOWIYVTHNEK5HELPWLAT2CF2CUPELIK4SZH2VCJHLFU22B5U2IIZUO {
  }
}

By using nsc it is possible to create a mem based resolver for the trusted operator setup:

nsc generate config --mem-resolver --sys-account SYS

An example configuration from the second node with the trusted operator setup could then be:

port = 4223

cluster {
  port = 6223
  routes [ nats://127.0.0.1:6222 ]
}

# debug = true
# trace = true

# Operator
operator = eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5In0.eyJqdGkiOiJQNDJBSkFTVVA0TUdGRU1EQzVCRVVGUkM1MlQ1M05OTzRIWkhRNEdETVk0S0RZTFVRV0JBIiwiaWF0IjoxNTc0Mzc1OTE2LCJpc3MiOiJPQ09KSk5aSUNINkNHUU5LM1NRMzVXTFpXWkpDUkRBTFJIWjZPVzQ0RFpZVVdNVVYzV1BSSEZSRCIsIm5hbWUiOiJLTyIsInN1YiI6Ik9DT0pKTlpJQ0g2Q0dRTkszU1EzNVdMWldaSkNSREFMUkhaNk9XNDREWllVV01VVjNXUFJIRlJEIiwidHlwZSI6Im9wZXJhdG9yIiwibmF0cyI6e319.pppa9-xhWXJLSCCtqj_dqlvXKR7WlVCh0cqoZ6lr8zg3WlWM8U0bNf6FHw_67-wRS7jj0n4PuA0P0MAJdE3pDA

system_account = ABKOWIYVTHNEK5HELPWLAT2CF2CUPELIK4SZH2VCJHLFU22B5U2IIZUO

resolver = MEMORY

resolver_preload = {
  # Account "A"
  ADFB2JXYTXOJEL6LNAXDREUGRX35BOLZI3B4PFFAC7IRPR3OA4QNKBN2: eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5In0.eyJqdGkiOiIyQjNYNUJPQzQ3M05TU0hFWUVHTzJYRUhaTVNGRVBORFFBREJXWkJLRVdQVlg2TUlZU1JRIiwiaWF0IjoxNTc0Mzc1OTE2LCJpc3MiOiJPQ09KSk5aSUNINkNHUU5LM1NRMzVXTFpXWkpDUkRBTFJIWjZPVzQ0RFpZVVdNVVYzV1BSSEZSRCIsIm5hbWUiOiJBIiwic3ViIjoiQURGQjJKWFlUWE9KRUw2TE5BWERSRVVHUlgzNUJPTFpJM0I0UEZGQUM3SVJQUjNPQTRRTktCTjIiLCJ0eXBlIjoiYWNjb3VudCIsIm5hdHMiOnsiZXhwb3J0cyI6W3sibmFtZSI6InRlc3QiLCJzdWJqZWN0IjoidGVzdCIsInR5cGUiOiJzZXJ2aWNlIiwic2VydmljZV9sYXRlbmN5Ijp7InNhbXBsaW5nIjoxMDAsInJlc3VsdHMiOiJsYXRlbmN5Lm9uLnRlc3QifX1dLCJsaW1pdHMiOnsic3VicyI6LTEsImNvbm4iOi0xLCJsZWFmIjotMSwiaW1wb3J0cyI6LTEsImV4cG9ydHMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsIndpbGRjYXJkcyI6dHJ1ZX19fQ.zZBetgDN6nCFDVpwzF_124BPkc8amGPDnrOmiKUa12xski5zskUI0Y0OeIa1vTo0bkHIKTgM2QDYpmXUQOHnAQ

  # Account "B"
  ACWOMQA7PZTKJSBTR7BF6TBK3D776734PWHWDKO7HFMQOM5BIOYPSYZZ: eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5In0.eyJqdGkiOiJBTFNFQkZGWDZMR0pQTlVMU1NXTDNTRTNISkk2WUZSWlVKSDNLV0E1VE41WUtWRE5MVTJRIiwiaWF0IjoxNTc0Mzc1OTE2LCJpc3MiOiJPQ09KSk5aSUNINkNHUU5LM1NRMzVXTFpXWkpDUkRBTFJIWjZPVzQ0RFpZVVdNVVYzV1BSSEZSRCIsIm5hbWUiOiJCIiwic3ViIjoiQUNXT01RQTdQWlRLSlNCVFI3QkY2VEJLM0Q3NzY3MzRQV0hXREtPN0hGTVFPTTVCSU9ZUFNZWloiLCJ0eXBlIjoiYWNjb3VudCIsIm5hdHMiOnsiaW1wb3J0cyI6W3sibmFtZSI6InRlc3QiLCJzdWJqZWN0IjoidGVzdCIsImFjY291bnQiOiJBREZCMkpYWVRYT0pFTDZMTkFYRFJFVUdSWDM1Qk9MWkkzQjRQRkZBQzdJUlBSM09BNFFOS0JOMiIsInRvIjoidGVzdCIsInR5cGUiOiJzZXJ2aWNlIn1dLCJsaW1pdHMiOnsic3VicyI6LTEsImNvbm4iOi0xLCJsZWFmIjotMSwiaW1wb3J0cyI6LTEsImV4cG9ydHMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsIndpbGRjYXJkcyI6dHJ1ZX19fQ.AnzziBwt5Tnphc2prONUUOpMpkkAlJHvCPaag0GUtTYPCHKDphcJrwtAHi4v5NOI6npjoes0F0MlrfnHqidDAg

  # Account "SYS"
  ABKOWIYVTHNEK5HELPWLAT2CF2CUPELIK4SZH2VCJHLFU22B5U2IIZUO: eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5In0.eyJqdGkiOiI1WVUyWkc1UkRTSU1TN1pGVE1MU0NZQUtLVkVFWUpPUlc0TDJPTlY3N1g1TlJZWkFGSkRRIiwiaWF0IjoxNTc0Mzc1OTE2LCJpc3MiOiJPQ09KSk5aSUNINkNHUU5LM1NRMzVXTFpXWkpDUkRBTFJIWjZPVzQ0RFpZVVdNVVYzV1BSSEZSRCIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBQktPV0lZVlRITkVLNUhFTFBXTEFUMkNGMkNVUEVMSUs0U1pIMlZDSkhMRlUyMkI1VTJJSVpVTyIsInR5cGUiOiJhY2NvdW50IiwibmF0cyI6eyJsaW1pdHMiOnsic3VicyI6LTEsImNvbm4iOi0xLCJsZWFmIjotMSwiaW1wb3J0cyI6LTEsImV4cG9ydHMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsIndpbGRjYXJkcyI6dHJ1ZX19fQ.5FrO4sZbWuFgRLuy7c1eQLUq_BQ4PNhIAN5A-sRLkYWmvlc4c_Y4VfTbgl5zhNzCxfvj9SxT7ySgphup2BiRAA
}

Even though they have different authorization mechanisms, these two servers are able to route account messages because they share the same NKEY.

We have created at least one user, in this case with creds:

-----BEGIN NATS USER JWT-----
eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5In0.eyJqdGkiOiJNRkM3V1E1N0hKUE9aWUVOTEhVRTZTWFVQTDVKTURWSkxIQzJRTkpYNUVJS0RGR1U1REhRIiwiaWF0IjoxNTc0Mzc1OTE2LCJpc3MiOiJBREZCMkpYWVRYT0pFTDZMTkFYRFJFVUdSWDM1Qk9MWkkzQjRQRkZBQzdJUlBSM09BNFFOS0JOMiIsIm5hbWUiOiJ0ZXN0Iiwic3ViIjoiVUFQT0syUDdFTjNVRkJMN1NCSlBRSzNNM0pNTEFMWVJZS1g1WFdTVk1WWUs2M1pNQkhUT0hWSlIiLCJ0eXBlIjoidXNlciIsIm5hdHMiOnsicHViIjp7ImFsbG93IjpbIl9JTkJPWC5cdTAwM2UiLCJfUl8iLCJfUl8uXHUwMDNlIiwidGVzdCIsInRlc3QuXHUwMDNlIl19LCJzdWIiOnsiYWxsb3ciOlsiX0lOQk9YLlx1MDAzZSIsIl9SXyIsIl9SXy5cdTAwM2UiLCJsYXRlbmN5Lm9uLnRlc3QiLCJ0ZXN0IiwidGVzdC5cdTAwM2UiXX19fQ.MSU2aUIBK1iUsg7h52lLrfEfTwVMF_wB3HDq75ECskxSyyDDMtk9_3957UtQF-3yoGCIhKOkWjzX8C-WXnLADw
------END NATS USER JWT------
************************* IMPORTANT *************************
NKEY Seed printed below can be used to sign and prove identity.
NKEYs are sensitive and should be treated as secrets.
-----BEGIN USER NKEY SEED-----
SUANVBWRHHFMGHNIT6UJHPN2TGVBVIILE7VPVNEQ7DGCJ26ZD2V3KAHT4M
------END USER NKEY SEED------
*************************************************************

And this same user is able to connect to either one of the servers (bound to 4222 and 4223 respectively):

Subscriber Service:

package main

import (
    "log"

    "github.com/nats-io/nats.go"
)

func main() {
    opts := make([]nats.Option, 0)

    // Extract public nkey from seed
    //
    // Public:  UAPOK2P7EN3UFBL7SBJPQK3M3JMLALYRYKX5XWSVMVYK63ZMBHTOHVJR
    // Private: SUANVBWRHHFMGHNIT6UJHPN2TGVBVIILE7VPVNEQ7DGCJ26ZD2V3KAHT4M
    // 
    nkey, err := nats.NkeyOptionFromSeed("path/to/seed.nkey")
    if err != nil {
        log.Fatal(err)
    }
    opts = append(opts, nkey)
    nc, err := nats.Connect("127.0.0.1:4222", opts...)
    if err != nil {
        log.Fatal(err)
    }
    nc.Subscribe("test", func(m *nats.Msg){
        log.Printf("[Received] %q, replying... \n", string(m.Data))
        m.Respond([]byte("pong from nkeys based server"))
    })

    select {}
}

Requestor:

package main

import (
    "log"
    "time"

    "github.com/nats-io/nats.go"
)

func main() {
    nc, err := nats.Connect("127.0.0.1:4223", nats.UserCredentials("path/to/user.creds"))
    if err != nil {
        log.Fatal(err)
    }

    for range time.NewTicker(1 * time.Second).C {
        resp, err := nc.Request("test", []byte("test"), 1*time.Second)
        if err != nil {
            log.Println("[Error]", err)
            continue
        }
        log.Println("[Received]", string(resp.Data))
    }
}

Authorization

The NATS server supports authorization using subject-level permissions on a per-user basis. Permission-based authorization is available with multi-user authentication via the users list.

Each permission specifies the subjects the user can publish to and subscribe to. The parser is generous at understanding what the intent is, so both arrays and singletons are processed. For more complex configuration, you can specify a permission object which explicitly allows or denies subjects. The specified subjects can specify wildcards as well. Permissions can make use of variables.

A special field inside the authorization map is default_permissions. When present, it contains permissions that apply to users that do not have permissions associated with them.

Permissions Configuration Map

The permissions map specify subjects that can be subscribed to or published by the specified client.

Permission Map

The permission map provides additional properties for configuring a permissions map. Instead of providing a list of allowable subjects and optional queues, the permission map allows you to explicitly list those you want toallow or deny. Both lists can be provided. In case of overlap deny has priority.

Important Note It is important to not break request-reply patterns. In some cases (as shown below) you need to add rules for the _INBOX.> pattern. If an unauthorized client publishes or attempts to subscribe to a subject that has not been allow listed, the action fails and is logged at the server, and an error message is returned to the client. The allow responses option can simplify this.

Allow Responses Map

The allow_responses option dynamically allows publishing to reply subjects and is designed for service responders. When set to true, an implicit publish allow permission is enforced which enables the service to have temporary permission to publish to the reply subject during a request-reply exchange. If true, the client supports a one-time publish. If allow_responses is a map, it allows you to configure a maximum number of responses and how long the permission is valid.

警告

Note, when allow_responses is enabled, the reply subject is not constrained to the publish allow or deny list. The implication of this is that a reply subject can be provided by a client to a service (responder) that does not have permission to explicitly publish on that subject, but is temporarily allowed given this option. If explicit control over which subjects a client is allowed to reply to, do not use allow_responses and instead define allow/deny lists under the publish permission map.

When allow_responses is set to true, it defaults to the equivalent of { max: 1 } and no time limit.

Important Note When using nsc to configure your users, you can specify the --allow-pub-response and --response-ttl to control these settings.

Examples

Variables

Here is an example authorization configuration that uses variables which defines four users, three of whom are assigned explicit permissions.

authorization {
  default_permissions = {
    publish = "SANDBOX.*"
    subscribe = ["PUBLIC.>", "_INBOX.>"]
  }
  ADMIN = {
    publish = ">"
    subscribe = ">"
  }
  REQUESTOR = {
    publish = ["req.a", "req.b"]
    subscribe = "_INBOX.>"
  }
  RESPONDER = {
    subscribe = ["req.a", "req.b"]
    publish = "_INBOX.>"
  }
  users = [
    {user: admin,   password: $ADMIN_PASS, permissions: $ADMIN}
    {user: client,  password: $CLIENT_PASS, permissions: $REQUESTOR}
    {user: service,  password: $SERVICE_PASS, permissions: $RESPONDER}
    {user: other, password: $OTHER_PASS}
  ]
}

default_permissions is a special entry. If defined, it applies to all users that don't have specific permissions set.

• admin has ADMIN permissions and can publish/subscribe on any subject. We use the wildcard > to match any subject.
• client is a REQUESTOR and can publish requests on subjects req.a or req.b, and subscribe to anything that is a response (_INBOX.>).
• service is a RESPONDER to req.a and req.b requests, so it needs to be able to subscribe to the request subjects and respond to clients that can publish requests to req.a and req.b. The reply subject is an inbox. Typically inboxes start with the prefix _INBOX. followed by a generated string. The _INBOX.> subject matches all subjects that begin with _INBOX..
• other has no permissions granted and therefore inherits the default permission set.

Note that in the above example, any client with permission to subscribe to _INBOX.> can receive all responses published. More sensitive installations will want to add or subset the prefix to further limit subjects that a client can subscribe. Alternatively, Accounts allow complete isolation limiting what members of an account can see.

Allow/Deny Specified

Here's an example without variables, where the allow and deny options are specified:

authorization: {
    users = [
        {
            user: admin
            password: secret
            permissions: {
                publish: ">"
                subscribe: ">"
            }
        }
        {
            user: test
            password: test
            permissions: {
                publish: {
                    deny: ">"
                },
                subscribe: {
                    allow: "client.>"
                }
            }
        }
    ]
}

allow_responses

Here's an example with allow_responses:

authorization: {
    users: [
        { user: a, password: a },
        { user: b, password: b, permissions: {subscribe: "q", allow_responses: true } },
        { user: c, password: c, permissions: {subscribe: "q", allow_responses: { max: 5, expires: "1m" } } }
        { user: d, password: d, permissions: {subscribe: "q", publish: "x", allow_responses: true } }
    ]
}

• User a has no restrictions.
• User b can listen on q for requests and can only publish once to reply subjects. All other publish subjects are denied implicitly when allow_responses is set.
• User c can listen on q for requests, but is able to return at most 5 reply messages, and the reply subject can be published at most for 1 minute.
• User d has the same behavior as user b, except that it can explicitly publish to subject x as well, which overrides the implicit deny from allow_responses.

Queue Permissions

User a can only subscribe to foo as part of the queue subscriptions queue. User b has permissions for queue subscriptions as well as plain subscriptions. You can allow plain subscriptions on foo but constrain the queues to which a client can join, as well as preventing any service from using a queue subscription with the name *.prod:

users = [
  {
    user: "a", password: "a", permissions: {
      sub: {
        allow: ["foo queue"]
     }
  }
  {
    user: "b", password: "b", permissions: {
      sub: {
        # Allow plain subscription foo, but only v1 groups or *.dev queue groups
        allow: ["foo", "foo v1", "foo v1.>", "foo *.dev"]

        # Prevent queue subscriptions on prod groups
        deny: ["> *.prod"]
     }
  }
]

Multi Tenancy using Accounts

In modern microservice architecture it is common to share infrastructure - such as NATS - between services. Accounts are securely isolated communication contexts that allow multi-tenancy in a NATS deployment. They allow users to bifurcate technology from business driven use cases, where data silos are created by design, not software limitations. Furthermore, they facilitate the controlled exchange of information between those data silos/Tenants/Accounts.

Accounts

Accounts expand on the authorization foundation. With traditional authorization, all clients can publish and subscribe to anything unless explicitly configured otherwise. To protect clients and information, you have to carve the subject space and permission clients carefully.

Accounts allow the grouping of clients, isolating them from clients in other accounts, thus enabling multi-tenancy in the server. With accounts, the subject space is not globally shared, greatly simplifying the messaging environment. Instead of devising complicated subject name carving patterns, clients can use short subjects without explicit authorization rules. System Events are an example of this isolation at work.

Accounts configuration is done in accounts map. The contents of an account entry includes:

The accounts list is a map, where the keys on the map are an account name.

accounts: {
    A: {
        users: [
            {user: a, password: a}
        ]
    },
    B: {
        users: [
            {user: b, password: b}
        ]
    },
}

In the most straightforward configuration above you have an account named A which has a single user identified by the username a and the password a, and an account named B with a user identified by the username b and the password b.

These two accounts are isolated from each other. Messages published by users in A are not visible to users in B.

The user configuration map is the same as any other NATS user configuration map . You can use:

• username/password
• nkeys
• and add permissions

While the name account implies one or more users, it is much simpler and enlightening to think of one account as a messaging container for one application. Users in the account are simply the minimum number of services that must work together to provide some functionality. In simpler terms, more accounts with few (even one) clients is a better design topology than a large account with many users with complex authorization configuration.

Exporting and Importing

Messaging exchange between different accounts is enabled by exporting streams and services from one account and importing them into another. Each account controls what is exported and imported.

• Streams are messages your application publishes. Importing applications won't be able to make requests from your applications but will be able to consume messages you generate.
• Services are messages your application can consume and act on, enabling other accounts to make requests that are fulfilled by your account.

提示

The term stream in the context of import and export account configuration does not refer to and should not be confused with a JetStream stream (unfortunate collision of terms as the import/export between accounts predates JetStream), it is just a 'stream of (Core NATS) messages'

The exports configuration list enable you to define the services and streams that others can import. Exported services and streams are expressed as an Export configuration map. The imports configuration lists the services and streams that an Account imports. Imported services and streams are expressed as an Import configuration map.

Export Configuration Map

The export configuration map binds a subject for use as a service or stream and optionally defines specific accounts that can import the stream or service. Here are the supported configuration properties:

Here are some example exports:

accounts: {
    A: {
        users: [
            {user: a, password: a}
        ]
        exports: [
            {stream: puba.>}
            {service: pubq.>}
            {stream: b.>, accounts: [B]}
            {service: q.b, accounts: [B]}
        ]
    }
    ...
}

Here's what A is exporting:

• a public stream on the wildcard subject puba.>
• a public service on the wildcard subject pubq.>
• a stream to account B on the wildcard subject b.>
• a service to account B on the subject q.b

Import Configuration Map

An import enables an account to consume streams published by another account or make requests to services implemented by another account. All imports require a corresponding export on the exporting account. Accounts cannot do self-imports.

The prefix and to options are optional and allow you to remap the subject that is used locally to receive stream messages from or publish service requests to. This way the importing account does not depend on naming conventions picked by another. Currently, a service import can not make use of wildcards, which is why the import subject can be rewritten. A stream import may make use of wildcards. To retain information contained in the subject, it can thus only be prefixed with prefix...

Source Configuration Map

The source configuration map describes an export from a remote account by specifying the account and subject of the export being imported. This map is embedded in the import configuration map:

Import/Export Example

accounts: {
    A: {
        users: [
            {user: a, password: a}
        ]
        exports: [
            {stream: puba.>}
            {service: pubq.>}
            {stream: b.>, accounts: [B]}
            {service: q.b, accounts: [B]}
        ]
    },
    B: {
        users: [
            {user: b, password: b}
        ]
        imports: [
            {stream: {account: A, subject: b.>}}
            {service: {account: A, subject: q.b}}
        ]
    }
    C: {
        users: [
            {user: c, password: c}
        ]
        imports: [
            {stream: {account: A, subject: puba.>}, prefix: from_a}
            {service: {account: A, subject: pubq.C}, to: Q}
        ]
    }
}

Account B imports:

• the private stream from A that only B can receive on b.>
• the private service from A that only B can send requests on q.b

Account C imports the public service and stream from A, but also:

• remaps the puba.> stream to be locally available under from_a.puba.>. The messages will have their original subjects prefixed by from_a.
• remaps the pubq.C service to be locally available under Q. Account C only needs to publish to Q locally.

It is important to reiterate that:

• stream puba.> from A is visible to all external accounts that imports the stream.
• service pubq.> from A is available to all external accounts so long as they know the full subject of where to send the request. Typically an account will export a wildcard service but then coordinate with a client account on specific subjects where requests will be answered. On our example, account C access the service on pubq.C (but has mapped it for simplicity to Q).
• stream b.> is private, only account B can receive messages from the stream.
• service q.b is private; only account B can send requests to the service.
• When C publishes a request to Q, local C clients will see Q messages. However, the server will remap Q to pubq.C and forward the requests to account A.

No Auth User

Clients connecting without authentication can be associated with a particular user within an account.

accounts: {
    A: {
        users: [
            {user: a, password: a}
        ]
    },
    B: {
        users: [
            {user: b, password: b}
        ]
    }
}
no_auth_user: a

The above example shows how clients without authentication can be associated with the user a within account A.

Please note that the no_auth_user will not work with nkeys or bcrypted passwords. The user referenced can also be part of the authorization block.

Despite no_auth_user being set, clients still need to communicate that they will not be using credentials. The authentication timeout applies to this process as well. When your connection is slow, you may run into this timeout and the resulting Authentication Timeout error, despite not providing credentials.

Exporting and importing JetStream streams between accounts

It is possible to import/export messages stored in JetStream streams between accounts. While it is possible to allow a client application in one account to access a stream located in another account, in most use cases people want a setup where a stream in one account is mirrored or sourced in another account (and the applications in that other account simply use that mirrored/sourced stream in their account), as this is a more 'locked-down' way to share messages in streams between accounts, compared to letting the client applications directly use a stream in another account.

There are two resources documenting and giving examples of how to do this:

• Cross account JetStream sourcing explains and has a walkthrough example of how to do this using simple static security (as is probably the best one to start with) and
• Connect Streams Cross Accounts explains how to do the same thing but when using the 'operator' JWT-based security mode of operation.

See Also

• Multi-tenancy and resource management

OCSP Stapling

Supported since NATS Server version 2.3

OCSP Stapling is honored by default for certificates that have the status_request Must-Staple flag.

When a certificate is configured with OCSP Must-Staple, the NATS Server will fetch staples from the configured OCSP responder URL that is present in a certificate. For example, given a certificate with the following configuration:

[ ext_ca ]
...                                                                           
authorityInfoAccess = OCSP;URI:http://ocsp.example.net:80
tlsfeature = status_request
...

The NATS server will make a request to the OCSP responder to fetch a new staple which will then be presented to any TLS connection that is accepted by the server during the TLS handshake.

OCSP Stapling can be explicitly enabled or disabled in the NATS Server by setting the following flag in the NATS configuration file at the top-level:

ocsp: false

Note: When OCSP Stapling is disabled, the NATS Server will not request staples even if the certificate has the Must-Staple flag.

Advanced Configuration

By default, the NATS Server will be running in OCSP auto mode. In this mode the server will only fetch staples when the Must-Staple flag is configured in the certificate.

There are other OCSP modes that control the behavior as to whether OCSP should be enforced and the server should shutdown if the certificate runs with a revoked staple:

For example, in the following OCSP configuration, the mode is set to must. This means that staples will be fetched only for certificates that have the Must-Staple flag enabled as well, but in case of revocation the server will shutdown rather than run with a revoked staple.
In this configuration, the url will also override the OCSP responder URL that may have been configured in the certificate.

ocsp {
  mode: must
  url: "http://ocsp.example.net"
}

If staples are always required, regardless of the configuration of the certificate, you can enforce the behavior as follows:

ocsp {
  mode: always
  url: "http://ocsp.example.net"
}

Caching of Staples

When a store_dir is configured in the NATS Server, the directory will be used to cache staples on disk to allow the server to resume in case of restarts without having to make another request to the OCSP responder if the staple is still valid.

ocsp: true

store_dir: "/path/to/store/dir"

tls {
    cert_file: "configs/certs/ocsp/server-status-request-url.pem"
    key_file: "configs/certs/ocsp/server-status-request-url-key.pem"
    ca_file: "configs/certs/ocsp/ca-cert.pem"
    timeout: 5
}

If JetStream is enabled, then the same store_dir will be reused and disk caching will be automatically enabled.

Auth Callout

As of NATS v2.10.0

Auth Callout is an opt-in extension for delegating client authentication and authorization to an application-defined NATS service.

The motivation for this extension is to support applications using an alternate identity and access management (IAM) backend as the source of truth for managing users/applications/machines credentials and permissions. This could be services that implement standard protocols such as LDAP, SAML, and OAuth, an ad-hoc database, or even a file on disk.

Both centralized and decentralized authentication models are supported with slightly different considerations and semantics.

There are three phases to leveraging auth callout:

• service implementation
• migration considerations
• setup and configuration

提示

Note, the setup and configuration is deliberately last since enabling the configuration before deploying a service could cause issues for existing systems.

Centralized Auth

Centralized auth refers to all authentication and authorization mechanisms that are server config file-based.

Service implementation

Refer to the end-to-end example to get oriented with a basic service implementation.

There are three key data structures:

• authorization request claims
• authorization response claims
• user claims

提示

Language support for these structures currently exists for Go in the nats-io/jwt package.

Migration considerations

In this context, migration refers to the considerations and steps required to enable auth callout for an existing system without causing interruption.

In the centralized model, existing users defined in the config file will be ignored. The auth service will need to handle authenticating all users as well as assigning the target account and permissions. This includes the system account user(s) and an implicit "no auth" user.

As a result, prior to enabling auth callout, existing users and permissions must be ported to the target backend. Once the service is deployed, the auth_callout configuration can be enabled at which point client authentication will be delegated to the auth service. Assuming the credentials are the same, clients should not experience interruption on reconnect.

Setup and configuration

For centralized auth callout, configuration is declared in the auth_callout block under the top-level authorization block.

authorization {
  auth_callout {
    ...
  }
}

The available properties in the auth_callout block include:

To generate the account issuer NKey, the nsc tool can be used.

$ nsc generate nkey --account
SAANDLKMXL6CUS3CP52WIXBEDN6YJ545GDKC65U5JZPPV6WH6ESWUA6YAI
ABJHLOVMPA4CI6R5KLNGOB4GSLNIY7IOUPAJC4YFNDLQVIOBYQGUWVLA

提示

☝️ Be sure to generate your own keypair! Don't use this in production.

authorization {
  users: [ { user: auth, password: auth } ]
  auth_callout {
    issuer: ABJHLOVMPA4CI6R5KLNGOB4GSLNIY7IOUPAJC4YFNDLQVIOBYQGUWVLA
    auth_users: [ auth ]
  }
}

This minimum configuration would use the implicit default account $G.

Multiple accounts

If an existing system using multiple accounts is being migrated to auth callout, then the existing accounts configuration should remain with the users property removed (since it will no longer be used after being ported).

For new setups, it is recommended to use explicit accounts, such as the following configuration having the AUTH account for auth callout, APP (could be more) for application account (instead of relying on the $G account), and SYS for the system account.

accounts {
  AUTH: {
    users: [ { user: auth, password: auth } ]
  }
  APP: {}
  SYS: {}
}
system_account: SYS

authorization {
  auth_callout {
    issuer: ABJHLOVMPA4CI6R5KLNGOB4GSLNIY7IOUPAJC4YFNDLQVIOBYQGUWVLA
    auth_users: [ auth ]
    account: AUTH
  }
}

Encryption

The xkey property enables encrypting the request payloads. This is recommended as a security best practice, but not required.

To generate an XKey, nsc can be used again.

$ nsc generate nkey --curve
SXANPB47UINQR7EXT3BRP26A4LY2CMCDLTY2KX6BU3EGK2VZYREJ4IJRCE
XAMHJVPKHHPYZQQM2IVWXKJH36KDDZZMSJ32QKSQBUODFX4I4HARO4GL

提示

☝️ Again, don't use this and be sure to generate your own and keep the seed secret!

Incorporating the xkey, we have the following config:

accounts {
  AUTH: {
    users: [ { user: auth, password: auth } ]
  }
  APP: {}
  SYS: {}
}
system_account: SYS

authorization {
  auth_callout {
    issuer: ABJHLOVMPA4CI6R5KLNGOB4GSLNIY7IOUPAJC4YFNDLQVIOBYQGUWVLA
    auth_users: [ auth ]
    account: AUTH
    xkey: XAMHJVPKHHPYZQQM2IVWXKJH36KDDZZMSJ32QKSQBUODFX4I4HARO4GL
  }
}

Decentralized Auth

Coming soon!

Reference

Encryption

When encryption is enabled, the server will generate a one-time use XKey keypair per client connection/reconnect. The public key is included in the authorization request claims which enables the auth callout service to encrypt the authorization response payload when sending it back to the NATS server.

提示

The one-time use keypair prevents replay attacks since the public key will be thrown away after the first response was received by the server or the timeout was reached.

Once the authorization request is prepared, it is encoded and encrypted using the configured xkey public key. Once encrypted, the message is published for the auth service to receive.

The auth service is expected to have the private key to decrypt the authorization request before using the claims data. When preparing the response, the server-provided one-time public xkey will be used to encrypt the response before sending back to the server.

Schema

Authorization request claims

The claims is a standard JWT structure with a nested object named nats containing the following top-level fields:

• server_id - An object describing the NATS server, include the id field needed to be used in the authorization response.
• user_nkey - A user public NKey generated by the NATS server which is used as the subject of the authorization response.
• client_info - An object describing the client attempting to connect.
• connect_opts - An object containing the data sent by client in the CONNECT message.
• client_tls - An object containing any client certificates, if applicable.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "authorization-request-claims",
  "properties": {
    "aud": {
      "type": "string"
    },
    "exp": {
      "type": "integer"
    },
    "jti": {
      "type": "string"
    },
    "iat": {
      "type": "integer"
    },
    "iss": {
      "type": "string"
    },
    "name": {
      "type": "string"
    },
    "nbf": {
      "type": "integer"
    },
    "sub": {
      "type": "string"
    },
    "nats": {
      "properties": {
        "server_id": {
          "properties": {
            "name": {
              "type": "string"
            },
            "host": {
              "type": "string"
            },
            "id": {
              "type": "string"
            },
            "version": {
              "type": "string"
            },
            "cluster": {
              "type": "string"
            },
            "tags": {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            "xkey": {
              "type": "string"
            }
          },
          "additionalProperties": false,
          "type": "object",
          "required": [
            "name",
            "host",
            "id"
          ]
        },
        "user_nkey": {
          "type": "string"
        },
        "client_info": {
          "properties": {
            "host": {
              "type": "string"
            },
            "id": {
              "type": "integer"
            },
            "user": {
              "type": "string"
            },
            "name": {
              "type": "string"
            },
            "tags": {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            "name_tag": {
              "type": "string"
            },
            "kind": {
              "type": "string"
            },
            "type": {
              "type": "string"
            },
            "mqtt_id": {
              "type": "string"
            },
            "nonce": {
              "type": "string"
            }
          },
          "additionalProperties": false,
          "type": "object"
        },
        "connect_opts": {
          "properties": {
            "jwt": {
              "type": "string"
            },
            "nkey": {
              "type": "string"
            },
            "sig": {
              "type": "string"
            },
            "auth_token": {
              "type": "string"
            },
            "user": {
              "type": "string"
            },
            "pass": {
              "type": "string"
            },
            "name": {
              "type": "string"
            },
            "lang": {
              "type": "string"
            },
            "version": {
              "type": "string"
            },
            "protocol": {
              "type": "integer"
            }
          },
          "additionalProperties": false,
          "type": "object",
          "required": [
            "protocol"
          ]
        },
        "client_tls": {
          "properties": {
            "version": {
              "type": "string"
            },
            "cipher": {
              "type": "string"
            },
            "certs": {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            "verified_chains": {
              "items": {
                "items": {
                  "type": "string"
                },
                "type": "array"
              },
              "type": "array"
            }
          },
          "additionalProperties": false,
          "type": "object"
        },
        "request_nonce": {
          "type": "string"
        },
        "tags": {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "type": {
          "type": "string"
        },
        "version": {
          "type": "integer"
        }
      },
      "additionalProperties": false,
      "type": "object",
      "required": [
        "server_id",
        "user_nkey",
        "client_info",
        "connect_opts"
      ]
    }
  },
  "additionalProperties": false,
  "type": "object",
  "required": [
    "nats"
  ]
}

Authorization response claims

The claims is a standard JWT structure with a nested object named nats containing the following top-level fields:

• jwt - The encoded user claims JWT which will be used by the NATS server for the duration of the client connection.
• error - An error message sent back to the NATS server if authorization failed. This will be included log output.
• issuer_account - The public Nkey of the issuing account. If set, this indicates the claim was issued by a signing key.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://github.com/nats-io/jwt/v2/authorization-response-claims",
  "properties": {
    "aud": {
      "type": "string"
    },
    "exp": {
      "type": "integer"
    },
    "jti": {
      "type": "string"
    },
    "iat": {
      "type": "integer"
    },
    "iss": {
      "type": "string"
    },
    "name": {
      "type": "string"
    },
    "nbf": {
      "type": "integer"
    },
    "sub": {
      "type": "string"
    },
    "nats": {
      "properties": {
        "jwt": {
          "type": "string"
        },
        "error": {
          "type": "string"
        },
        "issuer_account": {
          "type": "string"
        },
        "tags": {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "type": {
          "type": "string"
        },
        "version": {
          "type": "integer"
        }
      },
      "additionalProperties": false,
      "type": "object"
    }
  },
  "additionalProperties": false,
  "type": "object",
  "required": [
    "nats"
  ]
}

User claims

The claims is a standard JWT structure with a nested object named nats containing the following, notable, top-level fields:

• issuer_account - The public Nkey of the issuing account. If set, this indicates the claim was issued by a signing key.

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "$id": "https://github.com/nats-io/jwt/v2/user-claims",
  "properties": {
    "aud": {
      "type": "string"
    },
    "exp": {
      "type": "integer"
    },
    "jti": {
      "type": "string"
    },
    "iat": {
      "type": "integer"
    },
    "iss": {
      "type": "string"
    },
    "name": {
      "type": "string"
    },
    "nbf": {
      "type": "integer"
    },
    "sub": {
      "type": "string"
    },
    "nats": {
      "properties": {
        "pub": {
          "properties": {
            "allow": {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            "deny": {
              "items": {
                "type": "string"
              },
              "type": "array"
            }
          },
          "additionalProperties": false,
          "type": "object"
        },
        "sub": {
          "properties": {
            "allow": {
              "items": {
                "type": "string"
              },
              "type": "array"
            },
            "deny": {
              "items": {
                "type": "string"
              },
              "type": "array"
            }
          },
          "additionalProperties": false,
          "type": "object"
        },
        "resp": {
          "properties": {
            "max": {
              "type": "integer"
            },
            "ttl": {
              "type": "integer"
            }
          },
          "additionalProperties": false,
          "type": "object",
          "required": [
            "max",
            "ttl"
          ]
        },
        "src": {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "times": {
          "items": {
            "properties": {
              "start": {
                "type": "string"
              },
              "end": {
                "type": "string"
              }
            },
            "additionalProperties": false,
            "type": "object"
          },
          "type": "array"
        },
        "times_location": {
          "type": "string"
        },
        "subs": {
          "type": "integer"
        },
        "data": {
          "type": "integer"
        },
        "payload": {
          "type": "integer"
        },
        "bearer_token": {
          "type": "boolean"
        },
        "allowed_connection_types": {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "issuer_account": {
          "type": "string"
        },
        "tags": {
          "items": {
            "type": "string"
          },
          "type": "array"
        },
        "type": {
          "type": "string"
        },
        "version": {
          "type": "integer"
        }
      },
      "additionalProperties": false,
      "type": "object"
    }
  },
  "additionalProperties": false,
  "type": "object"
}

Logging

Configuring Logging

The NATS server provides various logging options that you can set via the command line or the configuration file.

Command Line Options

The following logging operations are supported:

-l, --log FILE                   File to redirect log output.
-T, --logtime                    Timestamp log entries (default is true).
-s, --syslog                     Enable syslog as log method.
-r, --remote_syslog              Syslog server address.
-D, --debug                      Enable debugging output.
-V, --trace                      Trace the raw protocol.
-VV                              Verbose trace (traces system account as well)
-DV                              Debug and Trace.
-DVV                             Debug and verbose trace (traces system account as well)

Debug and trace

The -DV flag enables trace and debug for the server.

nats-server -DV -m 8222 -user foo -pass bar

Log file redirect

nats-server -DV -m 8222 -l nats.log

Timestamp

If -T false then log entries are not timestamped. Default is true.

Syslog

You can configure syslog with UDP:

nats-server -r udp://localhost:514

or syslog:

nats-server -r syslog://<hostname>:<port>

For example:

syslog://logs.papertrailapp.com:26900

Using the Configuration File

All of these settings are available in the configuration file as well.

debug:   false
trace:   true
logtime: false
logfile_size_limit: 1GB
logfile_max_num: 100
log_file: "/tmp/nats-server.log"

Log Rotation

Introduced in NATS Server v2.1.4, NATS allows for auto-rotation of log files when the size is greater than the configured limit set in logfile_size_limit. The backup files will have the same name as the original log file with the suffix .yyyy.mm.dd.hh.mm.ss.micros.

You can also use NATS-included mechanisms with logrotate, a simple standard Linux utility to rotate logs available on most distributions like Debian, Ubuntu, RedHat (CentOS), etc., to make log rotation simple.

For example, you could configure logrotate with:

/path/to/nats-server.log {
    daily
    rotate 30
    compress
    missingok
    notifempty
    postrotate
        kill -SIGUSR1 `cat /var/run/nats-server.pid`
    endscript
}

The first line specifies the location that the subsequent lines will apply to.

The rest of the file specifies that the logs will rotate daily ("daily" option) and that 30 older copies will be preserved ("rotate" option). Other options are described in logrorate documentation.

The "postrotate" section tells NATS server to reload the log files once the rotation is complete. The command `kill -SIGUSR1cat /var/run/nats-server.pid``` does not kill the NATS server process, but instead sends it a signal causing it to reload its log files. This will cause new requests to be logged to the refreshed log file.

The /var/run/nats-server.pid file is where NATS server stores the master process's pid.

Some Logging Notes

• The NATS server, in verbose mode, will log the receipt of UNSUB messages, but this does not indicate the subscription is gone, only that the message was received. The DELSUB message in the log can be used to determine when the actual subscription removal has taken place.

Enabling Monitoring

NATS Server Monitoring

To monitor the NATS messaging system, nats-server provides a lightweight HTTP server on a dedicated monitoring port. The monitoring server provides several endpoints, providing statistics and other information.

The NATS monitoring endpoints support JSONP and CORS, making it easy to create single page monitoring web applications.

Warning: nats-server does not have authentication/authorization for the monitoring endpoint. When you plan to open your nats-server to the internet make sure to not expose the monitoring port as well. By default monitoring binds to every interface 0.0.0.0 so consider setting monitoring to localhost or have appropriate firewall rules.

Enabling monitoring from the command line

To enable the monitoring server, start the NATS server with the monitoring flag -m and the monitoring port, or turn it on in the configuration file.

-m, --http_port PORT             HTTP PORT for monitoring
-ms,--https_port PORT            Use HTTPS PORT for monitoring

Example:

nats-server -m 8222

[4528] 2019/06/01 20:09:58.572939 [INF] Starting nats-server version 2.0.0
[4528] 2019/06/01 20:09:58.573007 [INF] Starting http monitor on port 8222
[4528] 2019/06/01 20:09:58.573071 [INF] Listening for client connections on 0.0.0.0:4222
[4528] 2019/06/01 20:09:58.573090 [INF] nats-server is ready

To test, run nats-server -m 8222, then go to http://localhost:8222/

Enable monitoring from the configuration file

You can also enable monitoring using the configuration file as follows:

http_port: 8222

Binding to localhost as well:

http: localhost:8222

For example, to monitor this server locally, the endpoint would be http://localhost:8222/varz. It reports various general statistics.

Monitoring Tools

In addition to writing custom monitoring tools, you can monitor nats-server in Prometheus. The Prometheus NATS Exporter allows you to configure the metrics you want to observe and store in Prometheus. There's a sample Grafana dashboard that you can use to visualize the server metrics.

MQTT

Supported since NATS Server version 2.2

NATS follows as closely as possible to the MQTT v3.1.1 specification. Refer to the MQTT implementation overview in the nats-server repo.

NATS supports MQTT QoS: 0, 1 and 2

When to Use MQTT

MQTT support in NATS is intended to be an enabling technology allowing users to leverage existing investments in their IoT deployments. Updating software on the edge or endpoints can be onerous and risky, especially when embedded applications are involved.

In greenfield IoT deployments, when possible, we prefer NATS extended out to endpoints and devices for a few reasons. There are significant advantages with security and observability when using a single technology end to end. Compared to MQTT, NATS is nearly as lightweight in terms of protocol bandwidth and maintainer supported clients efficiently utilize resources so we consider NATS to be a good choice to use end to end, including use on resource constrained devices.

In existing MQTT deployments or in situations when endpoints can only support MQTT, using a NATS server as a drop-in MQTT server replacement to securely connect to a remote NATS cluster or supercluster is compelling. You can keep your existing IoT investment and use NATS for secure, resilient, and scalable access to your streams and services.

JetStream Requirements

For an MQTT client to connect to the NATS server, the user's account must be JetStream enabled. This is because persistence is needed for the sessions and retained messages since even retained messages of QoS 0 are persisted.

MQTT Topics and NATS Subjects

MQTT Topics are similar to NATS Subjects, but have distinctive differences.

MQTT topic uses "/" as a level separator. For instance foo/bar would translate to NATS subject foo.bar. But in MQTT, /foo/bar/ is a valid subject, which, if simply translated, would become .foo.bar., which is NOT a valid NATS Subject.

NATS Server will convert an MQTT topic following those rules:

Note: Prior to NATS Server v2.10.0, the character . was not supported. At version v2.10.0 and above, the character . will be translated to //.

As indicated above, if an MQTT topic contains the character (or . prior to v2.10.0), NATS will reject it, causing the connection to be closed for published messages, and returning a failure code in the SUBACK packet for a subscriptions.

MQTT Wildcards

As in NATS, MQTT wildcards represent either multi or single levels. As in NATS, they are allowed only for subscriptions, not for published messages.

The wildcard # matches any number of levels within a topic, which means that a subscription on foo/# would receive messages on foo/bar, or foo/bar/baz, but also on foo. This is not the case in NATS where a subscription on foo.> can receive messages on foo/bar or foo/bar/baz, but not on foo. To solve this, NATS Server will create two subscriptions, one on foo.> and one on foo. If the MQTT subscription is simply on #, then a single NATS subscription on > is enough.

The wildcard + matches a single level, which means foo/+ can receive message on foo/bar or foo/baz, but not on foo/bar/baz nor foo. This is the same with NATS subscriptions using the wildcard *. Therefore foo/+ would translate to foo.*.

NATS Wilcards in MQTT Topics

MQTT treats * and > as ordinary literal characters. The NATS Server does not escape these characters during MQTT topic to NATS subject conversion. If either character becomes a complete NATS subject token after / is converted to ., NATS interprets it using normal NATS wildcard rules.

MQTT topics and NATS permissions

MQTT topics are converted to NATS subjects before authorization checks are applied. Permission entries for MQTT users should therefore be written as NATS literal or wildcard subjects.

Communication Between MQTT and NATS

When an MQTT client creates a subscription on a topic, the NATS server creates the similar NATS subscription (with conversion from MQTT topic to NATS subject) so that the interest is registered in the cluster and known to any NATS publishers.

That is, say an MQTT client connects to server "A" and creates a subscription of foo/bar, server "A" creates a subscription on foo.bar, which interest is propagated as any other NATS subscription. A publisher connecting anywhere in the cluster and publishing on foo.bar would cause server "A" to deliver a QoS 0 message to the MQTT subscription.

This works the same way for MQTT publishers. When the server receives an MQTT publish message, it is converted to the NATS subject and published, which means that any matching NATS subscription will receive the MQTT message.

If the MQTT subscription is QoS1 and an MQTT publisher publishes an MQTT QoS1 message on the same or any other server in the cluster, the message will be persisted in the cluster and routed and delivered as QoS 1 to the MQTT subscription.

QoS 1 and 2 Redeliveries

When the server delivers a QoS 1 or 2 message to a QoS 1 or 2 subscription, it will keep the message until it receives the PUBACK for the corresponding packet identifier. If it does not receive it within the "ack_wait" interval, that message will be resent.

Max Ack Pending

This is the amount of QoS 1 or 2 messages the server can send to a subscription without receiving any PUBACK for those messages. The maximum value is 65535.

The total of subscriptions' max_ack_pending on a given session cannot exceed 65535. Attempting to create a subscription that would bring the total above the limit would result in the server returning a failure code in the SUBACK for this subscription.

Due to how the NATS server handles the MQTT "#" wildcard, each subscription ending with "#" will use 2 times the max_ack_pending value.

Sessions

NATS Server will persist all sessions, even if they are created with the "clean session" flag, meaning that sessions only last for the duration of the network connection between the client and the server.

A session is identified by a client identifier. If two connections try to use the same client identifier, the server, per specification, will close the existing connection and accept the new one.

If the user incorrectly starts two applications that use the same client identifier, this would result in a very quick flapping if the MQTT client has a reconnect feature and quickly reconnects.

To prevent this, the NATS server will accept the new session and will delay the closing of the old connection to reduce the flapping rate.

Detection of the concurrent use of sessions also works in cluster mode.

Retained Messages

When a server receives an MQTT publish packet with the RETAIN flag set (regardless of its QoS), it stores the application message and its QoS, so that it can be delivered to future subscribers whose subscriptions match its topic name.

When a new subscription is established, the last retained message, if any, on each matching topic name will be sent to the subscriber.

A PUBLISH Packet with a RETAIN flag set to 1 and a payload containing zero bytes will be processed as normal and sent to clients with a subscription matching the topic name. Additionally any existing retained message with the same topic name will be removed and any future subscribers for the topic will not receive a retained message.

Clustering

NATS supports MQTT in a NATS cluster. The replication factor is automatically set based on the size of the cluster.

Connections with Same Client ID

If a client is connected to a server "A" in the cluster and another client connects to a server "B" and uses the same client identifier, server "A" will close its client connection upon discovering the use of an active client identifier.

Users should avoid this situation as this is not as easy and immediate as if the two applications are connected to the same server.

There may be cases where the server will reject the new connection if there is no safe way to close the existing connection, such as when it is in the middle of processing some MQTT packets.

Retained Messages

Retained messages are stored in the cluster and available to any server in the cluster. However, this is not immediate and if a producer connects to a server and produces a retained message and another connection connects to another server and starts a matching subscription, it may not receive the retained message if the server it connects to has not yet been made aware of this retained message.

In other words, retained messages in clustering mode is best-effort, and applications that rely on the presence of a retained message should connect on the server that produced them.

Limitations

• NATS messages published to MQTT subscriptions are always delivered as QoS 0 messages.
• MQTT published messages on topic names containing "````" or ".\" characters will cause the connection to be closed. Presence of those characters in MQTT subscriptions will result in error code in the SUBACK packet.
• MQTT wildcard # may cause the NATS server to create two subscriptions.
• MQTT concurrent sessions may result in the new connection to be evicted instead of the existing one.
• MQTT retained messages in clustering mode is best effort.

See Also

Replace your MQTT Broker with NATS Server

Configuration

To enable MQTT support in the server, add an mqtt configuration block in the server's configuration file like the following:

mqtt {
    # Specify a host and port to listen for websocket connections
    #
    # listen: "host:port"
    # It can also be configured with individual parameters,
    # namely host and port.
    #
    # host: "hostname"
    port: 1883

    # TLS configuration.
    #
    tls {
        cert_file: "/path/to/cert.pem"
        key_file: "/path/to/key.pem"

        # Root CA file
        #
        # ca_file: "/path/to/ca.pem"

        # If true, require and verify client certificates.
        #
        # verify: true

        # TLS handshake timeout in fractional seconds.
        #
        # timeout: 2.0

        # If true, require and verify client certificates and map certificate
        # values for authentication purposes.
        #
        # verify_and_map: true
    }

    # If no user name is provided when an MQTT client connects, will default
    # this user name in the authentication phase. If specified, this will
    # override, for MQTT clients, any `no_auth_user` value defined in the
    # main configuration file.
    # Note that this is not compatible with running the server in operator mode.
    #
    # no_auth_user: "my_username_for_apps_not_providing_credentials"

    # See below to know what is the normal way of limiting MQTT clients
    # to specific users.
    # If there are no users specified in the configuration, this simple authorization
    # block allows you to override the values that would be configured in the
    # equivalent block in the main section.
    #
    # authorization {
    #     # If this is specified, the client has to provide the same username
    #     # and password to be able to connect.
    #     # username: "my_user_name"
    #     # password: "my_password"
    #
    #     # If this is specified, the password field in the CONNECT packet has to
    #     # match this token.
    #     # token: "my_token"
    #
    #     # This overrides the main's authorization timeout. For consistency
    #     # with the main's authorization configuration block, this is expressed
    #     # as a number of seconds.
    #     # timeout: 2.0
    #}

    # This is the amount of time after which a QoS 1 message sent to
    # a client is redelivered as a DUPLICATE if the server has not
    # received the PUBACK packet on the original Packet Identifier.
    # The value has to be positive.
    # Zero will cause the server to use the default value (30 seconds).
    # Note that changes to this option is applied only to new MQTT subscriptions.
    #
    # Expressed as a time duration, with "s", "m", "h" indicating seconds,
    # minutes and hours respectively. For instance "10s" for 10 seconds,
    # "1m" for 1 minute, etc...
    #
    # ack_wait: "1m"

    # This is the amount of QoS 1 messages the server can send to
    # a subscription without receiving any PUBACK for those messages.
    # The valid range is [0..65535].
    #
    # The total of subscriptions' max_ack_pending on a given session cannot
    # exceed 65535. Attempting to create a subscription that would bring
    # the total above the limit would result in the server returning 0x80
    # in the SUBACK for this subscription.
    # Due to how the NATS Server handles the MQTT "#" wildcard, each
    # subscription ending with "#" will use 2 times the max_ack_pending value.
    # Note that changes to this option is applied only to new subscriptions.
    #
    # max_ack_pending: 100
}

Server name

MQTT requires a server name to be set. Server names must be unique in a cluster or super-cluster topology. The server name is set in the top-level section of the server configuration. Here is an example:

server_name: "my_mqtt_server"
mqtt {
    port: 1883
    ...
}

Authentication/Authorization of MQTT Users

Operator mode

In operator mode, all users need to provide a JWT in order to connect. In the standard authentication procedure of this mode, NATS clients are required to sign a nonce sent by the server using their private key (see JWTs and Privacy). MQTT clients cannot do that, therefore, the JWT is used for authentication, removing the need of the seed. It means that you need to pass the JWT token as the MQTT password and use any username (except empty, since MQTT protocol requires a username to be set if a password is set). The JWT has to have the Bearer boolean set to true, which can be done with nsc:

nsc edit user --name U --account A --bearer

Local mode

A new field when configuring users allows you to restrict which type of connections are allowed for a specific user.

Consider this configuration:

authorization {
  users [
    {user: foo password: foopwd, permission: {...}}
    {user: bar password: barpwd, permission: {...}}
  ]
}

If an MQTT client were to connect and use the username foo and password foopwd, it would be accepted. Now suppose that you would want an MQTT client to only be accepted if it connected using the username bar and password barpwd, then you would use the option allowed_connection_types to restrict which type of connections can bind to this user.

authorization {
  users [
    {user: foo password: foopwd, permission: {...}}
    {user: bar password: barpwd, permission: {...}, allowed_connection_types: ["MQTT"]}
  ]
}

The option allowed_connection_types (also can be named connection_types or clients) as you can see is a list, and you can allow several types of clients. Suppose you want the user bar to accept both standard NATS clients and MQTT clients, you would configure the user like this:

authorization {
  users [
    {user: foo password: foopwd, permission: {...}}
    {user: bar password: barpwd, permission: {...}, allowed_connection_types: ["STANDARD", "MQTT"]}
  ]
}

The absence of allowed_connection_types means that all types of connections are allowed (the default behavior).

The possible values are currently:

• STANDARD
• WEBSOCKET
• LEAFNODE
• MQTT

Special permissions

When an MQTT client creates a QoS 1 subscription, this translates to the creation of a JetStream durable subscription. To receive messages for this durable, the NATS Server creates a subscription with a subject such as $MQTT.sub. and sets it as the JetStream durable's delivery subject.

Therefore, if you have set some permissions for the MQTT user, you need to allow subscribe permissions on $MQTT.sub.>.

Here is an example of a basic configuration that sets some permissions to a user named "mqtt". As you can see, the subscribe permission $MQTT.sub.> is added to allow this client to create QoS 1 subscriptions.

    listen: 127.0.0.1:4222
    jetstream: enabled
    authorization {
        mqtt_perms = {
            publish = ["baz"]
            subscribe = ["foo", "bar", "$MQTT.sub.>"]
        }
        users = [
            {user: mqtt, password: pass, permissions: $mqtt_perms, allowed_connection_types: ["MQTT"]}
        ]
    }
    mqtt {
        listen: 127.0.0.1:1883
    }

Configuring Subject Mapping

Supported since NATS Server version 2.2

Subject mapping is a very powerful feature of the NATS server, useful for canary deployments, A/B testing, chaos testing, and migrating to a new subject namespace.

Configuring subject mapping

Subject mappings are defined and applied at the account level. If you are using static account security you will need to edit the server configuration file, however if you are using JWT Security (Operator Mode), then you need to use nsc or customer tools to edit and push changes to you account.

NOTE: You can also use subject mapping as part of defining imports and exports between accounts

Static authentication

In any of the static authentication modes the mappings are defined in the server configuration file, any changes to mappings in the configuration file will take effect as soon as a reload signal is sent to the server process (e.g. use nats-server --signal reload).

The mappings stanza can occur at the top level to apply to the global account or be scoped within a specific account.

mappings = {

  # Simple direct mapping.  Messages published to foo are mapped to bar.
  foo: bar

  # remapping tokens can be done with $<N> representing token position.
  # In this example bar.a.b would be mapped to baz.b.a.
  bar.*.*: baz.$2.$1

  # You can scope mappings to a particular cluster
  foo.cluster.scoped : [
    { destination: bar.cluster.scoped, weight:100%, cluster: us-west-1 }
  ]

  # Use weighted mapping for canary testing or A/B testing.  Change dynamically
  # at any time with a server reload.
  myservice.request: [
    { destination: myservice.request.v1, weight: 90% },
    { destination: myservice.request.v2, weight: 10% }
  ]

  # A testing example of wildcard mapping balanced across two subjects.
  # 20% of the traffic is mapped to a service in QA coded to fail.
  myservice.test.*: [
    { destination: myservice.test.$1, weight: 80% },
    { destination: myservice.test.fail.$1, weight: 20% }
  ]

  # A chaos testing trick that introduces 50% artificial message loss of
  # messages published to foo.loss
  foo.loss.>: [ { destination: foo.loss.>, weight: 50% } ]
}

JWT authentication

When using the JWT authentication mode, the mappings are defined in the account's JWT. Account JWTs can be created or modified either through the_ JWT API_ or using the nsc CLI too. For more detailed information see nsc add mapping --help, nsc delete mapping --help. Subject mapping changes take effect as soon as the modified account JWT is pushed to the nats servers (i.e. nsc push).

Examples of using nsc to manage mappings:

• Add a new mapping: nsc add mapping --from "a" --to "b"
• Modify an entry, say to set a weight after the fact: nsc add mapping --from "a" --to "b" --weight 50
• Add two entries from one subject, set weights and execute multiple times: nsc add mapping --from "a" --to "c" --weight 50
• Delete a mapping: nsc delete mapping --from "a"

Simple Mapping

The example of foo:bar is straightforward. All messages the server receives on subject foo are remapped and can be received by clients subscribed to bar.

Subject Token Reordering

Wildcard tokens may be referenced via $<position>. For example, the first wildcard token is $1, the second is $2, etc. Referencing these tokens can allow for reordering.

With this mapping:

  bar.*.*: baz.$2.$1

Messages that were originally published to bar.a.b are remapped in the server to baz.b.a. Messages arriving at the server on bar.one.two would be mapped to baz.two.one, and so forth.

Weighted Mappings for A/B Testing or Canary Releases

Traffic can be split by percentage from one subject to multiple subjects. Here's an example for canary deployments, starting with version 1 of your service.

Applications would make requests of a service at myservice.requests. The responders doing the work of the server would subscribe to myservice.requests.v1. Your configuration would look like this:

  myservice.requests: [
    { destination: myservice.requests.v1, weight: 100% }
  ]

All requests to myservice.requests will go to version 1 of your service.

When version 2 comes along, you'll want to test it with a canary deployment. Version 2 would subscribe to myservice.requests.v2. Launch instances of your service (don't forget about queue subscribers and load balancing).

Update the configuration file to redirect some portion of the requests made to myservice.requests to version 2 of your service. In this case we'll use 2%.

  myservice.requests: [
    { destination: myservice.requests.v1, weight: 98% },
    { destination: myservice.requests.v2, weight: 2% }
  ]

You can reload the server at this point to make the changes with zero downtime. After reloading, 2% of your requests will be serviced by the new version.

Once you've determined Version 2 stable switch 100% of the traffic over and reload the server with a new configuration.

  myservice.requests: [
    { destination: myservice.requests.v2, weight: 100% }
  ]

Now shutdown the version 1 instances of your service.

Traffic Shaping in Testing

Traffic shaping is useful in testing. You might have a service that runs in QA that simulates failure scenarios which could receive 20% of the traffic to test the service requestor.

  myservice.requests.*: [
    { destination: myservice.requests.$1, weight: 80% },
    { destination: myservice.requests.fail.$1, weight: 20% }
  ]

Artificial Loss

Alternatively, introduce loss into your system for chaos testing by mapping a percentage of traffic to the same subject. In this drastic example, 50% of the traffic published to foo.loss.a would be artificially dropped by the server.

  foo.loss.>: [ { destination: foo.loss.>, weight: 50% } ]

You can both split and introduce loss for testing. Here, 90% of requests would go to your service, 8% would go to a service simulating failure conditions, and the unaccounted for 2% would simulate message loss.

  myservice.requests: [
    { destination: myservice.requests.v3, weight: 90% },
    { destination: myservice.requests.v3.fail, weight: 8% }
    # the remaining 2% is "lost"
  ]

System Events

NATS servers leverage Accounts support and generate events such as:

• account connect/disconnect
• authentication errors
• server shutdown
• server stat summary

In addition the server supports a limited number of requests that can be used to query for account connections, server stat summaries, and pinging servers in the cluster.

These events are enabled by configuring system_account and subscribing/requesting using a system account user.

Accounts are used so that subscriptions from your applications, say >, do not receive system events and vice versa. Using accounts requires either:

• Configuring authentication locally and listing one of the accounts in system_account
• Or by using decentralized authentication and authorization via jwt as shown in this Tutorial. In this case system_account contains the account public key.

N.B. The default global account $G does not publish advisories.

Available Events and Services

System Account

The system account publishes messages under well known subject patterns.

Server initiated events:

• $SYS.ACCOUNT.<id>.CONNECT (client connects)
• $SYS.ACCOUNT.<id>.DISCONNECT (client disconnects)
• $SYS.ACCOUNT.<id>.SERVER.CONNS (connections for an account changed)
• $SYS.SERVER.<id>.CLIENT.AUTH.ERR (authentication error)
• $SYS.SERVER.<id>.STATSZ (stats summary)

In addition other tools with system account privileges, can initiate requests (Examples can be found here):

• $SYS.REQ.SERVER.<id>.STATSZ (request server stat summary)
• $SYS.REQ.SERVER.PING (discover servers - will return multiple messages)

Monitoring endpoints as listed in the table below are accessible as system services using the following subject pattern:

• $SYS.REQ.SERVER.<id>.<endpoint-name> (request server monitoring endpoint corresponding to endpoint name.)
• $SYS.REQ.SERVER.PING.<endpoint-name> (from all server, request server monitoring endpoint corresponding to endpoint name - will return multiple messages)

• "$SYS.REQ.ACCOUNT.<account-id>.<endpoint-name>(from all server, request account specific monitoring endpoint corresponding to account id and endpoint name - will return multiple messages)

Servers like nats-account-server publish system account messages when a claim is updated, the nats-server listens for them, and updates its account information accordingly:

• $SYS.ACCOUNT.<id>.CLAIMS.UPDATE

With these few messages you can build useful monitoring tools:

• health/load of your servers
• client connects/disconnects
• account connections
• authentication errors

Local Configuration

To make use of System events, just using accounts, your configuration can look like this:

accounts: {
    USERS: {
        users: [
            {user: a, password: a}
        ]
    },
    SYS: { 
        users: [
            {user: admin, password: changeit}
           ]
    },
}
system_account: SYS

Please note that applications now have to authenticate such that a connection can be associated with an account. In this example username and password were chosen for simplicity of the demonstration. Subscribe to all system events like this nats sub -s nats://admin:changeit@localhost:4222 ">" and observe what happens when you do something like nats pub -s "nats://a:a@localhost:4222" foo bar. Examples on how to use system services can be found here.

System Events & Decentralized JWT Tutorial

Enabling System Events with Decentralized Authentication/Authorization

To enable and access system events, you'll have to:

• Create an Operator, Account and User
• Run a NATS Account Server (or Memory Resolver)

Create an Operator, Account, User

Let's create an operator, system account and system account user:

nsc add operator -n SAOP

Generated operator key - private key stored "~/.nkeys/SAOP/SAOP.nk"
Success! - added operator "SAOP"

Add the system account

nsc add account -n SYS

Generated account key - private key stored "~/.nkeys/SAOP/accounts/SYS/SYS.nk"
Success! - added account "SYS"

Add a system account user

nsc add user -n SYSU

Generated user key - private key stored "~/.nkeys/SAOP/accounts/SYS/users/SYSU.nk"
Generated user creds file "~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds"
Success! - added user "SYSU" to "SYS"

By default, the operator JWT can be found in ~/.nsc/nats/<operator_name>/<operator.name>.jwt.

NATS-Account-Server

To vend the credentials to the nats-server, we'll use a nats-account-server. Let's start a nats-account-server to serve the JWT credentials:

nats-account-server -nsc ~/.nsc/nats/SAOP

The server will by default vend JWT configurations on the an endpoint at: http(s)://<server_url>/jwt/v1/accounts/.

NATS Server Configuration

The server configuration will need:

• The operator JWT - (~/.nsc/nats/<operator_name>/<operator.name>.jwt)
• The URL where the server can resolve accounts (http://localhost:9090/jwt/v1/accounts/)
• The public key of the system_account

The only thing we don't have handy is the public key for the system account. We can get it easy enough:

nsc list accounts

╭─────────────────────────────────────────────────────────────────╮
│                            Accounts                             │
├──────┬──────────────────────────────────────────────────────────┤
│ Name │ Public Key                                               │
├──────┼──────────────────────────────────────────────────────────┤
│ SYS  │ ADWJVSUSEVC2GHL5GRATN2LOEOQOY2E6Z2VXNU3JEIK6BDGPWNIW3AXF │
╰──────┴──────────────────────────────────────────────────────────╯

Because the server has additional resolver implementations, you need to enclose the server url like: URL(<url>).

Let's create server config with the following contents and save it to server.conf:

operator: /Users/synadia/.nsc/nats/SAOP/SAOP.jwt
system_account: ADWJVSUSEVC2GHL5GRATN2LOEOQOY2E6Z2VXNU3JEIK6BDGPWNIW3AXF
resolver: URL(http://localhost:9090/jwt/v1/accounts/)

Let's start the nats-server:

nats-server -c server.conf

Inspecting Server Events

Let's add a subscriber for all the events published by the system account:

nats sub --creds ~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds ">"

Very quickly we'll start seeing messages from the server as they are published by the NATS server. As should be expected, the messages are just JSON, so they can easily be inspected even if just using a simple nats sub to read them.

To see an account update:

nats pub --creds ~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds foo bar

The subscriber will print the connect and disconnect:

{
  "server": {
    "host": "0.0.0.0",
    "id": "NBTGVY3OKDKEAJPUXRHZLKBCRH3LWCKZ6ZXTAJRS2RMYN3PMDRMUZWPR",
    "ver": "2.0.0-RC5",
    "seq": 32,
    "time": "2019-05-03T14:53:15.455266-05:00"
  },
  "acc": "ADWJVSUSEVC2GHL5GRATN2LOEOQOY2E6Z2VXNU3JEIK6BDGPWNIW3AXF",
  "conns": 1,
  "total_conns": 1
}
{
  "server": {
    "host": "0.0.0.0",
    "id": "NBTGVY3OKDKEAJPUXRHZLKBCRH3LWCKZ6ZXTAJRS2RMYN3PMDRMUZWPR",
    "ver": "2.0.0-RC5",
    "seq": 33,
    "time": "2019-05-03T14:53:15.455304-05:00"
  },
  "client": {
    "start": "2019-05-03T14:53:15.453824-05:00",
    "host": "127.0.0.1",
    "id": 6,
    "acc": "ADWJVSUSEVC2GHL5GRATN2LOEOQOY2E6Z2VXNU3JEIK6BDGPWNIW3AXF",
    "user": "UACPEXCAZEYWZK4O52MEGWGK4BH3OSGYM3P3C3F3LF2NGNZUS24IVG36",
    "name": "NATS Sample Publisher",
    "lang": "go",
    "ver": "1.7.0",
    "stop": "2019-05-03T14:53:15.45526-05:00"
  },
  "sent": {
    "msgs": 1,
    "bytes": 3
  },
  "received": {
    "msgs": 0,
    "bytes": 0
  },
  "reason": "Client Closed"
}

User Services

$SYS.REQ.USER.INFO - Request Connected User Information

For the active connection, get basic user information including the account name, permissions, and expiry, if applicable. Note, this works with any connected user, not just a system account user.

nats request --creds ~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds \$SYS.REQ.USER.INFO ""

Published [$SYS.REQ.USER.INFO] : ''
Received  [_INBOX.DQD44ugVt0O4Ur3pWIOOD1.WQOBevoq] : '{
  "user": "UACPEXCAZEYWZK4O52MEGWGK4BH3OSGYM3P3C3F3LF2NGNZUS24IVG36",
  "account": "ADWJVSUSEVC2GHL5GRATN2LOEOQOY2E6Z2VXNU3JEIK6BDGPWNIW3AXF"
}'

System Services

$SYS.REQ.SERVER.PING.IDZ - Discovering Servers

To discover servers in the cluster to get their ID and name, publish a request to $SYS.REQ.SERVER.PING.IDZ.

nats request --creds ~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds \$SYS.REQ.SERVER.PING.IDZ ""

Published [$SYS.REQ.SERVER.PING.IDZ] : ''
Received  [_INBOX.DQD44ugVt0O4Ur3pWIOOD1.WQOBevoq] : '{
  "host": "0.0.0.0",
  "id": "NC7AKPQRC6CIZGWRJOTVFIGVSL7VW7WXTQCTUJFNG7HTCMCKQTGE5PUL",
  "name": "n1"
}'

$SYS.REQ.SERVER.PING - Discovering Servers + Stats

To discover servers in the cluster, and get a small health summary, publish a request to $SYS.REQ.SERVER.PING. Note that while the example below uses nats-req, only the first answer for the request will be printed. You can easily modify the example to wait until no additional responses are received for a specific amount of time, thus allowing for all responses to be collected.

nats request --creds ~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds \$SYS.REQ.SERVER.PING ""

Published [$SYS.REQ.SERVER.PING] : ''
Received  [_INBOX.G5mbsf0k7l7nb4eWHa7GTT.omklmvnm] : '{
  "server": {
    "host": "0.0.0.0",
    "id": "NCZQDUX77OSSTGN2ESEOCP4X7GISMARX3H4DBGZBY34VLAI4TQEPK6P6",
    "ver": "2.0.0-RC9",
    "seq": 47,
    "time": "2019-05-02T14:02:46.402166-05:00"
  },
  "statsz": {
    "start": "2019-05-02T13:41:01.113179-05:00",
    "mem": 12922880,
    "cores": 20,
    "cpu": 0,
    "connections": 2,
    "total_connections": 2,
    "active_accounts": 1,
    "subscriptions": 10,
    "sent": {
      "msgs": 7,
      "bytes": 2761
    },
    "received": {
      "msgs": 0,
      "bytes": 0
    },
    "slow_consumers": 0
  }
}'

$SYS.REQ.SERVER.<id>.STATSZ - Requesting Server Stats Summary

If you know the server id for a particular server (such as from a response to $SYS.REQ.SERVER.PING), you can query the specific server for its health information:

nats request --creds ~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds \$SYS.REQ.SERVER.NC7AKPQRC6CIZGWRJOTVFIGVSL7VW7WXTQCTUJFNG7HTCMCKQTGE5PUL.STATSZ ""

Published [$SYS.REQ.SERVER.NC7AKPQRC6CIZGWRJOTVFIGVSL7VW7WXTQCTUJFNG7HTCMCKQTGE5PUL.STATSZ] : ''
Received  [_INBOX.DQD44ugVt0O4Ur3pWIOOD1.WQOBevoq] : '{
  "server": {
    "host": "0.0.0.0",
    "id": "NC7AKPQRC6CIZGWRJOTVFIGVSL7VW7WXTQCTUJFNG7HTCMCKQTGE5PUL",
    "ver": "2.0.0-RC5",
    "seq": 25,
    "time": "2019-05-03T14:34:02.066077-05:00"
  },
  "statsz": {
    "start": "2019-05-03T14:32:19.969037-05:00",
    "mem": 11874304,
    "cores": 20,
    "cpu": 0,
    "connections": 2,
    "total_connections": 4,
    "active_accounts": 1,
    "subscriptions": 10,
    "sent": {
      "msgs": 26,
      "bytes": 9096
    },
    "received": {
      "msgs": 2,
      "bytes": 0
    },
    "slow_consumers": 0
  }
}'

$SYS.REQ.SERVER.<id>.PROFILEZ - Request Profiling Information

If profiling is enabled for a server, this service enables requesting it from the server. The request payload must specify the name of the profile being requested with an optional debug level, including:

• allocs - 0, 1
• block - 0
• goroutine - 0, 1, 2
• heap - 0, 1
• mutex - 0
• threadcount - 0

nats request --creds ~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds \$SYS.REQ.SERVER.NC7AKPQRC6CIZGWRJOTVFIGVSL7VW7WXTQCTUJFNG7HTCMCKQTGE5PUL.PROFILEZ '{"name": "heap", "debug": 1}'

Published [$SYS.REQ.SERVER.NC7AKPQRC6CIZGWRJOTVFIGVSL7VW7WXTQCTUJFNG7HTCMCKQTGE5PUL.PROFILEZ] : '{
  "name": "heap",
  "debug": 1
}'
Received  [_INBOX.DQD44ugVt0O4Ur3pWIOOD1.WQOBevoq] : '{
  "profile": "<base64-encoded profile output>"
}'

$SYS.REQ.SERVER.<id>.RELOAD - Hot Reload Configuration

Sending a request to this service will attempt to hot reload the server configuration, akin to nats-server --signal reload. If there are errors with the new configuration, they will be returned in an error field in the response.

nats request --creds ~/.nkeys/SAOP/accounts/SYS/users/SYSU.creds \$SYS.REQ.SERVER.NC7AKPQRC6CIZGWRJOTVFIGVSL7VW7WXTQCTUJFNG7HTCMCKQTGE5PUL.RELOAD ''

Published [$SYS.REQ.SERVER.NC7AKPQRC6CIZGWRJOTVFIGVSL7VW7WXTQCTUJFNG7HTCMCKQTGE5PUL.RELOAD] : ''
Received  [_INBOX.DQD44ugVt0O4Ur3pWIOOD1.WQOBevoq] : '{
  "server": {
    "host": "0.0.0.0",
    "id": "NC7AKPQRC6CIZGWRJOTVFIGVSL7VW7WXTQCTUJFNG7HTCMCKQTGE5PUL",
    "ver": "2.10.0-RC5",
    "seq": 25,
    "time": "2023-09-19T14:34:02.066077-04:00"
  }
}'

WebSocket

Supported since NATS Server version 2.2

WebSocket support can be enabled in the server and may be used alongside the traditional TCP socket connections. TLS, compression and Origin Header checking are supported.

Important

• NATS Supports only WebSocket data frames in Binary, not Text format (https://tools.ietf.org/html/rfc6455#section-5.6). The server will always send in Binary and your clients MUST send in Binary too.
• For writers of client libraries: a WebSocket frame is not guaranteed to contain a full NATS protocol (actually will generally not). Any data from a frame must be going through a parser that can handle partial protocols. See the protocol description here.

Configuration

To enable WebSocket support in the server, add a websocket configuration block in the server's configuration file like the following:

websocket {
    # Specify a host and port to listen for websocket connections
    #
    # listen: "host:port"

    # It can also be configured with individual parameters,
    # namely host and port.
    #
    # host: "hostname"
    port: 443

    # This will optionally specify what host:port for websocket
    # connections to be advertised in the cluster.
    #
    # advertise: "host:port"

    # TLS configuration is required by default
    #
    tls {
      cert_file: "/path/to/cert.pem"
      key_file: "/path/to/key.pem"
    }

    # For test environments, you can disable the need for TLS
    # by explicitly setting this option to `true`
    #
    # no_tls: true

    # [Cross-origin resource sharing option](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS).
    #
    # IMPORTANT! This option is used only when the http request presents an Origin
    # header, which is the case for web browsers. If no Origin header is present,
    # this check will not be performed.
    #
    # When set to `true`, the HTTP origin header must match the request’s hostname.
    # The default is `false`.
    #
    # same_origin: true

    # [Cross-origin resource sharing option](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS).
    #
    # IMPORTANT! This option is used only when the http request presents an Origin
    # header, which is the case for web browsers. If no Origin header is present,
    # this check will not be performed.
    #
    # List of accepted origins. When empty, and `same_origin` is `false`, clients from any origin are allowed to connect.
    # This list specifies the only accepted values for the client's request Origin header. The scheme,
    # host and port must match. By convention, the absence of TCP port in the URL will be port 80
    # for an "http://" scheme, and 443 for "https://".
    #
    # allowed_origins [
    #    "http://www.example.com"
    #    "https://www.other-example.com"
    # ]

    # This enables support for compressed websocket frames
    # in the server. For compression to be used, both server
    # and client have to support it.
    #
    # compression: true

    # This is the total time allowed for the server to
    # read the client request and write the response back
    # to the client. This includes the time needed for the
    # TLS handshake.
    #
    # handshake_timeout: "2s"

    # Name for an HTTP cookie, that if present will be used as a client JWT.
    # If the client specifies a JWT in the CONNECT protocol, this option is ignored.
    # The cookie should be set by the HTTP server as described [here](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies).
    # This setting is useful when generating NATS `Bearer` client JWTs as the
    # result of some authentication mechanism. The HTTP server after correct
    # authentication can issue a JWT for the user, that is set securely preventing
    # access by unintended scripts. Note these JWTs must be [NATS JWTs](https://docs.nats.io/nats-server/configuration/securing_nats/jwt).
    #
    # jwt_cookie: "my_jwt_cookie_name"

    # If no user name is provided when a websocket client connects, will default
    # this user name in the authentication phase. If specified, this will
    # override, for websocket clients, any `no_auth_user` value defined in the
    # main configuration file.
    # Note that this is not compatible with running the server in operator mode.
    #
    # no_auth_user: "my_username_for_apps_not_providing_credentials"

    # See below to know what is the normal way of limiting websocket clients
    # to specific users.
    # If there are no users specified in the configuration, this simple authorization
    # block allows you to override the values that would be configured in the
    # equivalent block in the main section.
    #
    # authorization {
    #     # If this is specified, the client has to provide the same username
    #     # and password to be able to connect.
    #     # username: "my_user_name"
    #     # password: "my_password"
    #
    #     # If this is specified, the password field in the CONNECT has to
    #     # match this token.
    #     # token: "my_token"
    #
    #     # This overrides the main's authorization timeout. For consistency
    #     # with the main's authorization configuration block, this is expressed
    #     # as a number of seconds.
    #     # timeout: 2.0
    #}
}

Authorization of WebSocket Users

Authentication

NATS supports different forms of authentication for clients connecting over WebSocket:

• username/password
• token
• NKEYS
• client certificates
• JWTs

You can get some more information about how applications connecting over WebSocket can use those different forms of authentication here

Restricting connection types

A new field when configuring users allows you to restrict which type of connections are allowed for a specific user.

Consider this configuration:

authorization {
  users [
    {user: foo password: foopwd, permission: {...}}
    {user: bar password: barpwd, permission: {...}}
  ]
}

If a WebSocket client were to connect and use the username foo and password foopwd, it would be accepted. Now suppose that you would want the WebSocket client to only be accepted if it connected using the username bar and password barpwd, then you would use the option allowed_connection_types to restrict which type of connections can bind to this user.

authorization {
  users [
    {user: foo password: foopwd, permission: {...}}
    {user: bar password: barpwd, permission: {...}, allowed_connection_types: ["WEBSOCKET"]}
  ]
}

The option allowed_connection_types (also can be named connection_types or clients) as you can see is a list, and you can allow several types of clients. Suppose you want the user bar to accept both standard NATS clients and WebSocket clients, you would configure the user like this:

authorization {
  users [
    {user: foo password: foopwd, permission: {...}}
    {user: bar password: barpwd, permission: {...}, allowed_connection_types: ["STANDARD", "WEBSOCKET"]}
  ]
}

The absence of allowed_connection_types means that all types of connections are allowed (the default behavior).

The possible values are currently:

• STANDARD
• WEBSOCKET
• LEAFNODE
• MQTT

Leaf nodes connections

You can configure remote Leaf node connections so that they connect to the Websocket port instead of the Leaf node port. See Leafnode section.

Docker

When running on Docker, WebSocket is not enabled by default, so you'll have to create a configuration file with the minimal entries, such as:

websocket 
{
     port: 8080
     no_tls: true
}

Assuming the configuration was stored in /tmp/nats.conf, you can start docker as follows:

docker run -it --rm  -v /tmp:/container -p 8080:8080 nats -c /container/nats.conf

Managing and Monitoring your NATS Server Infrastructure

Managing a NATS Server is simple, typical lifecycle operations include:

• Using the nats CLI tool to check server cluster connectivity and latencies, as well as get account information, and manage and interact with streams (and other NATS applications). Try the following examples to learn about the most common ways to use nats.
  • nats cheat
  • nats cheat server
  • nats stream --help to monitor, manage and interact with streams
  • nats consumer --help to monitor, manage stream consumers
  • nats context --help if you need to switch between servers, clusters or user credentials
• Using the nsc CLI tool when using JWT based authentication and authorization, to create, revoke operators, accounts, and user (i.e. client applications) JWTs and keys.
• Sending signals to a server to reload a configuration or rotate log files
• Upgrading a server (or cluster)
• Understanding slow consumers
• Monitoring the server via:
  • The monitoring endpoint and tools like nats-top
  • By subscribing to system events
    • Commerical option for high-cardinality monitoring via the system account: Synadia Insights
• Gracefully shut down a server with Lame Duck Mode

Monitoring

Monitoring NATS

To monitor the NATS messaging system, nats-server provides a lightweight HTTP server on a dedicated monitoring port. The monitoring server provides several endpoints, providing statistics and other information about the following:

• General Server Information (/varz)
• Connections (/connz)
• Routing (/routez)
• Gateway (/gatewayz)
• Leaf Nodes (/leafz)
• Subscription Routing (/subsz)
• Account Information (/accountz)
• Account Stats (/accstatz)
• JetStream Information (/jsz)
• Health (/healthz)

All endpoints return a JSON object.

Note that info from these monitoring endpoints is also available through System services

The NATS monitoring endpoints support JSONP and CORS, making it easy to create single page monitoring web applications. Part of the NATS ecosystem is a tool called nats-top that visualizes data from these endpoints on the command line.

注意

nats-server does not have authentication/authorization for the monitoring endpoint. When you plan to open your nats-server to the internet make sure to not expose the monitoring port as well. By default, monitoring binds to every interface 0.0.0.0 so consider setting monitoring to localhost or have appropriate firewall rules.

Enabling monitoring

Monitoring can be enabled in server configuration or as a server command-line option. The conventional port is 8222.

As server configuration:

http_port: 8222

As a command-line option:

nats-server -m 8222

Once the server is running using one of the two methods, go to http://localhost:8222 to browse the available endpoints detailed below.

Alternatively, if you have System account enabled, monitoring endpoints available as "System services"

Monitoring Endpoints

General Information (/varz)

The /varz endpoint returns general information about the server state and configuration.

Arguments

N/A

Example

https://demo.nats.io:8222/varz

Response

{
  "server_id": "NACDVKFBUW4C4XA24OOT6L4MDP56MW76J5RJDFXG7HLABSB46DCMWCOW",
  "version": "2.0.0",
  "proto": 1,
  "go": "go1.12",
  "host": "0.0.0.0",
  "port": 4222,
  "max_connections": 65536,
  "ping_interval": 120000000000,
  "ping_max": 2,
  "http_host": "0.0.0.0",
  "http_port": 8222,
  "https_port": 0,
  "auth_timeout": 1,
  "max_control_line": 4096,
  "max_payload": 1048576,
  "max_pending": 67108864,
  "cluster": {},
  "gateway": {},
  "leaf": {},
  "tls_timeout": 0.5,
  "write_deadline": 2000000000,
  "start": "2019-06-24T14:24:43.928582-07:00",
  "now": "2019-06-24T14:24:46.894852-07:00",
  "uptime": "2s",
  "mem": 9617408,
  "cores": 4,
  "gomaxprocs": 4,
  "cpu": 0,
  "connections": 0,
  "total_connections": 0,
  "routes": 0,
  "remotes": 0,
  "leafnodes": 0,
  "in_msgs": 0,
  "out_msgs": 0,
  "in_bytes": 0,
  "out_bytes": 0,
  "slow_consumers": 2,
  "subscriptions": 0,
  "http_req_stats": {
    "/": 0,
    "/connz": 0,
    "/gatewayz": 0,
    "/routez": 0,
    "/subsz": 0,
    "/varz": 1
  },
  "config_load_time": "2019-06-24T14:24:43.928582-07:00",
  "slow_consumer_stats": {
    "clients": 1,
    "routes": 1,
    "gateways": 0,
    "leafs": 0
  }
}

Connection Information (/connz)

The /connz endpoint reports more detailed information on current and recently closed connections. It uses a paging mechanism which defaults to 1024 connections.

Arguments

The server will default to holding the last 10,000 closed connections.

Sort Options

Examples

Get up to 1024 connections: https://demo.nats.io:8222/connz

Control limit and offset: https://demo.nats.io:8222/connz?limit=16&offset=128.

Get closed connection information: https://demo.nats.io:8222/connz?state=closed.

You can also report detailed subscription information on a per connection basis using subs=1. For example: https://demo.nats.io:8222/connz?limit=1&offset=1&subs=1.

Response

{
  "server_id": "NACDVKFBUW4C4XA24OOT6L4MDP56MW76J5RJDFXG7HLABSB46DCMWCOW",
  "now": "2019-06-24T14:28:16.520365-07:00",
  "num_connections": 2,
  "total": 2,
  "offset": 0,
  "limit": 1024,
  "connections": [
    {
      "cid": 5,
      "kind": "Client",
      "type": "nats",
      "ip": "127.0.0.1",
      "port": 62714,
      "start": "2021-09-09T23:16:43.040862Z",
      "last_activity": "2021-09-09T23:16:43.042364Z",
      "rtt": "95µs",
      "uptime": "5s",
      "idle": "5s",
      "pending_bytes": 0,
      "in_msgs": 0,
      "out_msgs": 0,
      "in_bytes": 0,
      "out_bytes": 0,
      "subscriptions": 1,
      "name": "NATS Benchmark",
      "lang": "go",
      "version": "1.12.1"
    },
    {
      "cid": 6,
      "kind": "Client",
      "type": "nats",
      "ip": "127.0.0.1",
      "port": 62715,
      "start": "2021-09-09T23:16:43.042557Z",
      "last_activity": "2021-09-09T23:16:43.042811Z",
      "rtt": "100µs",
      "uptime": "5s",
      "idle": "5s",
      "pending_bytes": 0,
      "in_msgs": 0,
      "out_msgs": 0,
      "in_bytes": 0,
      "out_bytes": 0,
      "subscriptions": 1,
      "name": "NATS Benchmark",
      "lang": "go",
      "version": "1.12.1"
    },
    {
      "cid": 7,
      "kind": "Client",
      "type": "mqtt",
      "ip": "::1",
      "port": 62718,
      "start": "2021-09-09T23:16:45.391459Z",
      "last_activity": "2021-09-09T23:16:45.395869Z",
      "rtt": "0s",
      "uptime": "2s",
      "idle": "2s",
      "pending_bytes": 0,
      "in_msgs": 0,
      "out_msgs": 0,
      "in_bytes": 0,
      "out_bytes": 0,
      "subscriptions": 2,
      "mqtt_client": "mqtt_sub"
    }
  ]
}

Route Information (/routez)

The /routez endpoint reports information on active routes for a cluster. Routes are expected to be low, so there is no paging mechanism with this endpoint.

Arguments

As noted above, the routez endpoint does support the subs argument from the /connz endpoint. For example: https://demo.nats.io:8222/routez?subs=1

Example

• Get route information: https://demo.nats.io:8222/routez?subs=1

Response

{
  "server_id": "NACDVKFBUW4C4XA24OOT6L4MDP56MW76J5RJDFXG7HLABSB46DCMWCOW",
  "now": "2019-06-24T14:29:16.046656-07:00",
  "num_routes": 1,
  "routes": [
    {
      "rid": 1,
      "remote_id": "de475c0041418afc799bccf0fdd61b47",
      "did_solicit": true,
      "ip": "127.0.0.1",
      "port": 61791,
      "pending_size": 0,
      "in_msgs": 0,
      "out_msgs": 0,
      "in_bytes": 0,
      "out_bytes": 0,
      "subscriptions": 0
    }
  ]
}

Gateway Information (/gatewayz)

The /gatewayz endpoint reports information about gateways used to create a NATS supercluster. Like routes, the number of gateways are expected to be low, so there is no paging mechanism with this endpoint.

Arguments

Examples

• Retrieve Gateway Information: https://demo.nats.io:8222/gatewayz

Response

{
  "server_id": "NANVBOU62MDUWTXWRQ5KH3PSMYNCHCEUHQV3TW3YH7WZLS7FMJE6END6",
  "now": "2019-07-24T18:02:55.597398-06:00",
  "name": "region1",
  "host": "2601:283:4601:1350:1895:efda:2010:95a1",
  "port": 4501,
  "outbound_gateways": {
    "region2": {
      "configured": true,
      "connection": {
        "cid": 7,
        "ip": "127.0.0.1",
        "port": 5500,
        "start": "2019-07-24T18:02:48.765621-06:00",
        "last_activity": "2019-07-24T18:02:48.765621-06:00",
        "uptime": "6s",
        "idle": "6s",
        "pending_bytes": 0,
        "in_msgs": 0,
        "out_msgs": 0,
        "in_bytes": 0,
        "out_bytes": 0,
        "subscriptions": 0,
        "name": "NCXBIYWT7MV7OAQTCR4QTKBN3X3HDFGSFWTURTCQ22ZZB6NKKJPO7MN4"
      }
    },
    "region3": {
      "configured": true,
      "connection": {
        "cid": 5,
        "ip": "::1",
        "port": 6500,
        "start": "2019-07-24T18:02:48.764685-06:00",
        "last_activity": "2019-07-24T18:02:48.764685-06:00",
        "uptime": "6s",
        "idle": "6s",
        "pending_bytes": 0,
        "in_msgs": 0,
        "out_msgs": 0,
        "in_bytes": 0,
        "out_bytes": 0,
        "subscriptions": 0,
        "name": "NCVS7Q65WX3FGIL2YQRLI77CE6MQRWO2Y453HYVLNMBMTVLOKMPW7R6K"
      }
    }
  },
  "inbound_gateways": {
    "region2": [
      {
        "configured": false,
        "connection": {
          "cid": 9,
          "ip": "::1",
          "port": 52029,
          "start": "2019-07-24T18:02:48.76677-06:00",
          "last_activity": "2019-07-24T18:02:48.767096-06:00",
          "uptime": "6s",
          "idle": "6s",
          "pending_bytes": 0,
          "in_msgs": 0,
          "out_msgs": 0,
          "in_bytes": 0,
          "out_bytes": 0,
          "subscriptions": 0,
          "name": "NCXBIYWT7MV7OAQTCR4QTKBN3X3HDFGSFWTURTCQ22ZZB6NKKJPO7MN4"
        }
      }
    ],
    "region3": [
      {
        "configured": false,
        "connection": {
          "cid": 4,
          "ip": "::1",
          "port": 52025,
          "start": "2019-07-24T18:02:48.764577-06:00",
          "last_activity": "2019-07-24T18:02:48.764994-06:00",
          "uptime": "6s",
          "idle": "6s",
          "pending_bytes": 0,
          "in_msgs": 0,
          "out_msgs": 0,
          "in_bytes": 0,
          "out_bytes": 0,
          "subscriptions": 0,
          "name": "NCVS7Q65WX3FGIL2YQRLI77CE6MQRWO2Y453HYVLNMBMTVLOKMPW7R6K"
        }
      },
      {
        "configured": false,
        "connection": {
          "cid": 8,
          "ip": "127.0.0.1",
          "port": 52026,
          "start": "2019-07-24T18:02:48.766173-06:00",
          "last_activity": "2019-07-24T18:02:48.766999-06:00",
          "uptime": "6s",
          "idle": "6s",
          "pending_bytes": 0,
          "in_msgs": 0,
          "out_msgs": 0,
          "in_bytes": 0,
          "out_bytes": 0,
          "subscriptions": 0,
          "name": "NCKCYK5LE3VVGOJQ66F65KA27UFPCLBPX4N4YOPOXO3KHGMW24USPCKN"
        }
      }
    ]
  }
}

Leaf Node Information (/leafz)

The /leafz endpoint reports detailed information about the leaf node connections.

Arguments

As noted above, the leafz endpoint does support the subs argument from the /connz endpoint. For example: https://demo.nats.io:8222/leafz?subs=1

Example

• Get leaf nodes information: https://demo.nats.io:8222/leafz?subs=1

Response

{
  "server_id": "NC2FJCRMPBE5RI5OSRN7TKUCWQONCKNXHKJXCJIDVSAZ6727M7MQFVT3",
  "now": "2019-08-27T09:07:05.841132-06:00",
  "leafnodes": 1,
  "leafs": [
    {
      "account": "$G",
      "ip": "127.0.0.1",
      "port": 6223,
      "rtt": "200µs",
      "in_msgs": 0,
      "out_msgs": 10000,
      "in_bytes": 0,
      "out_bytes": 1280000,
      "subscriptions": 1,
      "subscriptions_list": ["foo"]
    }
  ]
}

Subscription Routing Information (/subsz)

The /subsz endpoint reports detailed information about the current subscriptions and the routing data structure. It is not normally used.

Arguments

Example

• Get subscription routing information: https://demo.nats.io:8222/subsz

Response

{
  "num_subscriptions": 2,
  "num_cache": 0,
  "num_inserts": 2,
  "num_removes": 0,
  "num_matches": 0,
  "cache_hit_rate": 0,
  "max_fanout": 0,
  "avg_fanout": 0
}

Account Information (/accountz)

The /accountz endpoint reports information on a server's active accounts. The default behavior is to return a list of all accounts known to the server.

Example

• Get list of all accounts: https://demo.nats.io:8222/accountz
• Get details for specific account $G: https://demo.nats.io:8222/accountz?acc=$G

Response

Default behavior:

{
  "server_id": "NAB2EEQ3DLS2BHU4K2YMXMPIOOOAOFOAQAC5NQRIEUI4BHZKFBI4ZU4A",
  "now": "2021-02-08T17:31:29.551146-05:00",
  "system_account": "AAAXAUVSGK7TCRHFIRAS4SYXVJ76EWDMNXZM6ARFGXP7BASNDGLKU7A5",
  "accounts": ["AAAXAUVSGK7TCRHFIRAS4SYXVJ76EWDMNXZM6ARFGXP7BASNDGLKU7A5", "$G"]
}

Retrieve specific account:

{
  "server_id": "NAB2EEQ3DLS2BHU4K2YMXMPIOOOAOFOAQAC5NQRIEUI4BHZKFBI4ZU4A",
  "now": "2021-02-08T17:37:55.80856-05:00",
  "system_account": "AAAXAUVSGK7TCRHFIRAS4SYXVJ76EWDMNXZM6ARFGXP7BASNDGLKU7A5",
  "account_detail": {
    "account_name": "AAAXAUVSGK7TCRHFIRAS4SYXVJ76EWDMNXZM6ARFGXP7BASNDGLKU7A5",
    "update_time": "2021-02-08T17:31:22.390334-05:00",
    "is_system": true,
    "expired": false,
    "complete": true,
    "jetstream_enabled": false,
    "leafnode_connections": 0,
    "client_connections": 0,
    "subscriptions": 42,
    "exports": [
      {
        "subject": "$SYS.DEBUG.SUBSCRIBERS",
        "type": "service",
        "response_type": "Singleton"
      }
    ],
    "jwt": "eyJ0eXAiOiJqd3QiLCJhbGciOiJlZDI1NTE5In0.eyJqdGkiOiJVVlU2VEpXRU8zS0hYWTZVMkgzM0RCVklET1A3U05DTkJPMlM0M1dPNUM2T1RTTDNVSUxBIiwiaWF0IjoxNjAzNDczNzg4LCJpc3MiOiJPQlU1TzVGSjMyNFVEUFJCSVZSR0Y3Q05FT0hHTFBTN0VZUEJUVlFaS1NCSElJWklCNkhENjZKRiIsIm5hbWUiOiJTWVMiLCJzdWIiOiJBQUFYQVVWU0dLN1RDUkhGSVJBUzRTWVhWSjc2RVdETU5YWk02QVJGR1hQN0JBU05ER0xLVTdBNSIsInR5cGUiOiJhY2NvdW50IiwibmF0cyI6eyJsaW1pdHMiOnsic3VicyI6LTEsImNvbm4iOi0xLCJsZWFmIjotMSwiaW1wb3J0cyI6LTEsImV4cG9ydHMiOi0xLCJkYXRhIjotMSwicGF5bG9hZCI6LTEsIndpbGRjYXJkcyI6dHJ1ZX19fQ.CeGo16i5oD0b1uBJ8UdGmLH-l9dL8yNqXHggkAt2T5c88fM7k4G08wLguMAnlvzrdlYvdZvOx_5tHLuDZmGgCg",
    "issuer_key": "OBU5O5FJ324UDPRBIVRGF7CNEOHGLPS7EYPBTVQZKSBHIIZIB6HD66JF",
    "name_tag": "SYS",
    "decoded_jwt": {
      "jti": "UVU6TJWEO3KHXY6U2H33DBVIDOP7SNCNBO2S43WO5C6OTSL3UILA",
      "iat": 1603473788,
      "iss": "OBU5O5FJ324UDPRBIVRGF7CNEOHGLPS7EYPBTVQZKSBHIIZIB6HD66JF",
      "name": "SYS",
      "sub": "AAAXAUVSGK7TCRHFIRAS4SYXVJ76EWDMNXZM6ARFGXP7BASNDGLKU7A5",
      "nats": {
        "limits": {
          "subs": -1,
          "data": -1,
          "payload": -1,
          "imports": -1,
          "exports": -1,
          "wildcards": true,
          "conn": -1,
          "leaf": -1
        },
        "default_permissions": {
          "pub": {},
          "sub": {}
        },
        "type": "account",
        "version": 1
      }
    },
    "sublist_stats": {
      "num_subscriptions": 42,
      "num_cache": 6,
      "num_inserts": 42,
      "num_removes": 0,
      "num_matches": 6,
      "cache_hit_rate": 0,
      "max_fanout": 1,
      "avg_fanout": 0.8333333333333334
    }
  }
}

Account Statistics (/accstatz)

The /accstatz endpoint reports per-account statistics such as the number of connections, messages/bytes in/out, etc.

Arguments

Examples

• Accounts with active connections - https://demo.nats.io:8222/accstatz
• Include ones without any connections (in this case $SYS)- https://demo.nats.io:8222/accstatz?unused=1

Response

{
  "server_id": "NDJ5M4F5WAIBUA26NJ3QMH532AQPN7QNTJP3Y4SBHSHL4Y7QUAKNJEAF",
  "now": "2022-10-19T17:16:20.881296749Z",
  "account_statz": [
    {
      "acc": "default",
      "conns": 31,
      "leafnodes": 2,
      "total_conns": 33,
      "num_subscriptions": 45,
      "sent": {
        "msgs": 1876970,
        "bytes": 246705616
      },
      "received": {
        "msgs": 1347454,
        "bytes": 219438308
      },
      "slow_consumers": 29
    },
    {
      "acc": "$G",
      "conns": 1,
      "leafnodes": 0,
      "total_conns": 1,
      "num_subscriptions": 3,
      "sent": {
        "msgs": 0,
        "bytes": 0
      },
      "received": {
        "msgs": 107,
        "bytes": 1094
      },
      "slow_consumers": 0
    }
  ]
}

JetStream Information (/jsz)

The /jsz endpoint reports more detailed information on JetStream. For accounts, it uses a paging mechanism that defaults to 1024 connections.

Note: If you're in a clustered environment, it is recommended to retrieve the information from the stream's leader in order to get the most accurate and up-to-date data.

Arguments

Examples

Get basic JetStream information: https://demo.nats.io:8222/jsz

Request accounts and control limit and offset: https://demo.nats.io:8222/jsz?accounts=true&limit=16&offset=128.

You can also report detailed consumer information on a per connection basis using consumer=true. For example: https://demo.nats.io:8222/jsz?consumers=true.

Response

{
  "server_id": "NCVIDODSZ45C5OD67ZD7EJUIJPQDP6CM74SJX6TJIF2G7NLYS5LCVYHS",
  "now": "2021-02-08T19:08:30.555533-05:00",
  "config": {
    "max_memory": 10485760,
    "max_storage": 10485760,
    "store_dir": "/var/folders/9h/6g_c9l6n6bb8gp331d_9y0_w0000gn/T/srv_7500251552558",
    "unique_tag": "az"
  },
  "memory": 0,
  "storage": 66,
  "api": {
    "total": 5,
    "errors": 0
  },
  "total_streams": 1,
  "total_consumers": 1,
  "total_messages": 1,
  "total_message_bytes": 33,
  "meta_cluster": {
    "name": "cluster_name",
    "replicas": [
      {
        "name": "server_5500",
        "current": false,
        "active": 2932926000
      }
    ]
  },
  "account_details": [
    {
      "name": "BCC_TO_HAVE_ONE_EXTRA",
      "id": "BCC_TO_HAVE_ONE_EXTRA",
      "memory": 0,
      "storage": 0,
      "api": {
        "total": 0,
        "errors": 0
      }
    },
    {
      "name": "ACC",
      "id": "ACC",
      "memory": 0,
      "storage": 66,
      "api": {
        "total": 5,
        "errors": 0
      },
      "stream_detail": [
        {
          "name": "my-stream-replicated",
          "cluster": {
            "name": "cluster_name",
            "replicas": [
              {
                "name": "server_5500",
                "current": false,
                "active": 2931517000
              }
            ]
          },
          "state": {
            "messages": 1,
            "bytes": 33,
            "first_seq": 1,
            "first_ts": "2021-02-09T00:08:27.623735Z",
            "last_seq": 1,
            "last_ts": "2021-02-09T00:08:27.623735Z",
            "consumer_count": 1
          },
          "consumer_detail": [
            {
              "stream_name": "my-stream-replicated",
              "name": "my-consumer-replicated",
              "created": "2021-02-09T00:08:27.427631Z",
              "delivered": {
                "consumer_seq": 0,
                "stream_seq": 0
              },
              "ack_floor": {
                "consumer_seq": 0,
                "stream_seq": 0
              },
              "num_ack_pending": 0,
              "num_redelivered": 0,
              "num_waiting": 0,
              "num_pending": 1,
              "cluster": {
                "name": "cluster_name",
                "replicas": [
                  {
                    "name": "server_5500",
                    "current": false,
                    "active": 2933232000
                  }
                ]
              }
            }
          ]
        }
      ]
    }
  ]
}

Health (/healthz)

The /healthz endpoint returns OK if the server is able to accept connections.

Arguments

Example

• Default - https://demo.nats.io:8222/healthz
• Expect JetStream - https://demo.nats.io:8222/healthz?js-enabled-only=true

Response

{ "status": "ok" }

Creating Monitoring Applications

NATS monitoring endpoints support JSONP and CORS. You can easily create single page web applications for monitoring. To do this you simply pass the callback query parameter to any endpoint.

For example:

https://demo.nats.io:8222/connz?callback=cb

Here is a JQuery example implementation:

$.getJSON("https://demo.nats.io:8222/connz?callback=?", function (data) {
  console.log(data);
});

Monitoring Tools

In addition to writing custom monitoring tools, you can monitor nats-server in Prometheus. The Prometheus NATS Exporter allows you to configure the metrics you want to observe and store in Prometheus, and there are Grafana dashboards available for you to visualize the server metrics.

See the Walkthrough of Monitoring NATS with Prometheus and Grafana for more details.

Commercial Options

If you want a "batteries-included" approach to high-cardinality NATS monitoring and observability, try the standalone Synadia Insights.

Monitoring JetStream

Commercial Options

If you want a "batteries-included" approach to high-cardinality NATS monitoring and observability, including JetStream streams and consumers, try the standalone Synadia Insights.

Server Metrics

JetStream has a /jsz HTTP endpoint and advisories available.

Advisories

JetStream publishes a number of advisories that can inform operations about the health and the state of the Streams. These advisories are published to normal NATS subjects below $JS.EVENT.ADVISORY.> and one can store these advisories in JetStream Streams if desired.

The command nats event --js-advisory can view all these events on your console. The Golang package jsm.go can consume and render these events and have data types for each of these events.

All these events have JSON Schemas that describe them, schemas can be viewed on the CLI using the nats schema show <schema kind> command.

Dashboards

Check out NATS Surveyor Dashboards.

Managing JetStream

Once the server is running it's time to use the management tool.
Please refer to the installation section in the readme.

nats --help
nats cheat

We'll walk through the above scenario and introduce features of the CLI and of JetStream as we recreate the setup above.

Throughout this example, we'll show other commands like nats pub and nats sub to interact with the system. These are normal existing core NATS commands and JetStream is fully usable by only using core NATS.

We'll touch on some additional features but please review the section on the design model to understand all possible permutations.

Account Information

Account Information

JetStream is multi-tenant so you will need to check that your account is enabled for JetStream and is not limited. You can view your limits as follows:

nats account info

Connection Information:
               Client ID: 8
               Client IP: 127.0.0.1
                     RTT: 178.545µs
       Headers Supported: true
         Maximum Payload: 1.0 MiB
           Connected URL: nats://localhost:4222
       Connected Address: 127.0.0.1:4222
     Connected Server ID: NCCOHA6ONXJOGAEZP4WPU4UJ3IQP2VVXEPRKTQCGBCW4IL4YYW4V4KKL
JetStream Account Information:
           Memory: 0 B of 5.7 GiB
          Storage: 0 B of 11 GiB
          Streams: 0 of Unlimited
   Max Consumers: unlimited

Naming Streams, Consumers, and Accounts

Stream, Consumer (durable name), and Account names are used in both the subject namespace used by JetStream and the filesystem backing JetStream persistence. This means that when naming streams, consumers, and accounts, names must adhere to subject naming rules as well as being friendly to the file system.

We recommend the following guideline for stream, consumer, and account names:

• Alphanumeric values are recommended.
• Spaces, tabs, period (.), greater than (>) or asterisk (*) are prohibited.
• Path separators (i.e. forward slash and backward slash) are prohibited.
• Limit name length: The JetStream storage directories will include the account, stream name, and consumer name, so a generally safe approach would be to keep names under 32 characters.
• Do not use reserved file names like NUL, LPT1, etc.
• Be aware that some file systems are case insensitive so do not use stream or account names that would collide in a file system. For example, Foo and foo would collide on a Windows or Mac OSx System.

We plan to address these limitations in a future release.

Streams

The first step is to set up storage for our ORDERS related messages, these arrive on a wildcard of subjects all flowing into the same Stream and they are kept for 1 year.

Creating

nats str add ORDERS

? Subjects to consume ORDERS.*
? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Message count limit -1
? Message size limit -1
? Maximum message age limit 1y
? Maximum individual message size [? for help] (-1) -1
Stream ORDERS was created

Information for Stream ORDERS

Configuration:

             Subjects: ORDERS.*
     Acknowledgements: true
            Retention: File - Limits
             Replicas: 1
     Maximum Messages: -1
        Maximum Bytes: -1
          Maximum Age: 8760h0m0s
 Maximum Message Size: -1
  Maximum Consumers: -1

Statistics:

            Messages: 0
               Bytes: 0 B
            FirstSeq: 0
             LastSeq: 0
    Active Consumers: 0

You can get prompted interactively for missing information as above, or do it all on one command. Pressing ? in the CLI will help you map prompts to CLI options:

nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="0s" --replicas 1

Additionally one can store the configuration in a JSON file, the format of this is the same as $ nats str info ORDERS -j | jq .config:

nats str add ORDERS --config orders.json

Listing

We can confirm our Stream was created:

nats str ls

Streams:

    ORDERS

Querying

Information about the configuration of the Stream can be seen, and if you did not specify the Stream like below, it will prompt you based on all known ones:

nats str info ORDERS

Information for Stream ORDERS created 2021-02-27T16:49:36-07:00

Configuration:

             Subjects: ORDERS.*
     Acknowledgements: true
            Retention: File - Limits
             Replicas: 1
       Discard Policy: Old
     Duplicate Window: 2m0s
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: 1y0d0h0m0s
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited

State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 0
              LastSeq: 0
     Active Consumers: 0

Most commands that show data as above support -j to show the results as JSON:

nats str info ORDERS -j

{
  "config": {
    "name": "ORDERS",
    "subjects": [
      "ORDERS.*"
    ],
    "retention": "limits",
    "max_consumers": -1,
    "max_msgs": -1,
    "max_bytes": -1,
    "max_age": 31536000000000000,
    "max_msg_size": -1,
    "storage": "file",
    "discard": "old",
    "num_replicas": 1,
    "duplicate_window": 120000000000
  },
  "created": "2021-02-27T23:49:36.700424Z",
  "state": {
    "messages": 0,
    "bytes": 0,
    "first_seq": 0,
    "first_ts": "0001-01-01T00:00:00Z",
    "last_seq": 0,
    "last_ts": "0001-01-01T00:00:00Z",
    "consumer_count": 0
  }
}

This is the general pattern for the entire nats utility as it relates to JetStream - prompting for needed information but every action can be run non-interactively making it usable as a CLI API. All information output like seen above can be turned into JSON using -j.

Copying

A stream can be copied into another, which also allows the configuration of the new one to be adjusted via CLI flags:

nats str cp ORDERS ARCHIVE --subjects "ORDERS_ARCHIVE.*" --max-age 2y

Stream ORDERS was created

Information for Stream ORDERS created 2021-02-27T16:52:46-07:00

Configuration:

             Subjects: ORDERS_ARCHIVE.*
     Acknowledgements: true
            Retention: File - Limits
             Replicas: 1
       Discard Policy: Old
     Duplicate Window: 2m0s
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: 2y0d0h0m0s
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited

State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 0
              LastSeq: 0
     Active Consumers: 0

Editing

A stream configuration can be edited, which allows the configuration to be adjusted via CLI flags. Here I have an incorrectly created ORDERS stream that I fix:

nats str info ORDERS -j | jq .config.subjects

[
  "ORDERS.new"
]

Change the subjects for the stream

nats str edit ORDERS --subjects "ORDERS.*"

Stream ORDERS was updated

Information for Stream ORDERS

Configuration:

             Subjects: ORDERS.*
....

Additionally, one can store the configuration in a JSON file, the format of this is the same as $ nats str info ORDERS -j | jq .config:

nats str edit ORDERS --config orders.json

Publishing Into a Stream

Now let's add some messages to our Stream. You can use nats pub to add messages, pass the --wait flag to see the publish ack being returned.

You can publish without waiting for acknowledgement:

nats pub ORDERS.scratch hello

But if you want to be sure your messages got to JetStream and were persisted you can make a request:

nats req ORDERS.scratch hello

13:45:03 Sending request on [ORDERS.scratch]
13:45:03 Received on [_INBOX.M8drJkd8O5otORAo0sMNkg.scHnSafY]: '+OK'

Keep checking the status of the Stream while doing this and you'll see its stored messages increase.

nats str info ORDERS

Information for Stream ORDERS
...
Statistics:

            Messages: 3
               Bytes: 147 B
            FirstSeq: 1
             LastSeq: 3
    Active Consumers: 0

After putting some throwaway data into the Stream, we can purge all the data out - while keeping the Stream active:

Deleting All Data

To delete all data in a stream use purge:

nats str purge ORDERS -f

...
State:

            Messages: 0
               Bytes: 0 B
            FirstSeq: 1,000,001
             LastSeq: 1,000,000
    Active Consumers: 0

Deleting A Message

A single message can be securely removed from the stream:

nats str rmm ORDERS 1 -f

Deleting Sets

Finally, for demonstration purposes, you can also delete the whole Stream and recreate it. Then we're ready for creating the Consumers:

nats str rm ORDERS -f
nats str add ORDERS --subjects "ORDERS.*" --ack --max-msgs=-1 --max-bytes=-1 --max-age=1y --storage file --retention limits --max-msg-size=-1 --discard old --dupe-window="0s" --replicas 1

Consumers

Messages are read or consumed from the Stream by Consumers. We support pull and push-based Consumers and the example scenario has both, let's walk through that.

Creating Pull-Based Consumers

The NEW and DISPATCH Consumers are pull-based, meaning the services consuming data from them have to ask the system for the next available message. This means you can easily scale your services up by adding more workers and the messages will get spread across the workers based on their availability.

Pull-based Consumers are created the same as push-based Consumers, you just don't specify a delivery target.

nats con ls ORDERS

No Consumers defined

We have no Consumers, lets add the NEW one:

I supply the --sample options on the CLI as this is not prompted for at present, everything else is prompted. The help in the CLI explains each:

nats con add --sample 100

? Select a Stream ORDERS
? Consumer name NEW
? Delivery target
? Start policy (all, last, 1h, msg sequence) all
? Filter Stream by subject (blank for all) ORDERS.received
? Maximum Allowed Deliveries 20
Information for Consumer ORDERS > NEW

Configuration:

        Durable Name: NEW
           Pull Mode: true
             Subject: ORDERS.received
         Deliver All: true
        Deliver Last: false
          Ack Policy: explicit
            Ack Wait: 30s
       Replay Policy: instant
  Maximum Deliveries: 20
       Sampling Rate: 100

State:

  Last Delivered Message: Consumer sequence: 1 Stream sequence: 1
    Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0
        Pending Messages: 0
    Redelivered Messages: 0

This is a pull-based Consumer (empty Delivery Target), it gets messages from the first available message and requires specific acknowledgement of each and every message.

It only received messages that originally entered the Stream on ORDERS.received. Remember the Stream subscribes to ORDERS.*, this lets us select a subset of messages from the Stream.

A Maximum Delivery limit of 20 is set, this means if the message is not acknowledged it will be retried but only up to this maximum total deliveries.

Again this can all be done in a single CLI call, lets make the DISPATCH Consumer:

nats con add ORDERS DISPATCH --filter ORDERS.processed --ack explicit --pull --deliver all --sample 100 --max-deliver 20

Additionally, one can store the configuration in a JSON file, the format of this is the same as $ nats con info ORDERS DISPATCH -j | jq .config:

nats con add ORDERS MONITOR --config monitor.json

Creating Push-Based Consumers

Our MONITOR Consumer is push-based, has no ack and will only get new messages and is not sampled:

nats con add

? Select a Stream ORDERS
? Consumer name MONITOR
? Delivery target monitor.ORDERS
? Start policy (all, last, 1h, msg sequence) last
? Acknowledgement policy none
? Replay policy instant
? Filter Stream by subject (blank for all)
? Maximum Allowed Deliveries -1
Information for Consumer ORDERS > MONITOR

Configuration:

      Durable Name: MONITOR
  Delivery Subject: monitor.ORDERS
       Deliver All: false
      Deliver Last: true
        Ack Policy: none
     Replay Policy: instant

State:

  Last Delivered Message: Consumer sequence: 1 Stream sequence: 3
    Acknowledgment floor: Consumer sequence: 0 Stream sequence: 2
        Pending Messages: 0
    Redelivered Messages: 0

Again you can do this with a single non-interactive command:

nats con add ORDERS MONITOR --ack none --target monitor.ORDERS --deliver last --replay instant --filter ''

Additionally one can store the configuration in a JSON file, the format of this is the same as $ nats con info ORDERS MONITOR -j | jq .config:

nats con add ORDERS --config monitor.json

Listing

You can get a quick list of all the Consumers for a specific Stream:

nats con ls ORDERS

Consumers for Stream ORDERS:

        DISPATCH
        MONITOR
        NEW

Querying

All details for a Consumer can be queried, lets first look at a pull-based Consumer:

$ nats con info ORDERS DISPATCH
Information for Consumer ORDERS > DISPATCH

Configuration:

      Durable Name: DISPATCH
         Pull Mode: true
           Subject: ORDERS.processed
       Deliver All: true
      Deliver Last: false
        Ack Policy: explicit
          Ack Wait: 30s
     Replay Policy: instant
     Sampling Rate: 100

State:

  Last Delivered Message: Consumer sequence: 1 Stream sequence: 1
    Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0
        Pending Messages: 0
    Redelivered Messages: 0

More details about the State section will be shown later when discussing the ack models in depth.

Stream vs Consumer sequence numbers

The two number are not directly related: the Stream sequence number is the pointer to the exact message, while the Consumer sequence number is an ever-increasing counter for consumer actions.

So for example a stream with 1 message in it would have stream sequence of 1, but if the consumer attempted 10 deliveries of that message consumer sequence would be 10 or 11.

Consuming Pull-Based Consumers

Pull-based Consumers require you to specifically ask for messages and ack them, typically you would do this with the client library Request() feature, but the nats utility has a helper:

First, we ensure we have a message:

nats pub ORDERS.processed "order 1"
nats pub ORDERS.processed "order 2"
nats pub ORDERS.processed "order 3"

We can now read them using nats:

nats con next ORDERS DISPATCH

--- received on ORDERS.processed
order 1

Acknowledged message

Consumer another one

nats con next ORDERS DISPATCH

--- received on ORDERS.processed
order 2

Acknowledged message

You can prevent ACKs by supplying --no-ack.

To do this from code you'd send a Request() to $JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH:

nats req '$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH' ''

Published [$JS.API.CONSUMER.MSG.NEXT.ORDERS.DISPATCH] : ''
Received [ORDERS.processed] : 'order 3'

Here nats req cannot ack, but in your code you'd respond to the received message with a nil payload as an Ack to JetStream.

Consuming Push-Based Consumers

Push-based Consumers will publish messages to a subject and anyone who subscribes to the subject will get them, they support different Acknowledgement models covered later, but here on the MONITOR Consumer we have no Acknowledgement.

nats con info ORDERS MONITOR

Output extract

...
  Delivery Subject: monitor.ORDERS
...

The Consumer is publishing to that subject, so let's listen there:

nats sub monitor.ORDERS

Listening on [monitor.ORDERS]
[#3] Received on [ORDERS.processed]: 'order 3'
[#4] Received on [ORDERS.processed]: 'order 4'

Note the subject here of the received message is reported as ORDERS.processed this helps you distinguish what you're seeing in a Stream covering a wildcard, or multiple subjects, subject space.

This Consumer needs no ack, so any new message into the ORDERS system will show up here in real-time.

Data Replication

Replication allows you to move data between streams in either a 1:1 mirror style or by multiplexing multiple source streams into a new stream. In future builds this will allow data to be replicated between accounts as well, ideal for sending data from a Leafnode into a central store.

Here we have 2 main streams - ORDERS and RETURNS - these streams are clustered across 3 nodes. These Streams have short retention periods and are memory based.

We create a ARCHIVE stream that has 2 sources set, the ARCHIVE will pull data from the sources into itself. This stream has a very long retention period and is file based and replicated across 3 nodes. Additional messages can be added to the ARCHIVE by sending to it directly.

Finally, we create a REPORT stream mirrored from ARCHIVE that is not clustered and retains data for a month. The REPORT Stream does not listen for any incoming messages, it can only consume data from ARCHIVE.

Mirrors

A mirror copies data from 1 other stream, as far as possible IDs and ordering will match exactly the source. A mirror does not listen on a subject for any data to be added. A mirror can filter by subject and the Start Sequence and Start Time can be set. A stream can only have 1 mirror and if it is a mirror it cannot also have any source.

Sources

A source is a stream where data is copied from, one stream can have multiple sources and will read data in from them all. The stream will also listen for messages on it's own subject. We can therefore not maintain absolute ordering, but data from 1 single source will be in the correct order but mixed in with other streams. You might also find the timestamps of streams can be older and newer mixed in together as a result.

A Stream with sources may also listen on subjects, but could have no listening subject. When using the nats CLI to create sourced streams use --subjects to supply subjects to listen on.

A source can have Start Time or Start Sequence and can filter by a subject.

Configuration

The ORDERS and RETURNS streams as normal, I will not show how to create them.

nats s report

Obtaining Stream stats

+---------+---------+-----------+----------+-------+------+---------+----------------------+
| Stream  | Storage | Consumers | Messages | Bytes | Lost | Deleted | Cluster              |
+---------+---------+-----------+----------+-------+------+---------+----------------------+
| ORDERS  | Memory  | 0         | 0        | 0 B   | 0    | 0       | n1-c2, n2-c2*, n3-c2 |
| RETURNS | Memory  | 0         | 0        | 0 B   | 0    | 0       | n1-c2*, n2-c2, n3-c2 |
+---------+---------+-----------+----------+-------+------+---------+----------------------+

We now add the ARCHIVE:

nats s add ARCHIVE --source ORDERS --source RETURNS

? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Message size limit -1
? Maximum message age limit -1
? Maximum individual message size -1
? Duplicate tracking time window 2m0s
? Allow message Roll-ups No
? Allow message deletion Yes
? Allow purging subjects or the entire stream Yes
? Replicas 1
? Adjust source "ORDERS" start Yes
? ORDERS Source Start Sequence 0
? ORDERS Source UTC Time Stamp (YYYY:MM:DD HH:MM:SS)
? ORDERS Source Filter source by subject
? Import "ORDERS" from a different JetStream domain No
? Import "ORDERS" from a different account No
? Adjust source "RETURNS" start No
? Import "RETURNS" from a different JetStream domain No
? Import "RETURNS" from a different account No
Stream ARCHIVE was created

Information for Stream ARCHIVE created 2022-01-21T11:49:52-08:00

Configuration:

     Acknowledgements: true
            Retention: File - Limits
             Replicas: 1
       Discard Policy: Old
     Duplicate Window: 2m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: unlimited
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited
              Sources: ORDERS
                       RETURNS


State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 0
              LastSeq: 0
     Active Consumers: 0

And we add the REPORT:

nats s add REPORT --mirror ARCHIVE

? Storage backend file
? Retention Policy Limits
? Discard Policy Old
? Stream Messages Limit -1
? Message size limit -1
? Maximum message age limit -1
? Maximum individual message size -1
? Allow message Roll-ups No
? Allow message deletion Yes
? Allow purging subjects or the entire stream Yes
? Replicas 1
? Adjust mirror start No
? Import mirror from a different JetStream domain No
? Import mirror from a different account No
Stream REPORT was created

Information for Stream REPORT created 2022-01-21T11:50:55-08:00

Configuration:

     Acknowledgements: true
            Retention: File - Limits
             Replicas: 1
       Discard Policy: Old
     Duplicate Window: 2m0s
    Allows Msg Delete: true
         Allows Purge: true
       Allows Rollups: false
     Maximum Messages: unlimited
        Maximum Bytes: unlimited
          Maximum Age: unlimited
 Maximum Message Size: unlimited
    Maximum Consumers: unlimited
               Mirror: ARCHIVE


State:

             Messages: 0
                Bytes: 0 B
             FirstSeq: 0
              LastSeq: 0
     Active Consumers: 0

When configured we'll see some additional information in a nats stream info output:

nats stream info ARCHIVE

Output extract

...
Source Information:

          Stream Name: ORDERS
                  Lag: 0
            Last Seen: 2m23s

          Stream Name: RETURNS
                  Lag: 0
            Last Seen: 2m15s
...

$ nats stream info REPORT
...
Mirror Information:

          Stream Name: ARCHIVE
                  Lag: 0
            Last Seen: 2m35s
...

Here the Lag is how far behind we were reported as being last time we saw a message.

We can confirm all our setup using a nats stream report:

nats s report

+--------------------------------------------------------------------------------------------------------+
|                                            Stream Report                                               |
+---------+---------+-------------+-----------+----------+-------+------+---------+----------------------+
| Stream  | Storage | Replication | Consumers | Messages | Bytes | Lost | Deleted | Cluster              |
+---------+---------+-------------+-----------+----------+-------+------+---------+----------------------+
| ARCHIVE | File    | Sourced     | 1         | 0        | 0 B   | 0    | 0       | n1-c2*, n2-c2, n3-c2 |
| ORDERS  | Memory  |             | 1         | 0        | 0 B   | 0    | 0       | n1-c2, n2-c2*, n3-c2 |
| REPORT  | File    | Mirror      | 0         | 0        | 0 B   | 0    | 0       | n1-c2*               |
| RETURNS | Memory  |             | 1         | 0        | 0 B   | 0    | 0       | n1-c2, n2-c2, n3-c2* |
+---------+---------+-------------+-----------+----------+-------+------+---------+----------------------+

+---------------------------------------------------------+
|                   Replication Report                    |
+---------+--------+---------------+--------+-----+-------+
| Stream  | Kind   | Source Stream | Active | Lag | Error |
+---------+--------+---------------+--------+-----+-------+
| ARCHIVE | Source | ORDERS        | never  | 0   |       |
| ARCHIVE | Source | RETURNS       | never  | 0   |       |
| REPORT  | Mirror | ARCHIVE       | never  | 0   |       |
+---------+--------+---------------+--------+-----+-------+

We then create some data in both ORDERS and RETURNS:

nats req ORDERS.new "ORDER {{Count}}" --count 100
nats req RETURNS.new "RETURN {{Count}}" --count 100

We can now see from a Stream Report that the data has been replicated:

nats s report --dot replication.dot

Obtaining Stream stats

+---------+---------+-----------+----------+---------+------+---------+----------------------+
| Stream  | Storage | Consumers | Messages | Bytes   | Lost | Deleted | Cluster              |
+---------+---------+-----------+----------+---------+------+---------+----------------------+
| ORDERS  | Memory  | 1         | 100      | 3.3 KiB | 0    | 0       | n1-c2, n2-c2*, n3-c2 |
| RETURNS | Memory  | 1         | 100      | 3.5 KiB | 0    | 0       | n1-c2*, n2-c2, n3-c2 |
| ARCHIVE | File    | 1         | 200      | 27 KiB  | 0    | 0       | n1-c2, n2-c2, n3-c2* |
| REPORT  | File    | 0         | 200      | 27 KiB  | 0    | 0       | n1-c2*               |
+---------+---------+-----------+----------+---------+------+---------+----------------------+

+---------------------------------------------------------+
|                   Replication Report                    |
+---------+--------+---------------+--------+-----+-------+
| Stream  | Kind   | Source Stream | Active | Lag | Error |
+---------+--------+---------------+--------+-----+-------+
| ARCHIVE | Source | ORDERS        | 14.48s | 0   |       |
| ARCHIVE | Source | RETURNS       | 9.83s  | 0   |       |
| REPORT  | Mirror | ARCHIVE       | 9.82s  | 0   |       |
+---------+--------+---------------+--------+-----+-------+

Here we also pass the --dot replication.dot argument that writes a GraphViz format map of the replication setup.

Disaster Recovery

In the event of unrecoverable JetStream message persistence on one (or more) server nodes, there are two recovery scenarios:

• Automatic recovery from intact quorum nodes
• Manual recovery from existing stream snapshots (backups)

警告

For R1 streams, data is persisted on one server node only. If that server node is unrecoverable then recovery from backup is the sole option.

Automatic Recovery

NATS will create replacement stream replicas automatically under the following conditions:

• Impacted stream is of replica configuration R3 (or greater)
• Remaining intact nodes (stream replicas) meet minimum RAFT quorum: floor(R/2) + 1
• Available node(s) in the stream's cluster for new replica(s)
• Impacted node(s) removed from the stream's domain RAFT Meta group (e.g. nats server cluster peer-remove)

Manual Recovery

Snapshots (also known as backups) can pro-actively be made of any stream regardless of replication configuration.

The backup includes (by default):

• Stream configuration and state
• Stream durable consumer configuration and state
• All message payload data including metadata like timestamps and headers

Backup

The nats stream backup CLI command is used to create snapshots of a stream and its durable consumers.

提示

As an account owner, if you wish to make a backup of ALL streams in your account, you may use nats account backup instead.

注意

Memory storage streams do not support snapshots. Only file-based storage streams can be backed up.

nats stream backup ORDERS '/data/js-backup/backup1'

Output

Starting backup of Stream "ORDERS" with 13 data blocks

2.4 MiB/s [====================================================================] 100%

Received 13 MiB bytes of compressed data in 3368 chunks for stream "ORDERS" in 1.223428188s, 813 MiB uncompressed

During a backup operation, the stream is placed in a status where it's configuration cannot change and no data will be evicted based on stream retention policies.

提示

Progress using the terminal bar can be disabled using --no-progress, it will then issue log lines instead.

Restore

An existing backup (as above) can be restored to the same or a new NATS server (or cluster) using the nats stream restore command.

提示

nats stream restore restores a single stream from one backup directory. To restore all streams at once, use nats account restore as described below.

nats stream restore '/data/js-backup/backup1'

Output

Starting restore of Stream "ORDERS" from file "/data/js-backup/backup1"

13 MiB/s [====================================================================] 100%

Restored stream "ORDERS" in 937.071149ms

Information for Stream ORDERS

Configuration:

             Subjects: ORDERS.>
...

Progress using the terminal bar can be disabled using --no-progress, it will then issue log lines instead.

Account-Level Backup and Restore

In environments where the nats CLI is used interactively to configure the server you do not have a desired state to recreate the server from. This is not the ideal way to administer the server, we recommend Configuration Management, but many will use this approach.

The nats account backup and nats account restore commands allow you to back up and restore all streams in an account at once, including their configuration, consumer state, and all message data.

Account Backup

nats account backup /data/js-backup

Output

Performing backup of all streams to /data/js-backup

    Streams: 3
       Size: 14 KiB
  Consumers: 2

Starting backup of Stream "EVENTS" with 0 B
Received 1.5 KiB compressed data in 2 chunks for stream "EVENTS" in 0s, 16 KiB uncompressed

Starting backup of Stream "ORDERS" with 55 B
Received 976 B compressed data in 2 chunks for stream "ORDERS" in 1ms, 9.5 KiB uncompressed

Starting backup of Stream "WORK" with 7.3 KiB
Received 7.3 KiB compressed data in 2 chunks for stream "WORK" in 0s, 30 KiB uncompressed

This creates a subdirectory per stream inside /data/js-backup, each containing the full stream snapshot (configuration, consumer state, and message data) in the same format as nats stream backup.

Available flags for nats account backup:

Account Restore

nats account restore /data/js-backup

Output

Restoring backup of all 3 streams in directory "/data/js-backup"

Starting restore of Stream "EVENTS" from file "/data/js-backup/EVENTS"
Restored stream "EVENTS" in 0s
...

Starting restore of Stream "ORDERS" from file "/data/js-backup/ORDERS"
Restored stream "ORDERS" in 1ms
...

Starting restore of Stream "WORK" from file "/data/js-backup/WORK"
Restored stream "WORK" in 0s
...

This restores all stream subdirectories found in /data/js-backup, including their full message data and consumer state.

Available flags for nats account restore:

注意

nats account restore will fail if a stream with the same name already exists. You must remove the existing stream before restoring from backup.

Encryption at Rest

Supported since NATS server version 2.3.0

TPM is supported on Windows since NATS Server version 2.11.0

注意

Note, that although encryption at rest by the NATS server is fully supported, we recommend using file system encryption where available.

File system encryption, in particular when provided by Cloud hosted services, is optimized for throughput, does not place a burden on the NATS server and removes the need for secret management from the NATS installation.

The NATS server can be configured to encrypt message blocks which includes message headers and payloads. Other metadata files are encrypted as well, such as the stream metadata file and consumer metadata files.

Two choices of ciphers are currently supported:

• chachapoly - ChaCha20-Poly1305
• aes - AES-GCM

Enabling encryption is done through the jetstream configuration block on the server.

jetstream : {
  cipher: chachapoly
  key : "6dYfBV0zzEkR3vxZCNjxmnVh/aIqgid1"
}

It is recommended to provide the encryption key through an environment variable at runtime, such as $JS_KEY, so it will not be persisted in a file.

jetstream : {
  cipher: chachapoly
  key: $JS_KEY
}

The variable can be exported in the environment or passed when the server starts up.

JS_KEY="mykey" nats-server -c js.conf

TPM (Windows only)

jetstream {
  store_dir: nats
  max_file_store: 10G
  tpm {
          keys_file: "keys"
          encryption_password: "pwd"
  }
}

Changing encryption settings

Enabling with existing data

Enabling encryption on a server with existing data is supported. Do note that existing unencrypted message blocks will not be re-encrypted, however any new blocks that are stored will be encrypted going forward.

If it is desired to encrypt the existing blocks, the stream can be backed up and restored (which decrypts on backup and then re-encrypts when restoring it).

Disabling or changing the key

If encryption was enabled on the server and the server is restarted with a different key or disabled all together, the server will fail to decrypt messages when attempting to load them from the store. If this happens, you’ll see log messages like the following:

Error decrypting our stream metafile: chacha20poly1305: message authentication failed

Note, that this will impact JetStream functionality, but the server will still support core NATS functionality.

Changing the cipher

It is possible to change the cipher, however the same key must be used. The server will properly encrypt new message blocks with the new cipher and decrypt existing messages blocks with the existing cipher.

Performance considerations

Performance considerations: As expected, encryption is likely to decrease performance, but by how much is hard to define. In some performance tests on a MacbookPro 2.8 GHz Intel Core i7 with SSD, we have observed as little as 1% decrease to more than 30%. In addition to CPU cycles required for encryption, the encrypted files may be larger, which results in more data being stored or read.

Managing JWT Security

If you are using the JWT model of authentication to secure your NATS infrastructure or implementing an Auth callout service, you can administer authentication and authorization without having to change the servers' configuration files.

You can use the nsc CLI tool to manage identities. Identities take the form of nkeys. Nkeys are a public-key signature system based on Ed25519 for the NATS ecosystem. The nkey identities are associated with NATS configuration in the form of a JSON Web Token (JWT). The JWT is digitally signed by the private key of an issuer forming a chain of trust. The nsc tool creates and manages these identities and allows you to deploy them to a JWT account server, which in turn makes the configurations available to nats-servers.

You can also use nk CLI tool and library to manage keys.

Creating, updating and managing JWTs programmatically

You can create, update and delete accounts and users programmatically using the following libraries:

• Golang: see NKEYS and JWT.
• Java: see NKey.java and JwtUtils.java

Integrating with your existing authentication/authorization system

You can integrate NATS with your existing authentication/authorization system or create your own custom authentication using the Auth callout.

Examples

See NATS by Example under "Authentication and Authorization" for JWT and Auth callout server implementation examples.

User JWTs

User creation in Golang

Account JWTs

Golang example from https://natsbyexample.com/examples/auth/nkeys-jwts/go

package main

import (
	"flag"
	"fmt"
	"log"

	"github.com/nats-io/jwt/v2"
	"github.com/nats-io/nkeys"
)

func main() {
	log.SetFlags(0)

	var (
		accountSeed  string
		operatorSeed string
		name         string
	)

	flag.StringVar(&operatorSeed, "operator", "", "Operator seed for creating an account.")
	flag.StringVar(&accountSeed, "account", "", "Account seed for creating a user.")
	flag.StringVar(&name, "name", "", "Account or user name to be created.")

	flag.Parse()

	if accountSeed != "" && operatorSeed != "" {
		log.Fatal("operator and account cannot both be provided")
	}

	var (
		jwt string
		err error
	)

	if operatorSeed != "" {
		jwt, err = createAccount(operatorSeed, name)
	} else if accountSeed != "" {
		jwt, err = createUser(accountSeed, name)
	} else {
		flag.PrintDefaults()
		return
	}
	if err != nil {
		log.Fatalf("error creating account JWT: %v", err)
	}

	fmt.Println(jwt)
}

func createAccount(operatorSeed, accountName string) (string, error) {
	akp, err := nkeys.CreateAccount()
	if err != nil {
		return "", fmt.Errorf("unable to create account using nkeys: %w", err)
	}

	apub, err := akp.PublicKey()
	if err != nil {
		return "", fmt.Errorf("unable to retrieve public key: %w", err)
	}

	ac := jwt.NewAccountClaims(apub)
	ac.Name = accountName

	// Load operator key pair
	okp, err := nkeys.FromSeed([]byte(operatorSeed))
	if err != nil {
		return "", fmt.Errorf("unable to create operator key pair from seed: %w", err)
	}

	// Sign the account claims and convert it into a JWT string
	ajwt, err := ac.Encode(okp)
	if err != nil {
		return "", fmt.Errorf("unable to sign the claims: %w", err)
	}

	return ajwt, nil
}

func createUser(accountSeed, userName string) (string, error) {
	ukp, err := nkeys.CreateUser()
	if err != nil {
		return "", fmt.Errorf("unable to create user using nkeys: %w", err)
	}

	upub, err := ukp.PublicKey()
	if err != nil {
		return "", fmt.Errorf("unable to retrieve public key: %w", err)
	}

	uc := jwt.NewUserClaims(upub)
	uc.Name = userName

	// Load account key pair
	akp, err := nkeys.FromSeed([]byte(accountSeed))
	if err != nil {
		return "", fmt.Errorf("unable to create account key pair from seed: %w", err)
	}

	// Sign the user claims and convert it into a JWT string
	ujwt, err := uc.Encode(akp)
	if err != nil {
		return "", fmt.Errorf("unable to sign the claims: %w", err)
	}

	return ujwt, nil
}

Notes

You can see the key (and any signing keys) of your operator using nsc list keys --show-seeds, you should use a 'signing key' to create the account JWTs (as singing keys can be revoked/rotated easily)

To delete accounts use the "$SYS.REQ.CLAIMS.DELETE" (see reference) and make sure to enable JWT deletion in your nats-server resolver (config allow_delete: true in the resolver stanza of the server configuration).

The system is just like any other account, the only difference is that it is listed as system account in the operator's JWT (and the server config).

In Depth JWT Guide

This document provides a step by step deep dive into JWT usage within NATS. Starting with related concepts, it will introduce JWTs and how they can be used in NATS. This will NOT list every JWT/nsc option, but will focus on the important options and concepts.

• Concepts
  • What are Accounts?
    • Key Takeaways
  • What are NKEYs?
    • Key Takeaways
• JSON Web Tokens (JWT)
  • Motivation for JWT
    • Key Takeaways
  • Decentralized Authentication/Authorization using JWT
    • Key Takeaways
  • NATS JWT Hierarchy
    • Decentralized Chain of Trust
      • Obtain an Account JWT
      • JWT and Chain of Trust Verification
      • Obtain a User JWT - Client Connect
    • Key Takeaways
  • Deployment Models Enabled by Chain of Trust
    • Key Takeaways
• Accounts Re-visited
  • Key Takeaways
• Tooling And Key Management
  • nsc
    • Environment
      • Backup
        • NKEYS store directory
        • JWT store directory
      • Names in JWT
    • Setup an Operator
      • Create/Edit Operator - Operator Environment - All Deployment modes
      • Import Operator - Non Operator/Administrator Environment - Decentralized/Self Service Deployment Modes
      • Import Operator - Self Service Deployment Modes
    • Setup an Account
      • Create/Edit Account - All Environments - All Deployment modes
      • Export Account - Non Operator/Administrator Environment - Decentralized Deployment Modes
      • Export Account - Non Operator/Administrator Environment - Self Service Deployment Modes
    • Publicize an Account with Push - Operator Environment/Environment with push permissions - All Deployment Modes
      • nats-resolver setup and push example - Operator Environment/Environment with push permissions - All Deployment Modes
    • Setup User
      • Create/Edit Account - All Environments - All Deployment modes
  • Automated sign up services - JWT and NKEY libraries
    • Simple user creation
      • Create user NKEY
      • Create user JWT
      • Distributed User Creation
    • User creation using NATS
      • Straight forward Setup
      • Account-based Setup
    • Stamping JWT in languages other than Go
  • System Account
    • Event Subjects
    • Service Subjects
      • Subjects always available
      • Subjects available when using NATS-based resolver
      • Old Subjects
    • Leaf Node Connections - Outgoing
      • Non-Operator Mode
      • Operator Mode
  • Connecting Accounts
    • Exports
    • Imports
      • Import Subjects
      • Import Remapping
      • Visualizing Export/Import Relationships
  • Managing Keys
    • Protect Identity NKEYs
    • Reissue Identity NKEYs
      • Operator
      • Account
    • Revocations
      • User
      • Activations
      • Accounts
      • Signing keys

To exercise listed examples please have the following installed:

• nats-server: https://github.com/nats-io/nats-server
• nats (cli): https://github.com/nats-io/natscli
• nk (cli & library): https://github.com/nats-io/nkeys
• nsc (cli): https://github.com/nats-io/nsc
• jwt (library): https://github.com/nats-io/jwt

To install nats-server, nats, nk, nsc:

export GO111MODULE=on
go install github.com/nats-io/nats-server/v2@latest
go install github.com/nats-io/natscli/nats@latest
go install github.com/nats-io/nkeys/nk@latest
go install github.com/nats-io/nsc/v2@latest

To practice the examples below:

Save the configuration below in a file named say server.conf, and start nats server via:

nats-server -c server.conf

So that the server started. Later when we change the configuration we can do a reload like this:

nats-server --signal reload

Concepts

What are Accounts?

Accounts are the NATS isolation context.

accounts: {
    A: {
        users: [{user: a, password: a}]
    },
    B: {
        users: [{user: b, password: b}]
    },
}

Messages published in one account won't be received in another.

Listen for any message on account a:

nats -s nats://a:a@localhost:4222 sub ">"

Publish a message from account b:

nats -s nats://b:b@localhost:4222 pub "foo" "user b"

Note that you do not see this message received by your subscriber.

Now publish a messages from account a:

nats -s nats://a:a@localhost:4222 pub "foo" "user a"

This time the message is received by the subscriber:

17:57:06 [#1] Received on "foo"
user a

The above example shows no message flow between user a associated with account A and user b in account B. Messages are delivered only within the same account. That is, unless you explicitly define it.

Below is a similar example, this time with messages crossing explicit account boundaries.

accounts: {
    A: {
        users: [{user: a, password: a}]
        imports: [{stream: {account: B, subject: "foo"}}]
    },
    B: {
        users: [{user: b, password: b}]
        exports: [{stream: "foo"}]
    },
}

Modify server.conf and run nats-server --signal reload

Subscribe to everything as user 'a'

nats -s nats://a:a@localhost:4222 sub ">"

Publish on 'foo' as user 'b':

nats -s nats://b:b@localhost:4222 pub "foo" "user b"

This time the message is received by the subscriber:

18:28:25 [#1] Received on "foo"
user b

Accounts are a lot more powerful than what has been demonstrated here. Take a look at the complete documentation of accounts and the users associated with them. All of this is in a plain NATS config file. (Copy the above config and try it using this command: nats-server -c <filename>) In order to make any changes, every participating nats-server config file in the same security domain has to change. This configuration is typically controlled by one organization or the administrator.

Key Takeaways

• Accounts are isolated from each other.
• One can selectively combine accounts,
• Need to modify a config file to add/remove/modify accounts and users,
• The config file can be applied to take effect via nats-server --signal reload.

What are NKEYs?

NKEYs are decorated, Base32 encoded, CRC16 check-summed, Ed25519 keys.

Ed25519 is:

• a public key signature system. (can sign and verify signatures)
• resistant to side channel attacks (no conditional jumps in algorithm)

NATS server can be configured with public NKEYs as user (identities). When a client connects the nats-server sends a challenge for the client to sign in order to prove it is in possession of the corresponding private key. The nats-server then verifies the signed challenge. Unlike with a password based scheme, the secret never left the client.

To assist with knowing what type of key one is looking at, in config or logs, the keys are decorated as follows:

• Public Keys, have a one byte prefix: O, A, U for various types, O for operator, A for account, and U meaning user.
• Private Keys, have a two byte prefix SO, SA, SU. S stands for seed. The remainders(O, A and U) are the same meaning as in public keys.

NKEYs are generated as follows:

nk -gen user -pubout > a.nk

To view the key:

cat a.nk

SUAAEZYNLTEA2MDTG7L5X7QODZXYHPOI2LT2KH5I4GD6YVP24SE766EGPA
UC435ZYS52HF72E2VMQF4GO6CUJOCHDUUPEBU7XDXW5AQLIC6JZ46PO5

Create another key:

nk -gen user -pubout > b.nk

View the key:

cat b.nk

SUANS4XLL5NWBTM57GSVHLN4TMFW55WGGWNI5YXXSIOYFJQYFVNHJK5GFY
UARZVI6JAV7YMJTPRANXANOOW4K3ZCD45NYP6S7C7XKCBHPVN2TFZ7ZC

Replacing the user/password with NKEY in account config example:

accounts: {
    A: {
        users: [{nkey:UC435ZYS52HF72E2VMQF4GO6CUJOCHDUUPEBU7XDXW5AQLIC6JZ46PO5}]
        imports: [{stream: {account: B, subject: "foo"}}]
    },
    B: {
        users: [{nkey:UARZVI6JAV7YMJTPRANXANOOW4K3ZCD45NYP6S7C7XKCBHPVN2TFZ7ZC}]
        exports: [{stream: "foo"}]
    },
}