Release Notes What's New! The NATS.io team is continually working to bring you features that enhance your NATS experience. Below, you will find summaries of new NATS implementations. Release notes for the latest patch releases are available on GitHub Releases Roadmap for future releases See https://nats.io/about/#roadmap Server release v2.14.0 Check out the: Upgrade guide Release notes Server release v2.12.0 Check out the: Upgrade guide Release notes Server release v2.11.0 Check out the: Upgrade guide Release notes Server release v2.10.0 Check out the: Upgrade guide Podcast EP06: The journey and features of the NATS.io 2.10 release Release notes Server release v2.9.0 Please check out the announcement post on the blog and the detailed release notes in the server repo. Server release v2.8.0 LeafNode Support for a min_version in the leafnodes{} that would reject servers with a lower version. Note that this would work only for servers that are v2.8.0 and above. Monitoring Server version in monitoring landing page. Logging to /healthz endpoint when failure occurs. MQTT and Websocket blocks in the /varz endpoint. JetStream Consumer check added to healthz endpoint. Max stream bytes checks. Ability to limit a consumer's MaxAckPending value. Allow streams and consumers to migrate between clusters. This feature is considered "beta". New unique_tag option in jetstream{} configuration block to prevent placing a stream in the same availability zone twice. Stream Alternates field in StreamInfo response. They provide a priority list of mirrors and the source in relation to where the request originated. Deterministic subject tokens to partition mapping. For full release information, see links below; Release notes 2.8.0 Full list of Changes 2.7.4...2.8.0 Server release v2.7.0 Notice for JetStream Users See important note if using LeafNode regarding domains. Configuration Ability to configure account limits ( max_connections, max_subscriptions, max_payload, max_leafnodes) in server configuration file. JetStream Overflow placement for streams. A stream can now be placed in the closest cluster from the origin request if it can be placed there. Support for ephemeral Pull consumers (client libraries will need to be updated to allow those). New consumer configuration options For Pull Consumers: MaxRequestBatch to limit the batch size any client can request MaxRequestExpires to limit the expiration any client can request For ephemeral consumers: InactiveThreshold duration that instructs the server to cleanup ephemeral consumers that are inactive for that long. Ability to configure max_file_store and max_memory_store in the jetstream{} block as strings with the following suffixes K, M, G and T, for instance: max_file_store: "256M". Support for the JWT field MaxBytesRequired, which defines a per-account maximum bytes for assets. MQTT Support for websocket protocol. MQTT clients must connect to the opened websocket port and add /mqtt to the URL path. TLS Ability to rate-limit the clients connections by adding the connection_rate_limit: in the tls{} top-level block. For full release information, see links below; Release notes 2.7.0 Full list of Changes 2.6.6...2.7.0 Server release v2.6.0 Notice for JetStream Users See important note if upgrading from a version prior to NATS Server v2.4.0. Notice for MQTT Users See important notes if upgrading from a version prior to v2.5.0. Monitoring JetStream's reserved memory and memory used from accounts with reservations in /jsz and /varz endpoints. Hardened systemd service. For full release information, see links below; Release notes 2.6.0 Full list of Changes 2.5.0...2.6.0 Server release v2.5.0 Notice for JetStream Users See important note if upgrading from a version prior to NATS Server v2.4.0. MQTT/Monitoring MQTTClient in the /connz connections report and system events CONNECT and DISCONNECT. Ability to select on mqtt_client. MQTT Improvement Sessions are now all stored inside a single stream, as opposed to individual streams, reducing resources usage. MQTT Update Due to the aforementioned improvement described above, when an MQTT client connects for the first time after an upgrade to this server version, the server will migrate all individual $MQTT_sess_ streams to a new $MQTT_sess stream for the user's account. For full release information, see links below; Release notes 2.5.0 Full list of Changes 2.4.0...2.5.0 Server release v2.4.0 Notice for JetStream Users With the latest release of the NATS server, we have fixed bugs around queue subscriptions and have restricted undesired behavior that could be confusing or introduce data loss by unintended/undefined behavior of client applications. If you are using queue subscriptions on a JetStream Push Consumer or have created multiple push subscriptions on the same consumer, you may be affected and need to upgrade your client version along with the server version. We’ve detailed the behavior with different client versions below. With a NATS Server prior to v2.4.0 and client libraries prior to these versions: NATS C client v3.1.0, Go client v1.12.0, Java client 2.12.0-SNAPSHOT, NATS.js v2.2.0, NATS.ws v1.3.0, NATS.deno v1.2.0, NATS .NET 0.14.0-pre2: It was possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is not correct since each instance will receive the same copy of a message and acknowledgment is therefore meaningless since the first instance to acknowledge the message will prevent other instances to control if/when a message should be acknowledged. Similar to the first issue, it was possible to create many different queue groups for one single JetStream consumer. For queue subscriptions, if no consumer nor durable name was provided, the libraries would create ephemeral JetStream consumers, which meant that each member of the same group would receive the same message as the other members, which was not the expected behavior. Users assumed that 2 members subscribing to “foo” with the queue group named “bar” would load-balance the consumption of messages from the stream/consumer. It was possible to create a queue subscription on a JetStream consumer configured with heartbeat and/or flow control. This does not make sense because by definition, queue members would receive some (randomly distributed) messages, so the library would think that heartbeats are missed, and flow control would also be disrupted. If above client libraries are not updated to the latest but the NATS Server is upgraded to v2.4.0: It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. Since the check is performed by the library (with the help of a new field called PushBound in the consumer information object set by the server), this misbehavior is still possible. Queue subscriptions will not receive any message. This is because the server now has a new field DeliverGroup in the consumer configuration, which won’t be set for existing JetStream consumers and by the older libraries, and detects interest (and starts delivering) only when a subscription on the deliver subject for a queue subscription matching the “deliver group” name is found. Since the JetStream consumer is thought to be a non-deliver-group consumer, the opposite happens: the server detects a core NATS queue subscription on the “deliver subject”, therefore does not trigger delivery on the JetStream consumer’s “deliver subject”. The 2 other issues are still present because those checks are done in the updated libraries. If the above client libraries are updated to the latest version, but the NATS Server is still to version prior to v2.4.0 (that is, up to v2.3.4): It is still possible to create multiple non-queue subscription instances for the same JetStream durable consumer. This is because the JetStream consumer’s information retrieved by the library will not have the PushBound boolean set by the server, therefore will not be able to alert the user that they are trying to create multiple subscription instances for the same JetStream consumer. Queue subscriptions will fail because the consumer information returned will not contain the DeliverGroup field. The error will be likely to the effect that the user tries to create a queue subscription to a non-queue JetStream consumer. Note that if the application creates a queue subscription for a non-yet created JetStream consumer, then this call will succeed, however, adding new members or restarting the application with the now existing JetStream consumer will fail. Creating queue subscriptions without a named consumer/durable will now result in the library using the queue name as the durable name. Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message. For completeness, using the latest client libraries and NATS Server v2.4.0: Trying to start multiple non-queue subscriptions instances for the same JetStream consumer will now return an error to the effect that the user is trying to create a “duplicate subscription”. That is, there is already an active subscription on that JetStream consumer. It is now only possible to create a queue group for a JetStream consumer created for that group. The DeliverGroup field will be set by the library or need to be provided when creating the consumer externally. Trying to create a queue subscription without a durable nor consumer name results in the library creating/using the queue group as the JetStream consumer’s durable name. Trying to create a queue subscription with a consumer configuration that has heartbeat and/or flow control will now return an error message. Note that if the server v2.4.0 recovers existing JetStream consumers that were created prior to v2.4.0 (and with older libraries), none of them will have a DeliverGroup, so none of them can be used for queue subscriptions. They will have to be recreated. JetStream Domain to the content of a PubAck protocol PushBound boolean in ConsumerInfo to indicate that a push consumer is already bound to an active subscription DeliverGroup string in ConsumerConfig to specify which deliver group (or queue group name) the consumer is created for Warning log statement in situations where catchup for a stream resulted in an error Monitoring The ability for normal accounts to access scoped connz information Misc Operator option resolver_pinned_accounts to ensure users are signed by certain accounts For full release information, see links below; Release notes 2.4.0 Full list of Changes 2.3.4...2.4.0 Server release v2.3.0 OCSP support JetStream Richer API errors. JetStream errors now contain an ErrCode that uniquely describes the error. Ability to send more advanced Stream purge requests that can purge all messages for a specific subject Stream can now be configured with a per-subject message limit Encryption of JetStream data at rest For full release information, see links below; Release notes 2.3.0 Full list of Changes 2.2.6...2.3.0 Server release v2.2.0 See NATS 2.2 for new features. Server release v2.1.7 Monitoring Endpoints Available via System Services Monitoring endpoints as listed in the table below are accessible as system services using the following subject pattern: $SYS.REQ.SERVER.. (request server monitoring endpoint corresponding to endpoint name.) $SYS.REQ.SERVER.PING. (from all server request server monitoring endpoint corresponding to endpoint name - will return multiple messages) For more information on monitoring endpoints see NATS Server Configurations System Events. Addition of no_auth_user Configuration Configuration of no_auth_user allows you to refer to a configured user/account when no credentials are provided. For more information and examples, see Securing NATS For full release information, see links below; Release notes 2.1.7 Full list of Changes 2.1.6...2.1.7 Server release v2.1.6 TLS Configuration for Account Resolver This release adds the ability to specify TLS configuration for the account resolver. resolver_tls { cert_file: ... key_file: ... ca_file: ... } Additional Trace & Debug Verbosity Options trace_verbose and command line parameters -VV and -DVV added. See NATS Logging Configuration Subscription Details in Monitoring Endpoints We've added the option to include subscription details in monitoring endpoints /routez and /connz. For instance /connz?subs=detail will now return not only the subjects of the subscription, but the queue name (if applicable) and some other details. Release notes 2.1.6 Full list of Changes 2.1.4...2.1.6 Server release v2.1.4 Log Rotation NATS introduces logfile_size_limit allowing auto-rotation of log files when the size is greater than the configured limit set in logfile_size_limit as a number of bytes. You can provide the size with units, such as MB, GB, etc. The backup files will have the same name as the original log file with the suffix .yyyy.mm.dd.hh.mm.ss.micros. For more information see Configuring Logging in the NATS Server Configuration section. Release notes 2.1.4 Full list of Changes 2.1.2...2.1.4 Server release v2.1.2 Queue Permissions Queue Permissions allow you to express authorization for queue groups. As queue groups are integral to implementing horizontally scalable microservices, control of who is allowed to join a specific queue group is important to the overall security model. Original PR - https://github.com/nats-io/nats-server/pull/1143 More information on Queue Permissions can be found in the Developing with NATS section. Server release v2.1.0 Service Latency Tracking As services and service mesh functionality has become prominent, we have been looking at ways to make running scalable services on NATS.io a great experience. One area we have been looking at is observability. With publish/subscribe systems, everything is inherently observable, however we realized it was not as simple as it could be. We wanted the ability to transparently add service latency tracking to any given service with no changes to the application. We also realized that global systems, such as those NATS.io can support, needed something more than a single metric. The solution was to allow any sampling rate to be attached to an exported service, with a delivery subject for all collected metrics. We collect metrics that show the requestor’s view of latency, the responder’s view of latency and the NATS subsystem itself, even when requestor and responder are in different parts of the world and connected to different servers in a NATS supercluster. Release notes 2.1.0 Full list of Changes 2.0.4...2.1.0 Server release v2.0.4 Response Only Permissions For services, the authorization for responding to requests usually included wildcards for _INBOX.> and possibly $GR.> with a supercluster for sending responses. What we really wanted was the ability to allow a service responder to only respond to the reply subject it was sent. Response Types Exported Services were originally tied to a single response. We added the type for the service response and now support singletons (default), streams and chunked. Stream responses represent multiple response messages, chunked represents a single response that may have to be broken up into multiple messages. Release notes 2.0.4 Full list of Changes 2.0.2...2.0.4 NATS 2.14 This guide is tailored for existing NATS users upgrading from NATS version v2.12.x. This will read as a summary with links to specific documentation pages to learn more about the feature or improvement. Features Streams Fast batch publish: The AllowBatchPublish stream configuration option allows for high throughput and flow controlled publishing into a stream. This includes support for replicated and non-replicated streams, as well as doing per-message consistency checks without intermediate staging of messages (like with atomic batch publish). More information is available in ADR-50. Recurring schedules: The AllowMsgSchedules stream configuration option now also allows the usage of recurring schedules, either based on a simple interval, or with Cron. More information is available in ADR-51. Scheduled subject sampling: The AllowMsgSchedules stream configuration option now also allows to source the data of the last message matching the subject in the scheduled message. Useful for downsampling data on an interval. More information is available in ADR-51. Reliable WorkQueue and Interest mirroring/sourcing: Sourcing or mirroring from a WorkQueue or Interest retention stream is now supported. A durable consumer, as opposed to an ephemeral one, will automatically be used to perform the async replication. A new ack policy of AckFlowControl is used to acknowledge messages after they were persisted, based on flow control. More information is available in ADR-60, see also the upgrade considerations below. Consumers Consumer reset API: Consumer delivery state can now be reset back to the acknowledgement floor, or to an arbitrary sequence (while still respecting start sequences etc). The consumer state after reset equals what would otherwise be a consumer delete and recreate at a specific starting sequence. More information is available in ADR-60. Operations Leafnode remote config reload: The leaf node remotes section can now be added and removed via a configuration reload, without requiring a server restart. Filestore I/O error handling: Previously, not all filestore I/O errors were properly handled, allowing the stream and server to continue to run. These errors will now be surfaced in the logs and show up in the health check, as well as freeze the stream to prevent further updates (see upgrade considerations below). Raft overrun protection: The server now recognizes if its Raft layer is being overrun by proposals, bounding the number of memory and disk resources the server is allowed to use during such an event (see upgrade considerations below). Improvements Deduplication changes when using stream sourcing: Streams with sources now allow deduplication to be disabled. Additionally, sourcing streams can now perform deduplication when fanning in multiple sources. Atomic batch publish, EOB commit support: Atomic batches can now be committed through an EOB (End of Batch) message without persisting this final message. This is also supported when using the new fast batch publish. Scheduled subject rollups: The Nats-Schedule-Rollup header can now be used to place a rollup on the scheduled message, similar to the Nats-Schedule-TTL header. Feature flags: The server now supports a feature_flags field in the config which allows users to opt-in or opt-out to specific fixes or improvements prior to them becoming the default in a future release. Domain-aware acknowledgement and flow control subjects: The server now supports both v1 and v2 of the consumer acknowledgement and flow control subjects. More information is available in ADR-15. Traceparent header: The traceparent header is no longer modified by the message tracing. Asynchronous stream state snapshots for replicated streams: Allows stream state snapshots to be taken and written without pausing stream processing, improving tail latencies. This is particularly impactful in cases where the stream has a large number of interior deletes. Upgrade considerations Reliable WorkQueue and Interest mirroring/sourcing Version 2.14 adds support for sourcing and mirroring from WorkQueue or Interest streams. During an upgrade or downgrade, the server could temporarily log the following message: [WRN] Invalid JetStream request '$G > $JS.API.CONSUMER.CREATE.O': json: unknown field "sourcing" This means that an upgraded server tried to create the ‘new-style’ sourcing consumer but the older target server didn’t recognize it and logged a warning. The upgraded server will automatically respond to this error and will send the request again using the ‘old-style’ sourcing consumer. It is expected that these logs could temporarily occur during upgrades/downgrades, but these should resolve once the whole system is upgraded. Domain-aware acknowledgement and flow control subjects The server now supports both v1 and v2 acknowledgement and flow control reply subjects documented in ADR-15. The v2 format includes a domain and account hash to deconflict stream and consumer names across domains and accounts: v1: $JS.ACK....... v2: $JS.ACK......... While both v1 and v2 formats will be supported starting from 2.14, v1 remains the default. However, in version 2.15 the default will change to be the v2 format. Users that have defined account imports/exports or subject permissions containing the $JS.ACK..> or $JS.FC..> (or more granular) subjects will be required to update their ACLs and/or account imports/exports before the 2.15 release to allow the same stream and consumer names to be used in different domains or accounts without them conflicting with each other. If you have not defined such account imports/exports or subject permissions, for example if you use JetStream only within a single account, or you defined them as the “catch-all wildcard” $JS.ACK.> or $JS.FC.> then you will not need to make any changes. The default will change in version 2.15 and there should be no impact. To ease the migration path, the server now supports feature flags to test and enable this at your own convenience. Not specifying the feature flag means “use the server default” which for 2.14 will be the v1 format. Setting it to true will use the v2 format (also the v1 format will still be supported) and setting it to false will use the v1 format but still support v2. feature_flags { js_ack_fc_v2: true } Filestore I/O error handling Previously, not all filestore I/O errors were appropriately handled, allowing the stream and server to continue to run. In 2.14, these errors are surfaced: an affected stream freezes, logs the error, and reports an unhealthy state in health checks. Other streams on the same server remain unaffected, and for replicated streams another replica picks up the work transparently. This change introduces a new operational condition to watch for: if I/O errors are encountered, each stream affected by it will stop making progress. NATS core traffic is not affected by I/O errors, so the server keeps functioning, but will require a restart to recover from these I/O issues. This condition can be observed by the health check failing and reporting an I/O related error, the error message will contain write error and which error was encountered. Raft overrun protection A new back-pressure mechanism in the Raft consensus layer prevents unbounded memory growth in overloaded clusters. In prior versions, there could be situations where entries would be written to the Raft write-ahead log faster than they could be committed and applied, resulting in increased memory and disk usage. In 2.14, this condition is recognized and gives the relevant servers room to catch up. Leaders that detect they are falling behind step down so that a healthier peer can take over. If a majority of peers are equally overloaded, the system remains in this degraded state. The protection is a safety net for transient overload, not a substitute for adequate capacity, but it will allow the system to continue in a degraded but functional mode instead of allowing the system to become overloaded further. Downgrade considerations Reliable WorkQueue and Interest mirroring/sourcing The same consideration as mentioned above under the upgrade considerations holds here as well. The Invalid JetStream request log line may be observed during a downgrade. Additionally, stream sourcing or mirroring might temporarily be unable to function until all servers are downgraded to the 2.12 version. If you’ve started using stream sourcing or mirroring on WorkQueue or Interest streams after upgrading, this will still seem to function but will operate under the less reliable ephemeral consumer mode. Additionally, any durable consumers created using the new configuration of AckFlowControl will be marked as “offline” and will not be usable until upgraded back to 2.14. Feature flags If you have defined the feature_flags field in your server config, you’ll need to remove this prior to downgrading, since prior server versions will not recognize this field. NATS 2.12 This guide is tailored for existing NATS users upgrading from NATS version v2.11.x. This will read as a summary with links to specific documentation pages to learn more about the feature or improvement. Features Streams Atomic batch publish: The AllowAtomicPublish stream configuration option allows to atomically publish N messages into a stream. This includes support for replicated and non-replicated streams, as well as doing per-message consistency checks prior to committing the batch. More information is available in ADR-50. Distributed Counter CRDT: The AllowMsgCounter stream configuration option allows increment/decrement counter semantics on a stream. These counter streams can also be mirrored or aggregated through stream mirroring and sourcing. More information is available in ADR-49 Delayed Message Scheduling: The AllowMsgSchedules stream configuration option allows the scheduling of messages. Users can use this feature for delayed publishing/scheduling of messages. More information is available in ADR-51 Consumers Prioritized pull consumer policy: In addition to the consumer policies like overflow or client pinning, a new prioritized policy has been added. In contrast with the overflow policy, this allows a consumer to receive messages sooner instead of delaying failover, but at the cost of potentially flip-flopping work between clients. More information is available in ADR-42 Operations Server metadata: Similar to server_tags which contains a set of tags describing the server, server_metadata is a map containing string keys and values describing metadata of the server. Promoting mirrors: A stream that’s mirroring can now be promoted to be the primary, enabling new disaster recovery methodology. The current primary stream should be deleted or have its configured subjects removed prior to promoting mirrors, before configuring the promoted mirrors to start listening on those subjects. Exponential backoff on route and gateway connections: Cluster routes and gateways can now use exponential backoff on reconnection attempts by setting connect_backoff. If true, will start exponential backoff at 1 second up to 30 seconds. This can slow down the speed of reconnection but significantly reduces the amount of DNS queries and general connection attempts during server restarts or outages. Offline assets: When downgrading to an older version, the server can now recognize new features were used and puts the stream and/or consumer into an unsupported/offline mode. For more information, read also the downgrade considerations. More information is available in ADR-44 Stream/consumer scaleup and reset disk/state protection: The server now has better protections against leader elections based on empty state. This also improves reliability of replicated in-memory streams. Usually a quorum of servers needs to be online and contain data. Now all but one server can be restarted and the in-memory stream’s data can reliably be caught back up. However, during such a scenario all servers involved with replication of that stream will need to be available, not just what’s needed for quorum. This lets the servers decide the best course of action to preserve all data. Improvements Async stream flushing: Replicated streams will now asynchronously flush data to the underlying store on disk, resulting in a significant improvement in performance. Writes to a replicated stream are still persisted synchronously in the Raft log prior to committing them, so the improved performance has no downsides with respect to consistency. Elastic pointers in the filestore: File-based streams now use elastic pointers for its write-through caches. This allows the server to better respond during garbage collection, these caches can be evicted early to avoid out-of-memory conditions (see upgrade considerations below). Use cipher suites from crypto/tls: New cipher suites are now automatically added. Additionally, insecure cipher suites are disabled by default, but can be allowed when enabling allow_insecure_cipher_suites. System events for the $G account: The global account ( $G) will now also produce system events, such as connect and disconnect events. GOMAXPROCS and GOMEMLIMIT in server stats: The server stats already contained the CPU and memory usage of the server but now also contains the effective Go limits. New subject transforms: partition(n) and random(n): In addition to partition(n, …) which allows to determine a partition number based on tokens at specified indices, partition(n) and random(n) are convenience functions to create a partition or random number up to n based on the whole subject. Account name and user logging: Any logging related to a client connection, for example when reaching maximum connections or for authentication errors, will now include the account name and user of that client connection. Logging improvements: Any logging related to a client connection now includes the account and user name. Connection closed logging now includes the remote server name. Isolated leaf node property: In a large deployment with lots of leaf nodes, propagating east-west interest can result in a lot of traffic, which is wasted if leaf nodes don't need to be able to publish/subscribe to each other directly. Instead of the workaround of setting the cluster name of those leaf nodes to be the same, the isolate_leafnode_interest property can now be used. Disable leaf node connection through config reload: This allows disabling a remote leaf node using configuration reload, when using disabled: true. If changed from false to true, a solicited leaf node will be disconnected and will not reconnect. If changed from true to false, the leafnode will be solicited again. Upgrade Considerations Memory usage With the new elastic pointers in the filestore, it is expected that a NATS Server running 2.12 may show a different memory usage pattern to before. In some systems this may result in lower resident set size (RSS) reported, in others it may result in higher, depending on the number of assets and publish/access patterns. For the first time, the server will be able to respond to memory pressure by freeing filestore caches on demand and returning the memory to the operating system. This reduces the chance that sudden spikes in utilisation will result in an out-of-memory (OOM) kill. However, this means that the server can more optimistically retain caches in memory when available resources allow in order to facilitate improved read access times. This behaviour is largely controlled by the GC thresholds as set by the GOMEMLIMIT environment variable. You may wish to tune this value in your environment based on available system memory, or in the case of Kubernetes environments, memory reservations. Strict JetStream API Starting from version v2.11, the server would start logging the following statement if an invalid JetStream request was received: [WRN] Invalid JetStream request '$G > $JS.API.STREAM.CREATE.test-stream': json: unknown field "unknown" Starting from version v2.12, the server will not only log, but also return an error to the client as “strict mode” is now enabled by default. This means invalid JetStream requests will be rejected by default. If the above log message is observed, please make sure that the application or client is sending correct requests to the server and that NATS client libraries are up-to-date. Strict mode can be temporarily disabled in the server configuration, allowing you more time to fix the issue: jetstream { strict: false } Downgrade Considerations Stream state When downgrading from v2.12 to v2.11, the stream state files on disk will be rebuilt due to a change in the format of these files in v2.12. This requires re-scanning all stream message blocks, which may use higher CPU than usual and will likely take longer for the restarted node to report healthy. This will only happen on the first restart after downgrading and will not result in data loss. When downgrading, only downgrade to v2.11.9 or higher. Starting from this version, the server will recognize the use of new v2.12 features and will safely put the stream and/or consumer that uses these new features into an unsupported/offline mode. Importantly, this will both protect the data as well as the server itself from accessing unsupported features or data. NATS 2.11 This guide is tailored for existing NATS users upgrading from NATS version v2.10.x. This will read as a summary with links to specific documentation pages to learn more about the feature or improvement. Features Observability Distributed message tracing: Users can now trace messages as they move through the system by setting a Nats-Trace-Dest header to an inbox subject. Servers on the message path will return events to the provided subject that report each time a message enters or leaves a server, by which connection type, when subject mappings occur, or when messages traverse an account import/export boundary. Additionally, the Nats-Trace-Only header (if set to true) will allow tracing events to propagate on a specific subject without delivering them to subscribers of that subject. Streams JetStream per-message TTLs: It is now possible to age out individual messages using a per-message TTL. The Nats-TTL header, in either string or integer format (in seconds) allows for individual message expiration independent of stream limits. This can be combined with other limits in place on the stream. More information is available in ADR-43. Subject delete markers on MaxAge: The SubjectDeleteMarkerTTL stream configuration option now allows for the placement of delete marker messages in the stream when the configured MaxAge limit causes the last message for a given subject to be deleted. The delete markers include a Nats-Marker-Reason header explaining which limit was responsible for the deletion. Stream ingest rate limiting: New options max_buffered_size and max_buffered_msgs in the jetstream configuration block enable rate limiting on Core NATS publishing into JetStream streams, protecting the system from overload. Consumers Pull consumer priority groups: Pull consumers now support priority groups with pinning and overflow, enabling flexible failover and priority management when multiple clients are pulling from the same consumer. Configurable policies based on the number of pending messages on the consumer, or the number of pending acks, can control when messages overflow from one client to another, enabling new design patterns or regional awareness. Consumer pausing: Message delivery to consumers can be temporarily suspended using the new pause API endpoint (or the PauseUntil configuration option when creating), ideal for maintenance or migrations. Message delivery automatically resumes once the configured deadline has passed. Consumer clients continue to receive heartbeat messages as usual to ensure that they do not surface errors during the pause. Operations Replication traffic in asset accounts: Raft replication traffic can optionally be moved into the same account in which replicated assets live on a per-account basis, rather than being sent and received in the system account using the new cluster_traffic property in the JetStream account settings of an account. When combined with multiple route connections, this can help to reduce latencies and avoid head-of-line blocking issues that may occur in heavily-loaded multi-tenant or multi-account deployments. TLS first on leafnode connections: A new handshake_first in the leafnode tls block allows setting up leafnode connections that perform TLS negotiation first, before any other protocol handshakes take place. Configuration state digest: A new -t command line flag on the server binary can generate a hash of the configuration file. The config_digest item in varz displays the hash of the currently running configuration file, making it possible to check whether a configuration file has changed on disk compared to the currently running configuration. TPM encryption on Windows: When running on Windows, the filestore can now store encryption keys in the TPM, useful in environments where physical access may be a concern. MQTT SparkplugB: The built-in MQTT support is now compliant with SparkplugB Aware, with support for NBIRTH and NDEATH messages. Improvements Replicated delete proposals: Message removals in clustered interest-based or workqueue streams are now propagated via Raft to guarantee consistent removal order across replicas, reducing a number of possible ways that a cluster failure can result in de-synced streams. Metalayer, stream and consumer consistency: A new leader now only responds to read/write requests after synchronizing with its Raft log, preventing desynchronization between KV key updates and the stream during leader changes. Replicated consumer reliability: Replicated consumers now consistently redeliver unacknowledged messages after a leader change. Consumer starting sequence: The consumer starting sequence is now always respected, except for internal hidden consumers for sources/mirrors. Upgrade Considerations Stream ingest rate limiting The NATS Server can now return a 429 error with type JSStreamTooManyRequests when too many messages have been queued up for a stream. It should not generally be possible to hit this limit while using JetStream publishes and waiting for PubAcks, but may trigger if trying to publish into JetStream using Core NATS publishes without waiting for PubAcks, which is not advised. The new max_buffered_size and max_buffered_msgs options control how many messages can be queued for each stream before the rate limit is hit, therefore if needed, you can increase these limits on your deployments. The default values for max_buffered_size and max_buffered_msgs are 128MB and 10,000 respectively, whereas in v2.10 these were unlimited. You can detect in the server logs whether running into a queue limit with the following warning: [WRN] Dropping messages due to excessive stream ingest rate on 'account' > 'my-stream': IPQ len limit reached If your application starts to log the above warnings then you can first try to increase the limits to higher values while investigating the fast publishers, for example: jetstream { max_buffered_msgs: 50000 max_buffered_size: 256mib } Replicated delete proposals Since stream deletes are now replicated through group proposals in a replicated stream, there may be a slight increase in replication traffic on this version. JetStream healthcheck The js-server-only healthcheck no longer checks for the health of the metaleader on v2.11.0. Since this healthcheck was designed to detect the server readiness (or in k8s for the readiness probe) checking the metaleader would sometimes cause a NATS server to be considered unhealthy when restarting the servers. In v2.11, this should no longer be an issue. If the previous behavior from v2.10 is preferred, there is a new healthcheck option js-meta-only which can be used to check whether the meta group is healthy. Exit code Earlier versions of the NATS Server would return an exit code 1 when gracefully shut down, i.e. after SIGTERM. From v2.11, an exit code of 0 (zero) will now be returned instead. Server, cluster and gateway names Configurations that have server, cluster, and gateway names with spaces are now considered invalid, as this can cause problems at the protocol level. A server running NATS v2.11 will fail to start with spaces configured in these names. Please ensure that spaces are not used in server, cluster or gateway names. Downgrade Considerations Stream state When downgrading from v2.11 to v2.10, the stream state files on disk will be rebuilt due to a change in the format of these files in v2.11. This requires re-scanning all stream message blocks, which may use higher CPU than usual and will likely take longer for the restarted node to report healthy. This will only happen on the first restart after downgrading and will not result in data loss. NATS 2.10 This guide is tailored for existing NATS users upgrading from NATS version 2.9.x. This will read as a summary with links to specific documentation pages to learn more about the feature or improvement. Upgrade considerations Client versions Although all existing client versions will work, new client versions will expose additional options used to leverage new features. The minimum client versions that have full 2.10.0 support include: CLI - v0.1.0 nats.go - v1.30.0 nats.rs - v0.32.0 nats.deno - v1.17.0 nats.js - v2.17.0 nats.ws - v1.18.0 nats.java - v2.17.0 nats.net - v1.1.0 nats.net.v2 - Coming soon! nats.py - Coming soon! nats.c - Coming soon! Helm charts k8s/nats - v1.1.0 k8s/nack - v0.24.0 Downgrade warnings For critical infrastructure like NATS, zero downtime upgrades are table stakes. Although the best practice for all infrastructure like this is for users to thoroughly test a new release against your specific workloads, inevitably there are cases where an upgrade occurs in production followed by a decision to downgrade. This is never recommended and can cause more harm than good for most infrastructure and data systems. Below are a few important considerations if downgrading is required. Storage format changes 2.10.0 brings on-disk storage changes which bring significant performance improvements. These are not compatible with previous versions of the NATS Server. If an upgrade is performed to a server with existing stream data on disk, followed by a downgrade, the older version server will not understand the stream data in the new format. However, being mindful of the possibility of the need to downgrade, a special version of the 2.9.x series was released with awareness of key changes in the new storage format, allowing it to startup properly. The takeaway is that if a downgrade is the only resort, it must be to 2.9.22 or later to ensure storage format changes are handled appropriately. Stream and consumer config options There are new stream and consumer configuration options that could be problematic if a downgrade occurs since previous versions of the server have no awareness of them. Examples include: Multi-filter consumers - Downgrading would result in no filter being applied since the new field is configured as a list rather than a single string. Subject-transform on streams - Downgrading would result in the subject transform not being applied since the server has no awareness of it. Compression on streams - Downgrading when compression is enabled on streams will cause those streams to become unloadable since the older server versions will not understand the compression being used. Features Platforms Experimental support for IBM z/OS Experimental support for NetBSD Reload A server reload can now be performed by sending a message on $SYS.REQ.SERVER..RELOAD by a client authenticated in the system account. JetStream A new sync_interval server config option has been added to change the default sync interval of stream data when written to disk, including allowing all writes to be flushed immediately. This option is only relevant if you need to modify durability guarantees. Subject mapping Subject mappings can now be cluster-scoped and weighted, enabling the ability to have different mappings or weights on a per cluster basis. The requirement to use all wildcard tokens in subject mapping or transforms has been relaxed. This can be applied to config or account-based subject mapping, stream subject transforms, and stream republishing, but not on subject mappings that are associated with stream and service import/export between accounts. Streams A subject_transform field has been added enabling per-stream subject transforms. This applies to standard streams, mirrors, and sourced streams. A metadata field has been added to stream configuration enabling arbitrary user-defined key-value data. This is to supplant or augment the description field. A first_seq field has been added to stream configuration enabling explicitly setting the initial sequence on stream creation. A compression field has been added to stream configuration enabling on-disk compression for file-based streams. The ability to edit the republish config option on a stream after stream creation was added. A Nats-Time-Stamp header is now included in republished messages containing the original message's timestamp. A ts field has been added to stream info responses indicating the server time of the snapshot. This was added to allow for local time calculations relying on the local clock. An array of subject-transforms (subject filter + subject transform destination) can be added to a mirror or source configuration (can not use the single subject filter/subject transform destination fields at the same time as the array). A stream configured with sources can source from the same stream multiple times when distinct filter+transform options are used, allowing for some messages of a stream to be sourced more than once. Consumers A filter_subjects field has been added which enables applying server-side filtering against multiple disjoint subjects, rather than only one. A metadata field has been added to consumer configuration enabling arbitrary user-defined key-value data. This is to supplant or augment the description field. A ts field has been added to consumer info responses indicating the server time of the snapshot. This was added to allow for local time calculations without relying on the local clock. Key-value A metadata field has been added to key-value configuration enabling arbitrary user-defined key-value data. This is to supplant or augment the description field. A bucket configured as a mirror or sourcing from other buckets Object store A metadata field has been added to object store configuration enabling arbitrary user-defined key-value data. This is to supplant or augment the description field. Authn/Authz A pluggable server extension, referred to as auth callout, has been added. This provides a mechanism for delegating authentication checks against a bring-your-own (BYO) provider and, optionally, dynamically declaring permissions for the authenticated user. Monitoring A unique_tag field has been added to the /varz and /jsz HTTP endpoint responses, corresponding to the value of unique_tag defined in the server config. A slow_consumer_stats field has been added to the /varz HTTP endpoint providing a count of slow consumers for clients, routes, gateways, and leafnodes. A raft=1 query parameter has been added to the /jsz HTTP endpoint which adds stream_raft_group and consumer_raft_groups fields to the response. A num_subscriptions field has been added to the $SYS.REQ.SERVER.PING.STATZ NATS endpoint responses. A system account responder for $SYS.REQ.SERVER.PING.IDZ has been added which returns info for the server that the client is connected to. A system account responder for $SYS.REQ.SERVER.PING.PROFILEZ has been added and works even if a profiling port is not enabled in the server configuration. A user account responder for $SYS.REQ.USER.INFO has been added which allows a connected user to query for the account they are in and permissions they have. MQTT Support for QoS2 has been added. Check out the new MQTT implementation details overview. Clustering When defining routes between servers, a handful of optimizations have been introduced including a pool of TCP connections between servers, optional pinning of accounts to connections, and optional compression of traffic. There is quite a bit to dig into, so check out the v2 routes page for details. Leafnodes A handshake_first config option has been added enabling TLS-first handshakes for leafnode connections. Windows The NATS_STARTUP_DELAY environment variable has been added to allow changing the default startup for the server of 10 seconds Improvements Reload The nats-server --signal command now supports a glob expression on the argument which would match a subset of all nats-server instances running on the host. Streams Prior to 2.10, setting republish configuration on mirrors would result in an error. On sourcing streams, only messages that were actively between stored matching configured subjects would be republished. The behavior has been relaxed to allow republishing on mirrors and includes all messages on sourcing streams. Consumers A new header has been added on a fetch response that indicates to clients the fetch has been fulfilled without requiring clients to rely on heartbeats. It avoids some conditions in which the client would issue fetch requests that could go over limits or have more fetch requests pending than required. Leafnodes Previously, a leafnode configured with two or more remotes binding to the same hub account would be rejected. This restriction has been relaxed since each remote could be binding to a different local account. MQTT Previously a dot . in an MQTT topic was not supported, however now it is! Check out the topic-subject conversion table for details. NATS 2.2 NATS 2.2 is the largest feature release since version 2.0. The 2.2 release provides highly scalable, highly performant, secure and easy-to-use next generation streaming in the form of JetStream, allows remote access via websockets, has simplified NATS account management, native MQTT support, and further enables NATS toward our goal of securely democratizing streams and services for the hyperconnected world we live in. Next Generation Streaming JetStream is the next generation streaming platform for NATS, highly resilient, highly available, and easy to use. We’ve spent a long time listening to our community, learning from our experiences, looking at the needs of today, and thinking deeply about the needs of tomorrow. We built JetStream to address these needs. JetStream: is easy to deploy and manage, built into the NATS server simplifies and accelerates development supports wildcard subjects supports at least once delivery and exactly once within a window is horizontally scalable at runtime with no interruptions persists data via streams and delivers or replays via consumers supports multiple patterns to consume data on the same stream supports push and pull modes when consuming messages is account aware allows for detailed granularity of security, by stream, by consumer, by function Get started with JetStream. Security and Simplified Account Management Account management just became much easier. This version of NATS has a built-in account management system, eliminating the need to set up an account manager when not using the memory account resolver. With automated default system account generation, and the ability to preload accounts, simply enable a set of servers in your deployment to be account resolvers or account resolver caches, and they will handle public account information provided to the NATS system through the NATS nsc tooling. Have enterprise-scale account management up and running in minutes. CIDR Block Account Restrictions By specifying a CIDR block restriction for a user, policy can be applied to limit connections from clients within a certain range or set of IP addresses. Use this as another layer of security atop user credentials to better secure your distributed system. Ensure your applications can only connect from within a specific cloud, enterprise, geographic location, virtual or physical network. Time-Based Account Restrictions Scoped to the user, you can now specify a specific block of time during the day when applications can connect. For example, permit certain users or applications to access the system during specified business hours, or protect business operations during the busiest parts of the day from batch driven back-office applications that could adversely impact the system when run at the wrong time. Default User Permissions Now you can specify default user permissions within an account. This significantly reduces efforts around policy, reduces chances for error in permissioning, and simplifies the provisioning of user credentials. WebSockets Connect mobile and web applications to any NATS server using WebSockets. Built to more easily traverse firewalls and load balancers, NATS WebSocket support provides even more flexibility to NATS deployments and makes it easier to communicate to the edge and endpoints. This is currently supported in NATS server leaf nodes, nats.ts, nats.deno, and the nats.js clients. Native MQTT Support With the Adaptive Edge architecture and the ease with which NATS can extend a cloud deployment to the edge, it makes perfect sense to leverage existing investments in IoT deployments. It’s expensive to update devices and large edge deployments. Our goal is to enable the hyperconnected world, so we added first-class support for MQTT 3.1.1 directly into the NATS Server. Seamlessly integrate existing IoT deployments using MQTT 3.1.1 with a cloud-native NATS deployment. Add a leaf node that is MQTT enabled and instantly send and receive messages to your MQTT applications and devices from a NATS deployment whether it be edge, single-cloud, multi-cloud, on-premise, or any combination thereof. Build Better Systems We’ve added a variety of features to allow you to build a more resilient, secure, and simply better system at scale. Message Headers We’ve added the ability to optionally use headers, following the HTTP semantics familiar to developers. Headers naturally apply overhead, which was why we resisted adding them for so long. By creating new internal protocol messages transparent to developers, we maintain the extremely fast processing of simple NATS messages that we have always had while supporting headers for those who would like to leverage them. Adding headers to messages allows you to provide application-specific metadata, such as compression or encryption-related information, without touching the payload. We also provide some NATS specific headers for use in JetStream and other features. Seamless Maintenance with Lame Duck Notifications When taking down a server for maintenance, servers can be signaled to enter Lame Duck Mode where they do not accept new connections and evict existing connections over a period of time. Maintainer supported clients will notify applications that a server has entered this state and will be shutting down, allowing a client to smoothly transition to another server or cluster and better maintain business continuity during scheduled maintenance periods. React Quicker with No-Responder Notifications Why wait for timeouts when services aren’t available? When a request is made to a service (request-reply) and the NATS Server knows there are no services available the server will short circuit the request. A “no-responders” protocol message will be sent back to the requesting client which will break from blocking API calls. This allows applications to immediately react which further enables building a highly responsive system at scale, even in the face of application failures and network partitions. Subject Mapping and Traffic Shaping Reduce risk when onboarding new services. Canary deployments, A/B testing, and transparent teeing of data streams are now fully supported in NATS. The NATS Server allows accounts to form subject mappings from one subject to another for both client inbound and service import invocations and allows weighted sets for the destinations. Map any percentage - 1 to 100 percent of your traffic - to other subjects, and change this at runtime with a server configuration reload. You can even artificially drop a percentage of traffic to introduce chaos testing into your system. See Configuring Subject Mapping and Traffic Shaping in NATS Server configuration for more details. Account Monitoring - More Meaningful Metrics NATS now allows for fine-grained monitoring to identify usage metrics tied to a particular account. Inspect messages and bytes sent or received and various connection statistics for a particular account. Accounts can represent anything - a group of applications, a team or organization, a geographic location, or even roles. If NATS is enabling your SaaS solution you could use NATS account scoped metrics to bill users. NATS 2.0 NATS 2.0 was the largest feature release since the original code base for the server was released. NATS 2.0 was created to allow a new way of thinking about NATS as a shared utility, solving problems at scale through distributed security, multi-tenancy, larger networks, and secure sharing of data. Rationale NATS 2.0 was created to address problems in large scale distributed computing. It is difficult at best to combine identity management end-to-end (or end-to-edge), with data sharing, while adhering to policy and compliance. Current distributed systems increase significantly in operational complexity as they scale upward. Problems arise around service discovery, connectivity, scaling for volume, and application onboarding and updates. Disaster recovery is difficult, especially as systems have evolved to operate in silos defined by technology rather than business needs. As complexity increases, systems become expensive to operate in terms of time and money. They become fragile making it difficult to deploy services and applications hindering innovation, increasing time to value and total cost of ownership. We decided to: Reduce total cost of ownership: Users want reduced TCO for their distributed systems. This is addressed by an easy to use technology that can operate at global scale with simple configuration and a resilient and cloud-native architecture. Decrease Time to Value: As systems scale, time to value increases. Operations resist change due to risk in touching a complex and fragile system. Providing isolation contexts can help mitigate this. Support manageable large scale deployments: No data silos defined by software, instead easily managed through software to provide exactly what the business needs. We wanted to provide easy to configure disaster recovery. Decentralize security: Provide security supporting one technology end-to-end where organizations may self-manage making it easier to support a massive number of endpoints. To achieve this, we added a number of new features that are transparent to existing clients with 100% backward client compatibility. Accounts Accounts are securely isolated communication contexts that allow multi-tenancy spanning a NATS deployment. Accounts allow users to bifurcate technology from business driven use cases, where data silos are created by design, not software limitations. When a client connects, it specifies an account or will default to authentication with a global account. At least some services need to share data outside of their account. Data can be securely shared between accounts with secure services and streams. Only mutual agreement between account owners permit data flow, and the import account has complete control over its own subject space. This means within an account, limitations may be set and subjects can be used without worry of collisions with other groups or organizations. Development groups choose any subjects without affecting the rest of the system, and open up accounts to export or import only the services and streams they need. Accounts are easy, secure, and cost effective. There is one NATS deployment to manage, but organizations and development teams can self manage with more autonomy reducing time to value with faster, more agile development practices. Service and Streams Services and streams are mechanisms to share messages between accounts. Think of a service as an RPC endpoint into an account. Behind that account there might be many microservices working in concert to handle requests, but from outside the account there is simply one subject exposed. Service definitions share an endpoint: Export a service to allow other accounts to import Import a service to allow requests to be sent securely and seamlessly to another account Use cases include most applications - anything that accepts a request and returns a response. Stream definitions allow continuous data flow between accounts: Export a stream to allow egress Import a stream to allow ingress Use cases include Observability, Metrics, and Data analytics. Any application or endpoint reading a stream of data. Note that services and streams operate with zero client configuration or API changes. Services may even move between accounts, entirely transparent to end clients. System Accounts The system account publishes system messages under established subject patterns. These are internal NATS system messages that may be useful to operators. Server initiated events and data include: Client connection events Account connection status Authentication errors Leaf node connection events Server stats summary Tools and clients with proper privileges can request: Service statistics Server discovery and metrics Account servers will also publish messages when an account changes. With this information and system metadata you can build useful monitoring and anomaly detection tools. Global Deployments NATS 2.0 supports global deployments, allowing for global topologies that optimize for WANs while extend to the edge or devices. Self Healing While self healing features have been part of NATS 1.X releases, we ensured they continue to work in global deployments. These include: Client and server connections automatically reconnect Auto-Discovery where servers exchange server topology changes with each other and with clients, in real time with zero configuration changes and zero downtime while being entirely transparent to clients. Clients can failover to servers they were not originally configured with. NATS server clusters dynamically adjust to new or removed servers allowing for seamless rolling upgrades and scaling up or down. Superclusters Conceptually, superclusters are clusters of NATS clusters. Create superclusters to deploy a truly global NATS network. Superclusters use a novel spline based technology with a unique approach to topology, keeping one hop semantics and optimizing WAN traffic through optimistic sends with interest graph pruning. Superclusters provide transparent, intelligent support for geo-distributed queue subscribers. Disaster Recovery Superclusters inherently support disaster recovery. With geo-distributed queue subscribers, local clients are preferred, then an RTT is used to find the lowest latency NATS cluster containing a matching queue subscriber in the supercluster. What does this mean? Let's say you have a set of load balanced services in US East Coast (US-EAST), another set in the EU (EU-WEST), and a supercluster consisting of a NATS cluster in US-EAST connected to a NATS cluster in EU-WEST. Clients in the US would connect to a US-EAST, and services connected to that cluster would service those clients. Clients in Europe would automatically use services connected to EU-WEST. If the services in US-EAST disconnect, clients in US-EAST will begin using services in EU-WEST. Once the Eastern US services have reconnected to US-EAST, those services will immediately begin servicing the Eastern US clients since they're local to the NATS cluster. This is automatic and entirely transparent to the client. There is no extra configuration in NATS servers. This is zero configuration disaster recovery. Leaf Nodes Leaf nodes are NATS servers running in a special configuration, allowing hub and spoke topologies to extend superclusters. Leaf nodes can also bridge separate security domains. e.g. IoT, mobile, web. They are ideal for edge computing, IoT hubs, or data centers that need to be connected to a global NATS deployment. Local applications that communicate using the loopback interface with physical VM or Container security can leverage leaf nodes as well. Leaf nodes: Transparently and securely bind to a remote NATS account Securely bridge specific local data to a wider NATS deployment Are 100% transparent to clients which remain simple, lightweight, and easy to develop Allow for a local security scheme while using new NATS security features globally Can create a DMZ between a local NATS deployment and external NATS cluster or supercluster. Decentralized Security Operators, Accounts, and Users NATS 2.0 Security consists of defining Operators, Accounts, and Users within a NATS deployment. An Operator provides the root of trust for the system, may represent a company or enterprise Creates Accounts for account administrators. An account represents an organization, business unit, or service offering with a secure context within the NATS deployment, for example an IT system monitoring group, a set of microservices, or a regional IoT deployment. Account creation would likely be managed by a central group. Accounts define limits and may securely expose services and streams. Account managers create Users with permissions Users have specific credentials and permissions. Trust Chain PKI (NKeys encoded Ed25519) and signed JWTs create a hierarchy of Operators, Accounts, and Users creating a scalable and flexible distributed security mechanism. Operators are represented by a self signed JWT and is the only thing that is required to be configured in the server. This JWT is usually signed by a master key that is kept offline. The JWT will contain valid signing keys that can be revoked with the master updating this JWT. Operators will sign Account JWTs with various signing keys. Accounts sign User JWTs, again with various signing keys. Clients or leaf nodes present User credentials and a signed nonce when connecting. The server uses resolvers to obtain JWTs and verify the client trust chain. This allows for rapid change of permissions, authentication and limits, to a secure multi-tenant NATS system.