mqutils

Upgrading from v1

Mon, 01 Jan 0001 00:00:00 +0000

This page is rendered from UPGRADING.md in the repository.

v2 is a breaking release. The import paths, the handler contract, and several transport behaviors changed. This guide covers everything a v1 consumer needs.

Import paths

Every module gained the Go-required /v2 suffix:

v1	v2
`go.digitalxero.dev/mqutils`	`go.digitalxero.dev/mqutils/v2`
`go.digitalxero.dev/mqutils/types`	`go.digitalxero.dev/mqutils/v2/types`
—	`go.digitalxero.dev/mqutils/v2/runner` (new)
`go.digitalxero.dev/mq-amqp`	`go.digitalxero.dev/mq-amqp/v2`
`go.digitalxero.dev/mq-kafka`	`go.digitalxero.dev/mq-kafka/v2`
`go.digitalxero.dev/mq-nats`	`go.digitalxero.dev/mq-nats/v2`
`go.digitalxero.dev/mq-aws`	`go.digitalxero.dev/mq-aws/v2`
`go.digitalxero.dev/mq-gcp`	`go.digitalxero.dev/mq-gcp/v2`
`go.digitalxero.dev/mq-redis`	`go.digitalxero.dev/mq-redis/v2`

Handlers return an error (the headline change)

// v1: handlers acked/nacked manually and returned nothing.
func handler(ctx context.Context, msg types.Message) {
 if err := process(ctx, msg); err != nil {
 _ = msg.Nack()
 return
 }
 _ = msg.Ack()
}

// v2: return nil to acknowledge, an error to reject. The shared consumer
// runtime settles the message (with retry) based on the return value.
func handler(ctx context.Context, msg types.Message) error {
 return process(ctx, msg)
}

Manual msg.Ack()/msg.Nack() inside a handler still works; the runtime treats the duplicate settlement (types.ErrAlreadyAcknowledged) as success.
A handler that returns nil without acking is now acknowledged by the runtime; return an error when processing did not succeed.
Batch handlers (BatchHandlerFunc) follow the same contract: nil settles the whole batch, an error rejects every message the handler did not settle itself.
Double ack/nack on a message now uniformly returns types.ErrAlreadyAcknowledged on every backend (some previously returned nil).

Behavior changes to plan for

Connection loss with auto_reconnect: false returns an error from Run instead of idling forever. Enable auto_reconnect if you relied on the old hang.
Kafka has real at-least-once semantics. Acks commit offsets (the old transport committed everything on receipt regardless of handler outcome). Nacks republish to retry_topic with a retry_count header and land on dead_letter_topic once retry_max_retries is exhausted; without a retry_topic failures are retried locally with backoff. Handlers must be idempotent — rebalances can redeliver in-flight messages.
Kafka skip_verify no longer disables TLS. TLS turns on via kafkas:// or tls_enabled: true; skip_verify only relaxes certificate verification.
NATS no longer drops messages under backpressure — delivery blocks (context-aware) when the internal channel is full, matching every other backend.
Batch processing works on every backend (enable_batch_processing, batch_size, batch_timeout), not just AMQP. batch_timeout is a real duration string ("250ms"); the v1 AMQP implementation misapplied it by a factor of a million and never flushed partial batches.
Previously dead configuration is now live. If you set any of these in v1 they silently did nothing and WILL take effect after upgrading: AMQP channel_pool_size/mandatory/producer durable+auto_delete; NATS max_reconnects/reconnect_wait/queue_group/consumer_name; SQS dead_letter_queue_url (applies a RedrivePolicy), consumer tuning and FIFO group/dedup fields; GCP dead_letter_topic+max_delivery_attempts (DeadLetterPolicy — requires IAM grants to the Pub/Sub service account), publish batching thresholds, subscription tuning; Redis auth/pool/timeout fields, stream trimming, and claim_idle_time/pending_message_max_age (XAutoClaim pending recovery, Redis ≥ 6.2).

Removed API

amqp producer immediate config key (RabbitMQ ≥ 3.0 rejects the flag).
aws consumer message_retention_days (was never applied; honoring it would clobber externally provisioned retention).
GCP skip_verify config keys (meaningless for the gRPC client; the emulator path uses PUBSUB_EMULATOR_HOST).
Exported AcknowledgeMessage/NacknowledgeMessage helpers in the kafka and nats packages — use runner.AckWithRetry/runner.NackWithRetry.
Transport.DeclareExchange/Transport.BindQueue now take a context.Context, and BindQueue’s parameters are exchange-first with a durable flag (matching what every implementation always did).

New in v2 (non-breaking once you’re on the new paths)

Typed builders: mqutils.NewConsumerBuilder() / NewProducerBuilder() configure consumers without viper and take handler funcs directly.
Canonical config keys accepted by every backend alongside native keys: destination, max_retries, consumer_group.
Symmetric URL schemes: consumers and producers register identical scheme sets (sqss://, gcp://, redisstreams://, … all work on both sides now).
runner package: the shared consumer loop (ack-retry helpers, batching, graceful shutdown, reconnect backoff, retry-budget drops with a dead-letter hook) is exported for building custom backends.
AMQP blocked-connection handling: when RabbitMQ raises a memory/disk alarm (connection.blocked), fire-and-forget publishes are accepted into a bounded internal queue (publish_queue_size, default 1000; overflow returns types.ErrPublishQueueFull) and flushed in order on unblock — v1 wrote into the stalled socket and hung. New publisher-confirms mode (publisher_confirms: true, confirm_timeout): every publish waits for the broker’s ack (types.ErrPublishNacked on nack) and publishes during a block are rejected immediately with types.ErrConnectionBlocked for the caller to handle. NATS gains reconnect_buf_size to bound the client buffer that already queues publishes while disconnected.

Reliability update for existing v2 deployments

RabbitMQ queue expiration is opt-in

transient_queue_expires is now disabled when omitted or zero. A positive integer enables automatic x-expires in milliseconds for transient queues that RabbitMQ 4.3+ requires the library to declare durable. The durability workaround remains in effect. Intentionally durable queues do not acquire an expiration automatically.

Why mqutils

Mon, 01 Jan 0001 00:00:00 +0000

The problem it solves

Every broker SDK has its own connection lifecycle, its own way to acknowledge or reject a message, its own retry story, and its own configuration shape. A service that runs against RabbitMQ in one deployment and SQS in another ends up with two consumer implementations, or with an in-house abstraction that someone has to maintain.

mqutils is that abstraction, maintained once. It puts a single consumer and producer contract in front of RabbitMQ AMQP and Streams, Kafka, NATS Core and JetStream, AWS SQS, GCP Pub/Sub, and Redis Pub/Sub and Streams. The URL scheme selects the backend; each backend is its own Go module, so you compile only the ones you use.

Getting started

Mon, 01 Jan 0001 00:00:00 +0000

mqutils puts one consumer and producer interface across seven backend modules. This guide installs the core module plus one backend, then walks through a consumer, a producer, batching, health checks and shutdown.

Upgrading from v1? See Upgrading from v1: v2 changed the import paths and the handler contract.

Installation

Add the core library and the backend module(s) you need to your Go project:

bash

go get go.digitalxero.dev/mqutils/v2

# One or more backends
go get go.digitalxero.dev/mq-amqp/v2
go get go.digitalxero.dev/mq-kafka/v2
go get go.digitalxero.dev/mq-nats/v2
go get go.digitalxero.dev/mq-aws/v2
go get go.digitalxero.dev/mq-gcp/v2
go get go.digitalxero.dev/mq-redis/v2

The RabbitMQ Stream module requires Go 1.25 and has separate availability and local-checkout instructions while its new vanity mappings and release are prepared. Existing module imports above are unaffected.

Core types

Mon, 01 Jan 0001 00:00:00 +0000

import "go.digitalxero.dev/mqutils/v2/types"

Index

Constants
Variables
func ApplyConfigAliases(v *viper.Viper, aliases AliasMap)
func NewErrUnknownPublisherType(url string) error
func RedactURL(raw string) string
func RegisterBatchHandler(name string, handler BatchHandlerFunc)
func RegisterConsumer(consumer NewConsumerFunc, urlPrefixes …string)
func RegisterHandler(name string, handler HandlerFunc)
func RegisterProducer(producer NewProducerFunc, urlPrefixes …string)
func RegisteredConsumerPrefixes() []string
func RegisteredProducerPrefixes() []string
func SetConfigOverride(v *viper.Viper, key string, value any)
type AliasMap
type BatchHandlerFunc
- func GetBatchHandler(name string) BatchHandlerFunc
type ConfigurationError
- func (e *ConfigurationError) Error() string
type ConfigurationErrors
- func (e *ConfigurationErrors) Add(field, message string)
- func (e *ConfigurationErrors) Error() string
- func (e *ConfigurationErrors) HasErrors() bool
type Consumer
type ErrUnknownConsumerType
- func NewErrUnknownConsumerType(typeName string) *ErrUnknownConsumerType
- func (e ErrUnknownConsumerType) Error() string
type ErrUnknownPublisherType
- func (e ErrUnknownPublisherType) Error() string
type HandlerFunc
- func GetHandler(name string) HandlerFunc
type HealthCheck
type HealthCheckBuilder
- func NewHealthCheckBuilder() HealthCheckBuilder
type HealthCheckFunc
type HealthChecker
type HealthStatus
type Message
type MessageBuilder
type NewConsumerFunc
- func GetConsumer(url string) (NewConsumerFunc, bool)
type NewProducerFunc
- func GetProducer(url string) (NewProducerFunc, bool)
type Producer
type Publisher
type Transport

Constants

Canonical configuration keys accepted by every backend alongside its native keys. Backends map these onto their native keys with ApplyConfigAliases before unmarshalling their configuration.

Main package

Mon, 01 Jan 0001 00:00:00 +0000

import "go.digitalxero.dev/mqutils/v2"

Package mqutils provides a unified abstraction layer for working with multiple message queue systems. It supports AMQP/RabbitMQ, Apache Kafka, NATS, AWS SQS, GCP Pub/Sub, and Redis with a consistent API.

The package uses URL-based routing to automatically select the appropriate message queue implementation based on the connection URL scheme:

amqp://, amqps:// - RabbitMQ/AMQP
kafka://, kafkas:// - Apache Kafka
nats://, natss://, jetstream:// - NATS Core/JetStream
sqs://, sqss:// - AWS SQS
pubsub:// - GCP Pub/Sub
redis://, rediss://, redisstream:// - Redis Pub/Sub & Streams

Example usage:

Kafka security and transactions

Mon, 01 Jan 0001 00:00:00 +0000

The Kafka module supports TLS, SASL PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512. Its transactional producer publishes atomic groups of records across topics and partitions. Consumer offsets remain independent: this API does not provide an atomic consume-transform-produce transaction.

SASL configuration

Producer and consumer configurations use the same authentication settings. The settings also apply to reconnects, administrative requests, and retry/dead-letter publishers created by the Kafka transport.

config := viper.New()
config.Set("url", "kafkas://broker.example.com:9093")
config.Set("sasl_enabled", true)
config.Set("sasl_mechanism", "SCRAM-SHA-256")
config.Set("sasl_username", os.Getenv("KAFKA_USERNAME"))
config.Set("sasl_password", os.Getenv("KAFKA_PASSWORD"))

producer, err := kafka.NewKafkaProducer(config)
if err != nil {
 return err
}
defer producer.Close()
if err := producer.Start(ctx); err != nil {
 return err
}
return producer.Publish(ctx, "order-42", "orders", "customer-7",
 "application/json", []byte(`{"id":42}`))

Imports used in these fragments are github.com/spf13/viper, go.digitalxero.dev/mq-kafka/v2 as kafka, and the standard packages shown by usage. Put credentials into Viper explicitly or bind its environment variables; ${VARIABLE} inside YAML is not automatically expanded by mqutils.

Runner

Mon, 01 Jan 0001 00:00:00 +0000

import "go.digitalxero.dev/mqutils/v2/runner"

Package runner provides the shared consumer processing loop used by every mqutils backend. It owns the concerns that are identical across message queue systems — dispatching messages to handlers, acknowledging based on the handler’s returned error, batch collection with a real flush timer, retry-budget drops with a dead-letter hook, reconnect with exponential backoff, and graceful shutdown draining — so backends only supply the broker-specific pieces via callbacks.

RabbitMQ Streams

Mon, 01 Jan 0001 00:00:00 +0000

go.digitalxero.dev/mq-rmqstream/v2 uses RabbitMQ’s native Stream protocol. It is a separate backend from the AMQP module and requires Go 1.25 or newer. Importing it registers rabbitmq-stream:// and rabbitmq-stream+tls:// for both producers and consumers.

Publishing succeeds only after broker confirmation. Consumers acknowledge by storing progress, not deleting records. Only the settled prefix of delivered records can advance the checkpoint; unfinished earlier work prevents later handlers from skipping it. A crash can replay records, and stream retention can remove records independently of consumption. Design handlers for at-least-once processing.

AMQP / RabbitMQ

Mon, 01 Jan 0001 00:00:00 +0000

import "go.digitalxero.dev/mq-amqp/v2"

Index

func NewAMQPConsumer(config *viper.Viper) (types.Consumer, error)
func NewAMQPProducer(config *viper.Viper) (types.Producer, error)
func NewAMQPTransport(retry bool, prefetch int) types.Transport
func NewMessageBuilder() types.MessageBuilder

funcNewAMQPConsumer

func NewAMQPConsumer(config *viper.Viper) (types.Consumer, error)

NewAMQPConsumer creates a new AMQP/RabbitMQ consumer with the provided configuration. This function is typically called by mqutils.NewConsumer when it detects an AMQP URL.

Apache Kafka

Mon, 01 Jan 0001 00:00:00 +0000

See Kafka security and transactions for SASL configuration, the scoped transaction API, and commit-outcome handling.

import "go.digitalxero.dev/mq-kafka/v2"

Index

Variables
func NewKafkaConsumer(config *viper.Viper) (types.Consumer, error)
func NewKafkaProducer(config *viper.Viper) (types.Producer, error)
func NewKafkaTransport() types.Transport
func NewMessageBuilder() types.MessageBuilder
type TransactionalProducer
type TransactionalProducerBuilder
- func NewTransactionalProducerBuilder() TransactionalProducerBuilder

Variables

var (
 // ErrTransactionRequired rejects ordinary publishes on a transactional producer.
 ErrTransactionRequired = errors.New("Kafka publication requires InTransaction")
 // ErrTransactionClosed means the callback publisher is no longer usable.
 ErrTransactionClosed = errors.New("Kafka transaction scope is closed or aborted")
 // ErrTransactionOutcomeUnknown means commit may have reached Kafka. Do not replay
 // the callback automatically; reconcile application state before retrying.
 ErrTransactionOutcomeUnknown = errors.New("Kafka transaction commit outcome is unknown")
 // ErrTransactionalProducerClosed indicates that Close released the producer.
 ErrTransactionalProducerClosed = errors.New("Kafka transactional producer is closed")
 // ErrTransactionalProducerFailed requires an explicit Start or a new producer.
 ErrTransactionalProducerFailed = errors.New("Kafka transactional producer must be restarted after failure")
)

NATS

Mon, 01 Jan 0001 00:00:00 +0000

import "go.digitalxero.dev/mq-nats/v2"

Index

func NewMessageBuilder() types.MessageBuilder
func NewNatsConsumer(config *viper.Viper) (types.Consumer, error)
func NewNatsProducer(config *viper.Viper) (types.Producer, error)
func NewNatsTransport() types.Transport

funcNewMessageBuilder

func NewMessageBuilder() types.MessageBuilder

NewMessageBuilder creates a new NATS message builder. The builder provides a fluent interface for constructing NATS messages with all supported properties and attributes.

AWS SQS

Mon, 01 Jan 0001 00:00:00 +0000

import "go.digitalxero.dev/mq-aws/v2"

Index

func NewMessageBuilder() mqtypes.MessageBuilder
func NewSQSConsumer(config *viper.Viper) (types.Consumer, error)
func NewSQSProducer(config *viper.Viper) (types.Producer, error)
func NewSQSTransport() mqtypes.Transport

funcNewMessageBuilder

func NewMessageBuilder() mqtypes.MessageBuilder

NewMessageBuilder creates a new SQS message builder. The builder provides a fluent interface for constructing SQS messages with all supported properties and attributes.

GCP Pub/Sub

Mon, 01 Jan 0001 00:00:00 +0000

import "go.digitalxero.dev/mq-gcp/v2"

Index

func NewMessageBuilder() mqtypes.MessageBuilder
func NewPubSubConsumer(config *viper.Viper) (types.Consumer, error)
func NewPubSubProducer(config *viper.Viper) (types.Producer, error)
func NewPubSubTransport() mqtypes.Transport

funcNewMessageBuilder

func NewMessageBuilder() mqtypes.MessageBuilder

NewMessageBuilder creates a new Pub/Sub message builder. The builder provides a fluent interface for constructing Pub/Sub messages with all supported properties and attributes.

Redis

Mon, 01 Jan 0001 00:00:00 +0000

import "go.digitalxero.dev/mq-redis/v2"

Index

func NewMessageBuilder() mqtypes.MessageBuilder
func NewRedisConsumer(config *viper.Viper) (types.Consumer, error)
func NewRedisProducer(config *viper.Viper) (types.Producer, error)
func NewRedisTransport() mqtypes.Transport

funcNewMessageBuilder

func NewMessageBuilder() mqtypes.MessageBuilder

NewMessageBuilder creates a new Redis message builder. The builder provides a fluent interface for constructing Redis messages with support for both Pub/Sub and Streams modes.

RabbitMQ Streams

Mon, 01 Jan 0001 00:00:00 +0000

See the RabbitMQ Streams guide for configuration, ownership, filtering, durable outbox/inbox examples, and the current module availability note. This module requires Go 1.25 or newer. The optional Redis store has its own API reference.

import "go.digitalxero.dev/mq-rmqstream/v2"

Package rmqstream connects mqutils to RabbitMQ’s native Stream protocol.

Importing the package registers rabbitmq-stream:// and rabbitmq-stream+tls://. Producers await broker confirmations. Consumers checkpoint the settled delivery prefix, so concurrent handlers cannot commit past an earlier unfinished record. Acknowledgment stores progress; it does not delete stream data. Retention can remove data independently of consumption, and crash recovery can replay records.

Redis deduplication adapter

Mon, 01 Jan 0001 00:00:00 +0000

This optional module implements the Stream backend’s atomic deduplication-store contract. It does not register a broker URL scheme. See the application deduplication guide for leases, retention, manual acknowledgment, and transactional inbox boundaries. Redis must retain active claims and completed entries: the adapter requires maxmemory-policy=noeviction and checks it using CONFIG GET.

import "go.digitalxero.dev/mq-rmqstream-dedupredis/v2"

Package dedupredis provides a durable Redis claim store for RabbitMQ Stream consumer deduplication. Redis must use maxmemory-policy=noeviction. Restart durability depends on Redis persistence and replication: configure AOF and appropriate fsync/replication guarantees for the required failure model.

Examples

Mon, 01 Jan 0001 00:00:00 +0000

Every example on this page targets the v2 API:

Import paths carry a /v2 suffix: go.digitalxero.dev/mqutils/v2, go.digitalxero.dev/mqutils/v2/types, and one go.digitalxero.dev/mq-<backend>/v2 module per broker. Blank-import a backend module to register its URL schemes.
Handlers are func(ctx context.Context, msg types.Message) error. Return nil to acknowledge the message, return an error to reject it. The shared consumer runtime settles the message for you, so there is no Ack()/Nack() bookkeeping in handlers.
Consumers are built with mqutils.NewConsumerBuilder() (typed, no viper) or mqutils.NewConsumer(ctx, viperConfig). Producers use mqutils.NewProducerBuilder() or mqutils.NewProducer(ctx, viperConfig) and must be started with Start(ctx) before publishing. The Kafka-specific transactional builder is an exception: its Build connects immediately.
The canonical keys destination, max_retries, and consumer_group are accepted by every backend alongside its native keys. Native keys win when both are set.

Working samples also live in the repository’s _examples/ directory.

FAQ

Mon, 01 Jan 0001 00:00:00 +0000

General

Why use mqutils instead of the broker’s SDK?

Because the parts that are the same across brokers (settling a message from a handler’s return value, retry budgets, batching, reconnect, graceful shutdown, health checks) are written once in the shared runtime, and the parts that differ live behind a URL scheme. You learn one handler contract and one configuration vocabulary. See Why mqutils for what it does and, just as important, what it does not do.