MQTTPublishLatency |
Timer around every egress publish, success or failure (Sender.Send). |
Broker acceptance (PUBACK/PUBCOMP for QoS 1/2) is slowing down; correlate with broker load and MQTTPublishFailures. |
MQTTPublishFailures |
An egress publish failed or the broker rejected it with a non-success reason code (Sender.Send); circuit-open rejections also count here with a reason=circuit_open tag (CircuitBreakerSender.Send). |
The primary egress error counter — alert on a sustained non-zero rate. Failures surface to the route runner for retry/outbox handling, so a rising value is broker/topic trouble, not silent loss. |
MQTTConnectLatency |
Timer around a successful initial Session.Start connect (dial + CONNACK + connection-up callback). |
Broker connection establishment is slowing (TLS handshake, auth backend, broker load). Not emitted for background reconnect attempts. |
MQTTReconcileLatency |
Timer around each successful subscription reconcile (Session.reconcile: SUBSCRIBE/UNSUBSCRIBE convergence). |
The broker is slow to answer SUBACK/UNSUBACK; contributes directly to startup, reload, and failover time. |
MQTTHandlerPanics |
A router dispatch handler panicked; the panic is recovered and the delivery is left un-acked for broker redelivery (router.fanout / router.emitOne). No session_id tag. |
A bug in the receive pipeline. The un-acked publish is redelivered, so a panicking handler can loop the same message — find and fix the panic. |
MQTTRouterBuffered |
A publish arrived before any matching handler was registered (e.g. a persistent-session backlog delivered on CONNACK before Receiver.Run) and was held in the router’s bounded pending buffer during the unmatched_grace window instead of being immediately acked-and-dropped (router.dispatchCore). No session_id tag. |
Normal in small bursts at reconnect; a large or growing value means handlers register too slowly or the backlog exceeds the buffer. Publishes still unmatched when the grace window closes are settled as covered-retained or orphan-dropped. |
MQTTRouterDropped |
A QoS 0 publish was dropped under backpressure: the serialized dispatch queue was full under a flood, or the pre-registration pending buffer hit its entry/byte ceiling (defaultPendingBytesLimit=64 MiB), or an older QoS 0 entry was evicted to make room for QoS 1/2 (router.enqueueDispatch, router.dropQoS0, router.evictOldestQoS0Locked). The Warn log names the refusing bound — dispatch budget exhausted (the shared admission budget is full) versus pending buffer full during startup grace (a receiver has not registered) — because the remedies differ. The dispatch queue is sized to the effective receive_maximum (default 192, not a fixed constant — see docs/transports/mqtt.md). QoS 1/2 is never dropped here — it blocks on the dispatch queue or evicts an older QoS 0 entry. Also incremented ONCE by a raw pre-decode terminal reject (Session.rejectPredecodeIngress: malformed packet or total size above the advertised maximum — a broker bug that also fails the session closed). |
A QoS 0 flood is outrunning handler dispatch, or a large CONNACK backlog exceeded the pending cap before handlers registered. QoS 0 carries no delivery contract, so drops are expected under overload; a sustained rate means the consumer cannot keep up — add capacity or shed load upstream. A single count coinciding with a session terminal error is the pre-decode reject, not backpressure. |
MQTTRouterUnmatchedDropped |
A publish still matched no registered topic filter after the unmatched_grace window elapsed AND its topic is not covered by any subscription the session still wants: it was acked, dropped, and an UNSUBSCRIBE for its exact topic attempted (deduped, one warn per topic) (router.dropOrphan, Session.unsubscribeOrphan). The UNSUBACK decides what is reported: 0x00 converges and is logged at Debug; 0x11 (no subscription existed) means the orphan was created with a wildcard or shared filter that MQTT cannot unsubscribe by a delivered topic, and is reported with a Warn naming the remaining work instead of a cleanup that did not happen. Before the FIRST reconcile of a process lifetime nothing is ever counted here — the pre-plan backlog is retained as covered instead. |
The signature of an orphan broker-side subscription — a route removed from config whose subscription survived on the resumed clean_start=false session. Expected as a one-shot right after a route removal; a continuously rising value means the broker keeps delivering for a subscription no configured route covers — investigate the removed route (a surviving wildcard subscription cannot be cleared by the concrete-topic UNSUBSCRIBE; configure the managed subscription store to converge it). |
MQTTRouterCoveredRetained |
A publish on a topic a still-desired subscription covers was RETAINED un-acked past the grace window because its receiver handler had not registered yet (router.settlePending / router.retainCovered). NOT loss. |
A receiver registers later than unmatched_grace (or never). Investigate the slow route start; the retained backlog pins broker in-flight slots until the handler registers or the broker redelivers. |
MQTTRouterCoveredDropped |
A covered QoS 0 publish was dropped past the grace window because the bounded pending buffer could not hold it (router.retainCovered), or because a QoS 1/2 admission reclaimed its slot from the shared ingress budget (router.evictOldestQoS0Locked). Covered QoS 1/2 is never counted here. |
Best-effort loss on a live route during slow startup. Any non-zero value: speed up receiver registration or lengthen unmatched_grace. |
MQTTRouterOverflowDropped |
A QoS 1/2 publish was acked-and-dropped because the pending buffer’s count cap (== receive_maximum) was hit with no evictable QoS 0 (router.overflowAckDrop). Unreachable with a spec-compliant broker. |
The broker delivered more un-acked QoS 1/2 than the Receive Maximum it was granted — a protocol violation. MESSAGE LOST; investigate the broker. |
MQTTRouterStalePurged |
An old-connection publish was discarded: pending entries purged when the connection generation advances (their acks died with the old connection) or old-socket ingress released during a recovery/managed-cleanup recycle window (router.purgeStalePendingLocked, router.enqueueDispatch/router.dispatchCore discard branches). The generation advances once per reported connection teardown, opened by whichever comes first — the connection-up callback or the first packet of the replacement connection — so a CONNACK backlog that beats the callback belongs to the LIVE generation and is never counted here. It also counts ingress released while the session is CLOSING: the router is stopped before the SDK disconnect (otherwise a parked publish callback pins that disconnect for the whole close deadline), so publishes keep arriving for the length of the disconnect. QoS 1/2 released that way is redelivered on session resume; QoS 0 is counted on MQTTRouterDropped instead. |
QoS 1/2 counted here is redelivered by the resumed session (not lost); QoS 0 is a best-effort loss across a disconnect. A steadily rising value means frequent reconnects/recycles while traffic is in flight. |
MQTTIngressPoisonDropped |
An inbound publish violated a local representational cap the broker cannot enforce (max_payload_bytes, metadata byte cap, User Property count cap) while fitting the advertised Maximum Packet Size; it was ACKED-and-DROPPED to prevent a permanent redelivery/terminal loop (router.dropPoisonIngress). |
An authorized publisher is sending packets this bridge is configured to refuse; each count is a deliberate, acknowledged loss. Alert on any non-zero value and follow docs/runbooks/mqtt-ingress-poison.md. |
MQTTIngressUserPropertiesTruncated |
An inbound PUBLISH carried more User Properties than the adapter will ever decode, so the raw-bytes guard cut the list to one above the retained cap before the SDK saw the packet (mqttIngressConn.readPacket). Tagged session_id. |
Not a loss signal on its own – the packet is refused by the callback and counted on MQTTIngressPoisonDropped as well. It says the property count in that drop’s Error log was bounded by the guard, not the count the publisher sent (which is logged at Debug). A non-zero value means a publisher is sending packets whose metadata section alone would cost megabytes to decode; find the publisher. |
MQTTAckAfterReconnect |
A delivery settled after the connection that received it was gone, so its protocol ack could not reach the broker; the settlement was mapped to success (router.ackWithReconnectMapping). Detection compares the Paho CLIENT captured at receive against the live one, not the SDK error — paho marks an ack and flushes the acknowledged prefix asynchronously, so an ack marked just before the drop returns success and is still redelivered. It is deliberately not the connection epoch either: the epoch also advances for a recycle on a still-live socket, which would report every settlement in a routine drain as a redelivery. |
Each count is a guaranteed broker redelivery — a burst after a reconnect storm predicts duplicates on routes without downstream dedup (direct_hold). Verify downstream idempotency. |
MQTTReceiverEmitRejected |
The route pipeline refused a delivery at emit — a shutting-down or wedged route runner (Receiver.countEmitRejection). Tagged outcome=recovering for a durable QoS 1/2 delivery on a session that RESUMES — persistent or exclusive — (left un-acked; the bounded session recycle makes the broker redeliver it) or outcome=lost when nothing can redeliver it: QoS 0 (no ack to withhold), or QoS 1/2 on an ephemeral session, where clean_start=true leaves no session to resume. A lost delivery is ACKED so it stops pinning a Receive-Maximum slot for the life of the connection. |
A rising recovering rate is the leading indicator of the session recycles that follow, each of which redelivers every unsettled delivery on the session. Any non-zero lost count is acknowledged best-effort loss — alert on it if QoS 0 ingress is significant to the deployment. |
MQTTIngressHeaderDropped |
An inbound MQTT user property was dropped because its key/value is unsafe (invalid UTF-8, control characters) or over-long (EnvelopeFromPublish). No session_id tag. |
A peer publishes spec-legal-but-rejected headers; routes filtering on those headers misroute. Find the publisher. |
MQTTNonStringHeaderDropped |
An egress header value was dropped because it is not a string and cannot become an MQTT user property (PublishFromEnvelope). No session_id tag. |
Bridge-to-bridge metadata (idempotency key, tenant id) is being lost on egress — fix the producing route’s header types. |
MQTTEgressRejected |
A publish was refused BEFORE any byte reached the socket because the constructed packet violates a wire rule: a length-prefixed field above the MQTT v5 65,535-byte ceiling (which the SDK would silently truncate), a string field that is not well-formed UTF-8 or contains U+0000, or an encoded packet larger than the Maximum Packet Size the broker granted in its CONNACK (pahoConn.PublishEnvelope). No session_id tag. |
The message is returned to the route as a permanent rejection and DLQ’d, not retried. A producer or route is generating messages this broker cannot accept — find the oversized or malformed field, or lower the message size, before the DLQ fills. |
MQTTEventDropped |
The bounded session lifecycle-event channel was full and an event was evicted (Session.pushEvent). No session_id tag. |
Under an event storm a SessionConnected may be evicted, deferring a reconcile one connect edge. Alert if non-zero in steady state. |
MQTTSessionRecoveryRecycle |
A durable QoS 1/2 Delivery.Retry (or an emit-error recovery request) started an actual settlement-recovery session recycle (Session.recordRecoveryRecycleStart). |
The downstream is failing deliveries hard enough to force session recycles; every recycle redelivers ALL unsettled deliveries on the session (duplicates for innocent in-flight messages). |
MQTTUnsettled / MQTTOldestUnsettledAge / MQTTReceiveWindowUtilization |
Gauges snapshotting the un-acked QoS 1/2 window per health sweep (Session.Health). |
Rising unsettled count/age means settlement (outbox persist / target accept) is stalling; utilization → 1.0 means ingress is about to block on the broker’s Receive-Maximum window. An emit-error stranded delivery now triggers a bounded recovery recycle instead of wedging here permanently. |
MQTTSessionTakeover |
A server disconnect with reason code 0x8E (Session taken over): another client connected with the same ClientID (Session.noteSessionTakeover). Reason code 0x8F is Topic Filter Invalid — a different condition, NOT counted here. |
Two instances share a client_id and keep kicking each other — give each replica a distinct ClientID (client_id_suffix) or use an exclusive session. One takeover during exclusive failover is normal; a climbing streak is a collision. |
MQTTQoSDowngraded |
The broker granted a subscription at a LOWER QoS than requested (Session.reconcile on SUBACK). |
The delivery guarantee is weaker than the route assumes; readiness stays below Full. Investigate a broker QoS-cap policy. The SAME grant confirmed on three consecutive reconciles is treated as PERMANENT: the reconcile error then carries the permanent-closure marker and the session is failed terminally instead of restarting into the identical downgrade forever (an exclusive owner would otherwise release and re-seize its lease on every cycle). Fix it by lowering the route’s QoS to the granted level or lifting the broker’s cap. |
MQTTConnectFailures |
A CONNECT was rejected or failed (Session.noteConnectFailure, from autopaho’s OnConnectError). Tagged session_id and code — the bounded BridgeError code (UNAVAILABLE, NOT_AUTHORIZED, TIMEOUT, CONNECTION_LOST, …). The broker URL is deliberately not a dimension (it may carry credentials). The same cause is latched on SessionHealth.LastError until a connection comes up. |
MQTT authenticates only at CONNECT and autopaho then retries on its own, so this is the one place a reconnect names its cause. A rising NOT_AUTHORIZED rate is a credential problem (credential-expiry-rotation-failure.md); a rising UNAVAILABLE / CONNECTION_LOST rate is a broker or network outage (broker-outage-reconnect-storm.md). |
MQTTSessionResumeLost |
A persistent or exclusive session connected with clean_start=false and the CONNACK answered Session Present=false (Session.noteDurableResumeLost). Tagged session_id. Not emitted on a cold start — the signal needs a prior connection on this session, or a non-empty managed subscription history proving this client id previously held broker-side filters. |
The broker had no session to resume: the queued offline QoS 1/2 backlog and the broker-side subscriptions for this client_id are gone. Re-subscribing restores delivery for NEW messages only. Causes: a session expiry elapsed during a long outage, a broker restart without persistence, or an exclusive standby connecting after session_expiry_interval. Any non-zero value means offline continuity was broken at least once — verify session_expiry_interval exceeds your worst-case failover gap. |