Occasional anomalously-high `messages.inbound.rest.messages.refused` stat when a rest publish is refused

Incident Report for Ably

Resolved

The fix for this has been fully rolled-out.

Posted Oct 11, 2024 - 12:20 UTC

Identified

Since Monday 7th, a REST publish that was refused by Ably's servers (for example, for being above the maximum message size) had a small probability of contributing an incorrect, anomalously high message inbound refused stat (key `messages.inbound.rest.messages.refused`) to app and account stats.

Since refused messages do not count towards a customer's chargeable message quota, this bug did not result on anyone being charged more or having their quota used up. However, in a couple of cases it did momentarily trigger a customer's account-wide instantaneous message rate limit (`messages.maxRate`), for which we apologise.

We are rolling out a fix for this currently.

Posted Oct 11, 2024 - 10:12 UTC

This incident affected: Ably Global Service Availability.