Incident review: B-Storage degradation
14 October 2026, 08:12–09:03 CEST · zone sto-1b · severity: minor · no data loss
Published 16 October 2026. Written by the Cloud-B NOC shift.
What happened
Between 08:12 and 09:03 CEST, part of the S3 API requests in zone sto-1b returned 503 SlowDown. At peak, the error rate reached 2.1%; across the whole incident it averaged 0.4%. Read and write operations in other zones were not affected.
Replication did not stop, and all three copies of every object remained in place. No data recovery was required and no writes were lost.
Root cause
One NVMe drive on storage node s3-sto-b-07 degraded: median response time rose from 0.3 ms to 40 ms and p99 to 900 ms. The S3 gateway serving the zone exhausted its connection pool (512) and began rejecting new requests. Client retries with short intervals amplified the load: some clients retried immediately after a 503.
The node was not automatically drained from rotation because the p99 alert threshold was set to 1,500 ms — above the value actually reached during the peak.
Timeline (CEST)
Alert: the 5xx rate on the zone B S3 gateway exceeded 1%. The on-call engineer opened an incident.
Zone B traffic was moved to the zone A gateway. The error rate dropped to 0.2% and new requests were served without delay.
Root cause located: latency growth on NVMe node
s3-sto-b-07and exhaustion of the gateway connection pool.The node was drained from rotation and data was rebalanced to neighbouring nodes. Affected requests were retried by clients and internal retries.
The gateway connection pool was enlarged, and a retry rate limit for the zone was enabled at the edge.
The error rate returned to baseline and the incident was closed. Monitoring watched the zone for another 60 minutes.
Remediation
| Action | Status | Due |
|---|---|---|
| Enlarge the gateway connection pool to 2048 and add a wait queue | done | 18.10.2026 |
| Automatically drain a node when p99 latency exceeds 10 ms | done | 21.10.2026 |
| Rate-limit client retries at the edge (token bucket) | done | 16.10.2026 |
| Restore test of zone B buckets from a snapshot | done | 25.10.2026 |
| Replace the NVMe drive and update firmware on zone B nodes | in progress | 15.12.2026 |
| Lower the p99 alert threshold to 500 ms with a 3-minute window | done | 17.10.2026 |
What changed for customers
Nothing is required: the API and compatibility are unchanged. Customers that retry aggressively should use exponential backoff starting at 200 ms and honour the Retry-After header — this reduces the risk of a second degradation during similar failures.
Questions about the incident: support@cloud-b.myftp.org; include inc_2026_10_14_storage in the subject.