Microservice Rate Limiting, Circuit Breaking, and Degradation: A Practical Guide from Avalanche Prevention to Fine-Grained Traffic Governance

Overview

3 AM, phone buzzing wildly. Open it — the alert group is already on fire. Payment service P99 latency jumped from 50ms to 12 seconds, error rate past 80%. Worse, the failure cascaded like dominoes: user service timeout → auth failures → inventory service retry storm → message queue backed up → payment callbacks all timed out. The monitoring dashboard went entirely red in thirty seconds.

This isn’t a unique disaster. The finer you split microservices, the longer the call chain, and the easier it is for one local failure to cascade into a global avalanche. I once managed 50+ microservices at a new energy logistics platform and lived through three similar incidents. The last one was the worst — a downstream search service’s database connection pool got exhausted, causing all upstream services depending on search results to have their thread pools cascade-depleted. The entire transaction pipeline was down for 17 minutes.

After that incident, we spent two weeks rebuilding our rate limiting, circuit breaking, and degradation system. We brought P99 down from 200ms to 170ms, and more importantly, we never had a cascading failure again. This article covers every pitfall, architecture decision, and production-proven configuration I’ve encountered.

Let’s clarify the relationship between the three — people use them interchangeably, but they solve different problems:

Mechanism	What It Solves	Analogy	Position
Rate Limiting	Traffic exceeds system capacity	A supermarket opens only 3 checkout lanes; once full, no more customers enter	Entry point
Circuit Breaking	Downstream service failure; prevent propagation	A fuse that trips when current is too high	Middle of call chain
Degradation	Provide alternatives when core features unavailable	Power goes out, emergency lights kick in — dimmer but you can still see	Business logic layer

All three work together: rate limiting controls input at the front, circuit breaking cuts fault propagation in the middle, and degradation ensures user experience doesn’t completely collapse at the tail end.

Avalanche Effect: Why Microservices Fail Harder Than Monoliths

In a monolith, method calls are in-process function calls — microsecond-level. Split into microservices, every call becomes a network request — networks can jitter, DNS can be slow, TCP handshakes have overhead, serialization/deserialization takes time. A single user request passing through 5-7 service layers from the gateway is the norm.

Fault Propagation Path

Look at a real call chain:

User Request → API Gateway → Order Service → User Service (Auth)
                                  → Inventory Service (Deduct)
                                  → Coupon Service
                                  → Payment Service → Third-party Payment Gateway

Suppose the third-party payment gateway slows down (response time goes from 100ms to 5 seconds). What happens?

Payment service threads are occupied, waiting for the third party to respond
Payment service thread pool gradually exhausts, new requests queue up
Order service’s call to payment service times out, starts retrying
Retried requests pile onto the already-overloaded payment service, making things worse
Order service’s own thread pool gets dragged down
Inventory service, coupon service all affected
API Gateway thread pool exhausted, entire system refuses service

Key data: if the order service has 200 worker threads, each request calls the payment service, and payment service slows to 5 seconds per response — then 200 threads get occupied within 5 seconds. All subsequent new requests either queue or get rejected. The order service has no code bug, but it’s been killed by its downstream.

Three Typical Failure Modes

Failure Mode	Trigger	Symptom	Risk Level
Thread pool exhaustion	Slow downstream + no timeout	Service completely unresponsive	High
Connection pool exhaustion	Downstream doesn’t release connections	New requests can’t establish connections	High
Retry storm	Auto-retry after timeout	Traffic amplified N times	Critical
Cascading avalanche	Multi-layer dependency without isolation	Full-chain paralysis	Critical

Retry storms are the most easily overlooked. Suppose A calls B, timeout 3 seconds, retry 3 times. When B slows down, each original request from A becomes 3 requests hitting B. If B also calls C and retries 3 times — one request from C amplifies to 9 requests hitting B. The traffic amplification effect is terrifying. This is why I strongly recommend: retries and circuit breaking must work together — don’t retry when the circuit is open.

Rate Limiting: Caging the Traffic

Rate Limiting Algorithm Selection

Four mainstream algorithms, each with its use case:

Algorithm	Principle	Pros	Cons	Use Case
Fixed window counter	Count within fixed window	Simple	Boundary spike	Low precision
Sliding window	Smooth counting across sub-windows	High precision	Slightly complex	General purpose
Leaky bucket	Process requests at constant rate	Absolutely smooth flow	Can’t handle bursts	Protecting weak downstream
Token bucket	Issue tokens at fixed rate	Allows burst traffic	Needs tuning	API gateway entry

The difference between leaky bucket and token bucket is most commonly confused. Think of it this way:

Leaky bucket: A faucet keeps filling water, the bucket has a small hole at the bottom that leaks at a constant rate. If water comes in too fast, it overflows and gets discarded. No matter how urgent you are, the outflow rate never changes.
Token bucket: The system puts tokens into the bucket at a fixed rate. Requests need to grab a token first; if none available, they wait. The bucket can accumulate tokens, so short burst traffic can be handled.

My recommendation: use token bucket at API gateway entry (allows bursts), leaky bucket for protecting weak downstream (absolutely smooth). In practice, token bucket is sufficient for most scenarios; leaky bucket is for cases where no bursts are allowed at all (like writing to a database).

Go Token Bucket Implementation

Here’s a production-ready Go token bucket implementation. I use it on my ops API gateway, handling millions of requests daily without issues:

package ratelimit

import (
	"context"
	"sync"
	"time"
)

// TokenBucket token bucket rate limiter
type TokenBucket struct {
	mu         sync.Mutex
	rate       float64   // token refill rate (tokens/second)
	burst      float64   // bucket capacity (max tokens)
	tokens     float64   // current token count
	lastUpdate time.Time // last token refill time
}

// NewTokenBucket creates a rate limiter
// rate: tokens per second
// burst: max tokens in bucket (allowed burst)
func NewTokenBucket(rate, burst float64) *TokenBucket {
	return &TokenBucket{
		rate:       rate,
		burst:      burst,
		tokens:     burst, // start with full bucket
		lastUpdate: time.Now(),
	}
}

// Allow tries to take 1 token, non-blocking
func (tb *TokenBucket) Allow() bool {
	return tb.AllowN(1)
}

// AllowN tries to take n tokens, non-blocking
func (tb *TokenBucket) AllowN(n float64) bool {
	tb.mu.Lock()
	defer tb.mu.Unlock()

	now := time.Now()
	// Calculate elapsed time and refill tokens
	elapsed := now.Sub(tb.lastUpdate).Seconds()
	tb.tokens += elapsed * tb.rate
	// Tokens can't exceed bucket capacity
	if tb.tokens > tb.burst {
		tb.tokens = tb.burst
	}
	tb.lastUpdate = now

	// Check if enough tokens
	if tb.tokens >= n {
		tb.tokens -= n
		return true
	}
	return false
}

// Wait blocks until a token is available or context expires
func (tb *TokenBucket) Wait(ctx context.Context) error {
	for {
		if tb.Allow() {
			return nil
		}
		select {
		case <-ctx.Done():
			return ctx.Err()
		case <-time.After(10 * time.Millisecond):
			// Retry every 10ms, avoid busy-waiting
		}
	}
}

Key points of this implementation:

Lazy token refill: Instead of running a goroutine to periodically add tokens, it calculates the refill based on elapsed time on each request. This avoids extra goroutine overhead and is more stable under high concurrency.
Initial full bucket: When the service starts, the bucket is full, handling startup burst traffic.
Mutex protection: sync.Mutex ensures thread safety. For higher performance, you could use atomic operations, but code complexity increases.

Distributed Rate Limiting: When Single-Node Isn’t Enough

Single-node rate limiting only protects individual instances. If your service has 10 replicas, each limited to 1000 QPS, theoretically the whole service can handle 10000 QPS. But if the gateway doesn’t do global rate limiting, 10000 QPS might overwhelm one instance due to uneven distribution.

The core idea of distributed rate limiting: put the counter in Redis, all instances share one counter.

package ratelimit

import (
	"context"
	"fmt"
	"time"

	"github.com/redis/go-redis/v9"
)

// RedisRateLimiter distributed rate limiter based on Redis
// Uses sliding window algorithm
type RedisRateLimiter struct {
	client *redis.Client
	key    string        // rate limit key, e.g. "rate_limit:order_service"
	rate   int           // max requests within window
	window time.Duration // window size
}

func NewRedisRateLimiter(client *redis.Client, key string, rate int, window time.Duration) *RedisRateLimiter {
	return &RedisRateLimiter{
		client: client,
		key:    key,
		rate:   rate,
		window: window,
	}
}

// Allow uses Redis Lua script for atomicity
func (r *RedisRateLimiter) Allow(ctx context.Context) (bool, error) {
	// Lua script: atomically remove old records + count + decide
	script := `
		local key = KEYS[1]
		local now = tonumber(ARGV[1])
		local window = tonumber(ARGV[2])
		local rate = tonumber(ARGV[3])

		-- Remove records outside the window
		redis.call('ZREMRANGEBYSCORE', key, 0, now - window)

		-- Get current count within window
		local count = redis.call('ZCARD', key)

		if count < rate then
			-- Under limit, record this request
			redis.call('ZADD', key, now, now .. ':' .. math.random())
			redis.call('PEXPIRE', key, window)
			return 1
		else
			return 0
		end
	`

	now := time.Now().UnixMilli()
	result, err := r.client.Eval(ctx, script, []string{r.key},
		now, r.window.Milliseconds(), r.rate).Int()
	if err != nil {
		return false, fmt.Errorf("redis eval failed: %w", err)
	}
	return result == 1, nil
}

Production considerations:

Fallback when Redis is unavailable: If Redis goes down, should rate limiting pass through or reject? I recommend pass-through + degrade to single-node limiting. Rate limiting exists to protect the system, but if the Redis it depends on goes down and all requests get rejected, that’s worse than no rate limiting at all.
Lua script is mandatory: Without it, the two-step ZCARD + ZADD has race conditions. Lua scripts execute atomically in Redis.
Sorted Set memory overhead: Each request stores a record in the ZSet. At high QPS, memory grows fast. If QPS exceeds 100K, consider switching to Redis INCR + EXPIRE — slightly less precise but much smaller memory footprint.

Sentinel Rate Limiting in Practice

Alibaba’s Sentinel is the most popular traffic control component in the Java ecosystem. Compared to hand-written rate limiters, Sentinel’s advantage is its built-in dashboard, hot parameter limiting, and dynamic rule adjustment.

// Sentinel rate limiting rule configuration
FlowRule rule = new FlowRule();
rule.setResource("orderService.createOrder");
rule.setGrade(RuleConstant.FLOW_GRADE_QPS);  // QPS mode
rule.setCount(1000);                           // max 1000 per second
rule.setLimitApp("default");                    // applies to all sources

// Hot parameter limiting: per-user-ID limiting, max 10/s per user
ParamFlowRule paramRule = new ParamFlowRule("orderService.createOrder")
    .setParamIdx(0)        // use first parameter as limiting dimension
    .setGrade(RuleConstant.FLOW_GRADE_QPS)
    .setCount(10);          // per-user QPS limit
// VIP users get 100 QPS
paramRule.setParamItem(new ParamFlowItem()
    .setObject("user_vip_123")
    .setClassType(String.class.getName())
    .setCount(100));

// Load rules
FlowRuleManager.loadRules(Collections.singletonList(rule));
ParamFlowRuleManager.loadRules(Collections.singletonList(paramRule));

// Usage in business code
@SentinelResource(value = "orderService.createOrder",
    blockHandler = "createOrderBlockHandler")
public Order createOrder(String userId, OrderRequest req) {
    // normal business logic
    return orderRepository.save(req);
}

// Fallback handler when rate limited
public Order createOrderBlockHandler(String userId, OrderRequest req,
    BlockException ex) {
    // Return default order or throw friendly message
    throw new ServiceException("System busy, please retry later", 429);
}

Hot parameter limiting is Sentinel’s killer feature. Regular limiting is per-resource (“createOrder API allows 1000/s”), while hot parameter limiting is per-user (“User A max 10/s, VIP users up to 100/s”). This is especially useful in multi-tenant SaaS.

Circuit Breaking: Installing a Fuse for Failures

Circuit Breaker State Machine

A circuit breaker has three states, forming a finite state machine:

              failure rate exceeds threshold
  ┌─────────┐ ──────────────→ ┌─────────┐
  │ CLOSED  │                │  OPEN   │
  │ (normal)│ ←───────────── │ (tripped)│
  └─────────┘  probe success  └─────────┘
       ↑                        │
       │                        │ wait for cooldown
       │                        ↓
       │                   ┌──────────┐
       └────────────────── │HALF_OPEN │
           probe success     │ (probing)│
                           └──────────┘
                              │ probe failure
                              ↓
                           ┌─────────┐
                           │  OPEN   │
                           └─────────┘

State meanings:

CLOSED: Normal state, requests pass through. The breaker tracks success rate and slow call ratio in the background.
OPEN: Tripped state, all requests are immediately rejected without calling the downstream. Degradation logic executes. After the cooldown period, it transitions to half-open.
HALF_OPEN: Probing state, a small number of requests are allowed through to check if the downstream has recovered. Success → CLOSED; failure → OPEN.

Resilience4j Circuit Breaker Configuration

Resilience4j is Hystrix’s successor (Hystrix is no longer maintained). It’s modular — you only include the modules you need:

<!-- pom.xml: only include needed modules -->
<dependencies>
    <dependency>
        <groupId>io.github.resilience4j</groupId>
        <artifactId>resilience4j-circuitbreaker</artifactId>
        <version>2.2.0</version>
    </dependency>
    <dependency>
        <groupId>io.github.resilience4j</groupId>
        <artifactId>resilience4j-ratelimiter</artifactId>
        <version>2.2.0</version>
    </dependency>
    <dependency>
        <groupId>io.github.resilience4j</groupId>
        <artifactId>resilience4j-bulkhead</artifactId>
        <version>2.2.0</version>
    </dependency>
</dependencies>

Production-grade circuit breaker configuration:

# application.yml
resilience4j:
  circuitbreaker:
    configs:
      default:
        # Sliding window configuration
        slidingWindowType: COUNT_BASED       # count-based statistics
        slidingWindowSize: 100               # last 100 requests
        minimumNumberOfCalls: 20            # need at least 20 calls before evaluating
        # Failure thresholds
        failureRateThreshold: 50             # trip at >= 50% failure rate
        slowCallRateThreshold: 60            # trip at >= 60% slow call ratio
        slowCallDurationThreshold: 2s        # calls over 2s count as slow
        # State transitions
        waitDurationInOpenState: 30s         # wait 30s after tripping before half-open
        permittedNumberOfCallsInHalfOpenState: 10  # allow 10 probe calls in half-open
        # Which exceptions count as failures
        recordExceptions:
          - java.io.IOException
          - java.util.concurrent.TimeoutException
          - feign.FeignException.GatewayTimeout
        ignoreExceptions:
          - com.example.BusinessException    # business exceptions don't count
    instances:
      paymentService:
        baseConfig: default
        # Payment service: lower thresholds — payment chain is more sensitive
        failureRateThreshold: 30
        slowCallDurationThreshold: 1s
      userService:
        baseConfig: default
        # User service: allow higher failure rate since degradation is possible
        failureRateThreshold: 60

How to tune these parameters? Here’s a set of empirical values based on managing 50+ microservices:

Parameter	Recommended (Core)	Recommended (Non-core)	Rationale
slidingWindowSize	100	60	Too small: insufficient samples; too large: slow reaction
failureRateThreshold	30-50%	50-70%	Core services need more sensitivity
slowCallDurationThreshold	1-2s	3-5s	2-3x downstream P99
waitDurationInOpenState	30-60s	15-30s	Wait for downstream recovery
half-open probes	10	5	Not too many

Pitfall: minimumNumberOfCalls is often overlooked. If set to 0, right after startup with only one request that fails, the failure rate is 100% — immediately tripping the breaker. Setting it to 20 means at least 20 calls are needed before evaluation, preventing small-sample false positives.

Go Circuit Breaker Implementation

If your services are in Go (like our entire ops platform), you can use sony/gobreaker or implement your own. Here’s a lean but usable implementation:

package circuitbreaker

import (
	"errors"
	"sync"
	"time"
)

// State circuit breaker state
type State int

const (
	StateClosed   State = iota // normal
	StateOpen                  // tripped
	StateHalfOpen              // probing
)

// Settings circuit breaker configuration
type Settings struct {
	Name          string
	MaxRequests   uint32        // probe requests allowed in half-open
	Interval      time.Duration // stats window in closed state
	Timeout       time.Duration // cooldown in open state
	ReadyToTrip   func(counts Counts) bool // trip condition
	OnStateChange func(name string, from, to State)
}

// Counts request statistics
type Counts struct {
	Requests            uint32
	TotalSuccesses      uint32
	TotalFailures       uint32
	ConsecutiveFailures uint32
}

// CircuitBreaker circuit breaker
type CircuitBreaker struct {
	mu          sync.Mutex
	settings    Settings
	state       State
	generation  uint64
	expiry      time.Time
	counts      Counts
}

// NewCircuitBreaker creates a circuit breaker
func NewCircuitBreaker(s Settings) *CircuitBreaker {
	if s.ReadyToTrip == nil {
		// Default: trip after 5 consecutive failures
		s.ReadyToTrip = func(c Counts) bool {
			return c.ConsecutiveFailures >= 5
		}
	}
	cb := &CircuitBreaker{settings: s}
	cb.toNewGeneration(time.Now())
	return cb
}

// Execute runs a request with circuit breaking logic
func (cb *CircuitBreaker) Execute(req func() error) error {
	generation, err := cb.beforeRequest()
	if err != nil {
		return err
	}

	err = req()

	cb.afterRequest(generation, err)
	return err
}

func (cb *CircuitBreaker) beforeRequest() (uint64, error) {
	cb.mu.Lock()
	defer cb.mu.Unlock()

	now := time.Now()
	state, generation := cb.currentState(now)

	if state == StateOpen {
		// Tripped — reject without executing
		return generation, ErrCircuitOpen
	}
	return generation, nil
}

func (cb *CircuitBreaker) afterRequest(before uint64, err error) {
	cb.mu.Lock()
	defer cb.mu.Unlock()

	now := time.Now()
	state, generation := cb.currentState(now)
	if before != generation {
		// State changed, discard this count
		return
	}

	cb.counts.Requests++
	if err != nil {
		cb.counts.TotalFailures++
		cb.counts.ConsecutiveFailures++
	} else {
		cb.counts.TotalSuccesses++
		cb.counts.ConsecutiveFailures = 0
	}

	// Check if state should change
	if cb.state == StateClosed && cb.settings.ReadyToTrip(cb.counts) {
		cb.setState(StateOpen, now)
	} else if cb.state == StateHalfOpen && cb.counts.Requests >= cb.settings.MaxRequests {
		// Probes exhausted — decide based on results
		if cb.counts.TotalSuccesses > 0 {
			cb.setState(StateClosed, now)
		} else {
			cb.setState(StateOpen, now)
		}
	}
}

var ErrCircuitOpen = errors.New("circuit breaker is open")

func (cb *CircuitBreaker) currentState(now time.Time) (State, uint64) {
	switch cb.state {
	case StateClosed:
		if !cb.expiry.IsZero() && now.After(cb.expiry) {
			cb.toNewGeneration(now)
		}
	case StateOpen:
		if now.After(cb.expiry) {
			// Cooldown over — enter half-open
			cb.setState(StateHalfOpen, now)
		}
	}
	return cb.state, cb.generation
}

func (cb *CircuitBreaker) setState(state State, now time.Time) {
	if cb.state == state {
		return
	}
	prev := cb.state
	cb.state = state
	cb.toNewGeneration(now)
	if cb.settings.OnStateChange != nil {
		cb.settings.OnStateChange(cb.settings.Name, prev, state)
	}
}

func (cb *CircuitBreaker) toNewGeneration(now time.Time) {
	cb.generation++
	cb.counts = Counts{}

	switch cb.state {
	case StateClosed:
		if cb.settings.Interval == 0 {
			cb.expiry = time.Time{} // never expires
		} else {
			cb.expiry = now.Add(cb.settings.Interval)
		}
	case StateOpen:
		cb.expiry = now.Add(cb.settings.Timeout)
	case StateHalfOpen:
		cb.expiry = time.Time{} // never expires, waits for probe completion
	}
}

Usage:

cb := circuitbreaker.NewCircuitBreaker(circuitbreaker.Settings{
    Name:        "payment-service",
    MaxRequests: 5,                    // 5 probes in half-open
    Interval:    60 * time.Second,     // 60s stats window in closed
    Timeout:     30 * time.Second,     // 30s cooldown in open
    ReadyToTrip: func(c circuitbreaker.Counts) bool {
        // Trip at >50% failure rate with at least 10 requests
        if c.Requests < 10 {
            return false
        }
        failureRate := float64(c.TotalFailures) / float64(c.Requests)
        return failureRate >= 0.5
    },
    OnStateChange: func(name string, from, to circuitbreaker.State) {
        log.Printf("[CircuitBreaker] %s: %v → %v", name, from, to)
        // Push alert on state change
        alert.Send(fmt.Sprintf("Circuit breaker %s: %v → %v", name, from, to))
    },
})

// Usage in business code
err := cb.Execute(func() error {
    return paymentClient.Charge(ctx, order)
})
if errors.Is(err, circuitbreaker.ErrCircuitOpen) {
    // Circuit open — execute degradation
    return fallbackPayment(ctx, order)
}

Envoy Proxy-Layer Circuit Breaking

If you’re using Service Mesh (like Istio), circuit breaking can be configured at the mesh layer via Envoy — no code changes needed. This is my recommended approach: governance logic sinks to the infrastructure layer, keeping business code clean.

# Istio DestinationRule for Envoy circuit breaking
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: payment-service-cb
spec:
  host: payment-service.default.svc.cluster.local
  trafficPolicy:
    outlierDetection:
      # Eject after 5 consecutive 5xx errors
      consecutive5xxErrors: 5
      # Eject after 3 consecutive gateway errors (502/503/504)
      consecutiveGatewayErrors: 3
      # Detection interval
      interval: 30s
      # Base ejection duration
      baseEjectionTime: 30s
      # Max ejection ratio: at most 50% of instances
      maxEjectionPercent: 50
      # Min healthy ratio: keep at least 50% available
      minHealthPercent: 50
    connectionPool:
      tcp:
        maxConnections: 100        # max connections
        connectTimeout: 2s         # connection timeout
      http:
        http1MaxPendingRequests: 50    # max pending requests
        http2MaxRequests: 200          # max concurrent requests
        maxRequestsPerConnection: 10   # max requests per connection
        maxRetries: 2                  # max retries

What’s the difference between Envoy and Resilience4j circuit breaking?

Dimension	Envoy (Service Mesh)	Resilience4j (SDK)
Intrusiveness	Zero — config only	Requires code annotations
Granularity	Instance-level (evict unhealthy)	Request-level (per resource)
Language-agnostic	Yes	Java only
Dynamic adjustment	Yes (K8s CRD hot update)	Yes but needs config center
Use case	Multi-language microservices	Single-language Java ecosystem

My recommendation: If microservices span 3+ languages, use Service Mesh layer circuit breaking. If all Java, Resilience4j is more flexible. If all Go, use gobreaker or implement your own. (See Related: Istio Service Mesh Getting Started)

Degradation: Making the System Fail Gracefully

Degradation is the last line of defense. Rate limiting and circuit breaking both “reject requests” — degradation “provides alternatives.” A good degradation strategy makes failures nearly invisible to users.

Degradation Strategy Classification

Type	Trigger	Handling	Example
Return default value	Downstream unavailable	Return preset default response	Recommendation list down, return trending items
Return cached	Data source unavailable	Return stale data	Product details from cache, tolerate staleness
Simplify flow	Core path affected	Skip non-core steps	Skip coupon validation during checkout
Async-ify	Sync call timeout	Convert to async processing	Payment becomes async, return “processing”
Feature off	Non-core feature failure	Disable feature entirely	Comments service down, hide comment section

Design Principles

Separate core and non-core paths. Checkout and payment are core; recommendations and comments are non-core. Non-core can be degraded anytime; core paths must be protected.
Degradation must be pre-tested. Don’t wait for a failure to discover your degradation code has bugs. Chaos engineering exercises must include degradation validation.
Degradation needs switches. Toggle dynamically through config center — don’t hardcode. One command to degrade, one command to recover.
Degradation is not an error. The return should be a “usable second-best result,” not a 500 error code.

Degradation Implementation Example

Degradation chain in Go:

// PaymentService with degradation chain
type PaymentService struct {
	primaryPayment  PaymentClient   // primary payment channel
	fallbackPayment PaymentClient   // fallback payment channel
	cache           CacheClient     // cache
	cb              *circuitbreaker.CircuitBreaker // circuit breaker
}

// Charge with three-tier degradation strategy
func (s *PaymentService) Charge(ctx context.Context, order *Order) (*PaymentResult, error) {
	// Strategy 1: Primary channel normal → normal payment
	result, err := s.cb.Execute(func() error {
		result, err := s.primaryPayment.Charge(ctx, order)
		if err == nil {
			order.Result = result
		}
		return err
	})

	if err == nil {
		return result, nil
	}

	// Strategy 2: Primary circuit open → switch to fallback
	if errors.Is(err, circuitbreaker.ErrCircuitOpen) {
		log.Printf("primary payment circuit open, switching to fallback")
		result, err := s.fallbackPayment.Charge(ctx, order)
		if err == nil {
			return result, nil
		}
	}

	// Strategy 3: Both channels down → async processing
	// Return "processing", queue for background retry
	if order.Amount < 10000 { // small amounts can go async
		if err := s.enqueueAsyncPayment(order); err != nil {
			return nil, fmt.Errorf("payment unavailable, please retry later")
		}
		return &PaymentResult{
			Status:  "processing",
			Message: "Payment processing, please check results later",
		}, nil
	}

	// Large amounts can't go async — reject
	return nil, fmt.Errorf("payment service unavailable")
}

This code implements three-tier degradation: primary → fallback → async. Note that large orders (over 10,000) can’t use the async path — because if async payment ultimately fails, the refund process is complex and risky at high amounts. This was a real decision I made when building a payment system.

Configurable Degradation Switches

The most important capability in production is “one-click degradation.” Managed through config center (like Nacos, Apollo):

// Degradation config from config center
type DegradeConfig struct {
	// Feature toggles
	EnableRecommendation bool `json:"enable_recommendation"`
	EnableComments       bool `json:"enable_comments"`
	EnableCoupons        bool `json:"enable_coupons"`
	// Degradation strategy
	RecommendationDegrade string `json:"recommendation_degrade"` // "default" | "cache" | "disabled"
	// Rate limit thresholds (dynamically adjustable)
	OrderRateLimit int `json:"order_rate_limit"`
}

// DegradeManager
type DegradeManager struct {
	config *DegradeConfig
	mu     sync.RWMutex
}

// CheckFeature checks if a feature is enabled
func (dm *DegradeManager) CheckFeature(feature string) bool {
	dm.mu.RLock()
	defer dm.mu.RUnlock()
	switch feature {
	case "recommendation":
		return dm.config.EnableRecommendation
	case "comments":
		return dm.config.EnableComments
	case "coupons":
		return dm.config.EnableCoupons
	}
	return true // default enabled
}

// UpdateConfig hot-reload callback from config center
func (dm *DegradeManager) UpdateConfig(newConfig *DegradeConfig) {
	dm.mu.Lock()
	dm.config = newConfig
	dm.mu.Unlock()
	log.Printf("degrade config updated: %+v", newConfig)
}

After config changes, ops staff can one-click disable a feature through the Nacos console — no deployment, no restart needed.

Coordinating All Three: Production Traffic Governance Architecture

Rate limiting, circuit breaking, and degradation aren’t used in isolation — they form a multi-layer defense system. Here’s the architecture I’ve deployed in production:

                        User Request
                           │
                           ▼
                 ┌──────────────────┐
                 │   API Gateway    │  ← Global rate limiting (token bucket)
                 │  (Kong/APISIX)   │     Per-API, per-tenant
                 └────────┬─────────┘
                          │
              ┌───────────┼───────────┐
              ▼           ▼           ▼
         ┌────────┐ ┌────────┐ ┌────────┐
         │ Order  │ │ User   │ │ Search │  ← Service-level limiting (Sentinel/gobreaker)
         │Service │ │Service │ │Service │     Per-interface, per-user
         │        │ │        │ │        │
         │ CB-A   │ │ CB-B   │ │ CB-C   │  ← Circuit breaking on downstream calls
         └───┬────┘ └───┬────┘ └───┬────┘
             │          │          │
             ▼          ▼          ▼
         ┌────────┐ ┌────────┐ ┌────────┐
         │Payment │ │ Auth   │ │Database│  ← Resource-level protection
         │Service │ │Service │ │        │     Connection pool limiting, slow SQL breaking
         │Degrade │ │Degrade │ │        │
         └────────┘ └────────┘ └────────┘

Multi-Layer Defense Responsibilities

Layer	Mechanism	Tool	Configuration
Gateway	Global rate limiting	Kong/APISIX/Nginx	Per-API, QPS = peak × 1.5
Service	Interface rate limiting	Sentinel/Resilience4j	Per-interface + per-user
Call	Circuit breaking + retry	Resilience4j/gobreaker	50% failure trip, max 2 retries
Resource	Connection pool limit	HikariCP/Go sql.DB	Connections = CPU cores × 2 + 1
Business	Degradation switches	Config center	Protect core, degrade non-core

Timeout Hierarchy

Timeouts must decrease from outer to inner layers, otherwise the outer layer times out and retries before the inner layer finishes:

API Gateway timeout 5s
    └─ Order service timeout 3s
        └─ Payment service timeout 2s
            └─ Third-party payment timeout 1.5s

Each layer reserves 0.5-1 second for its own processing. If the innermost timeout is 1.5s, the payment service timeout must be greater than 1.5s (e.g., 2s), otherwise the payment service times out before the third party returns.

Pitfall case: We had a production incident caused by inverted timeout hierarchy. Order service timeout was 3s, payment service timeout was 5s. While payment service was waiting for the third party (not timed out yet), the order service timed out after 3s waiting for payment service, then retried — sending another request to payment service. Two requests processing simultaneously on payment service caused database row lock conflicts, slowing down even more requests. We had to urgently reduce the payment service timeout to 2s to recover.

For alerting strategy design, see Related: Alerting Strategy Design — From Noise to Signal — well-designed alert rules can warn you before circuit breakers trip.

Production Pitfall Records

Pitfall 1: Traffic Surge After Circuit Recovery

When the breaker transitions from OPEN to HALF_OPEN, it allows probe requests. If probes succeed, it immediately switches to CLOSED — and all previously queued requests rush in at once, overwhelming the just-recovered downstream.

Solution: Implement gradual traffic ramp-up after recovery. Instead of going to 100% flow instantly, start with 10%, observe 30 seconds, then 30%, then 100%.

// Gradual recovery: control traffic ratio after breaker recovery
type GradualRecovery struct {
	mu              sync.Mutex
	recoveryPhase   int      // 0=closed, 1=10%, 2=30%, 3=100%
	phaseStartTime  time.Time
}

func (gr *GradualRecovery) Allow() bool {
	gr.mu.Lock()
	defer gr.mu.Unlock()

	ratios := []float64{0.1, 0.3, 1.0} // gradual ratios
	if gr.recoveryPhase >= len(ratios) {
		return true // fully recovered
	}

	// Check if should advance to next phase
	if time.Since(gr.phaseStartTime) > 30*time.Second {
		gr.recoveryPhase++
		gr.phaseStartTime = time.Now()
	}

	// Allow traffic by ratio
	ratio := ratios[gr.recoveryPhase]
	if rand.Float64() < ratio {
		return true
	}
	return false
}

Pitfall 2: Retry Amplification

As mentioned, A calls B with 3 retries, B calls C with 3 retries — traffic amplifies 9x. But there’s a sneakier version:

Scenario: Order service calls inventory service, timeout 2s, retry 3 times. Inventory service is perfectly fine — just a 1.5s GC pause. Order service times out and retries. Inventory service receives 3 identical requests and deducts inventory 3 times.

Solutions:

Retries must be idempotent. Inventory deduction can’t be UPDATE stock SET count = count - 1 directly — use idempotency keys: check if the request ID was already processed, return the previous result if so.
Limit total retries across the chain. Not 3 retries per layer, but 3 retries for the entire chain. Pass remaining retry count through request headers.
Add jitter to retries. Don’t retry immediately — wait a random time (100ms-500ms) to avoid multiple callers retrying simultaneously.

Pitfall 3: Sentinel Rules Not Persisted

Sentinel stores rules in memory by default — they’re lost on restart. After a server restart with rules gone, a traffic peak hit and took down the downstream database.

Solution: Rules must be persisted to a config center (Nacos/Apollo). Sentinel pulls rules on startup, and rule changes are pushed to all instances.

Pitfall 4: No Alerting on Circuit Breaker State Changes

A breaker transitioning from CLOSED to OPEN is an important signal — it means something downstream is wrong. But without alerting, you might not know until users complain.

Solution: Push alerts in the OnStateChange callback. Not just for OPEN — HALF_OPEN and CLOSED too — knowing when recovery happened is equally important. Alert content should include: breaker name, state transition, current failure rate, trigger reason. (See Related: Alert Automation — From Alert Storms to Self-Healing Systems)

Pitfall 5: Envoy Ejection Causing Insufficient Instances

Envoy’s maxEjectionPercent: 50 means at most 50% of instances can be evicted. But if the service has only 2 instances, evicting 1 hits 50%, leaving only 1 instance to handle all traffic.

Solution: Lower the ejection ratio for small-scale services, or set minHealthPercent to 100 (no eviction, only load reduction). Another option: ensure at least 3 instances — easy in K8s environments.

Tool Comparison

Tool	Language	Rate Limiting	Circuit Breaking	Degradation	Dashboard	Use Case
Sentinel	Java	Strong	Strong	Strong	Yes	Java ecosystem first choice
Resilience4j	Java	Medium	Strong	Medium	No	Lightweight, modular
Hystrix	Java	Weak	Strong	Medium	Yes	End-of-life, not recommended
gobreaker	Go	None	Strong	None	No	Go circuit breaking
Envoy/Istio	Agnostic	Strong	Strong	Medium	Yes	Service Mesh approach
Kong/APISIX	Agnostic	Strong	Medium	Weak	Yes	API gateway layer

My selection advice:

Pure Java microservices: Sentinel + Resilience4j. Sentinel for rate limiting and hot parameter control, Resilience4j for circuit breaking.
Pure Go microservices: gobreaker + custom rate limiting. Go ecosystem lacks a Sentinel-level comprehensive solution, but a self-implemented rate limiter + circuit breaker is sufficient.
Multi-language: Istio/Envoy at the mesh layer — zero code intrusion.
Don’t want to change code: API gateway rate limiting + Envoy circuit breaking, zero code intrusion.

Monitoring and Observability

After deploying rate limiting, circuit breaking, and degradation, you must be able to see their operational status. Key metrics to monitor:

Metric	Meaning	Alert Threshold
Rate limit rejection rate	Rejected / total requests	> 5%
Circuit breaker state	OPEN/CLOSED/HALF_OPEN	OPEN > 1min
Circuit trip count	OPEN transitions per unit time	> 3/5min
Degradation trigger count	Degradation logic executions	> 100/min
Recovery time	OPEN → CLOSED duration	> 5min

Prometheus + Grafana monitoring config:

# Prometheus scrape config for Resilience4j metrics
scrape_configs:
  - job_name: 'resilience4j'
    metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['order-service:8080', 'payment-service:8080']

# Circuit breaker state (0=CLOSED, 1=OPEN, 2=HALF_OPEN)
resilience4j_circuitbreaker_state

# Circuit breaker failure rate
resilience4j_circuitbreaker_failure_rate

# Rate of available permissions (rate limiter)
rate(resilience4j_ratelimiter_available_permissions[1m])

# Alert rule: circuit breaker open for over 1 minute
ALERT CircuitBreakerOpen
IF resilience4j_circuitbreaker_state == 1
FOR 1m
LABELS { severity = "critical" }
ANNOTATIONS {
  summary = "Circuit breaker {{ $labels.name }} is open",
  description = "Circuit breaker has been open for over 1 minute, check downstream service status"
}

Capacity Assessment and Load Testing

Before configuring rate limit thresholds, you must know what your system can actually handle. Don’t just pick a number out of thin air.

Testing Methods

Baseline test: Gradually increase load until you find the inflection point (where response time starts to spike). The QPS before the inflection point is your safe water level.
Stress test: Keep increasing load until the system breaks. Record the max QPS before failure. Divide by 2-3 for your rate limit threshold reference.
Mixed workload test: Simulate real traffic ratios (read:write = 8:2), not just single-interface tests.

Capacity Formula

Safe QPS = baseline inflection QPS × safety factor (0.6-0.7)
Rate limit threshold = Safe QPS × instance count × load imbalance factor (0.8)

Example: Single-instance baseline inflection is 2000 QPS, safety factor 0.7, safe QPS = 1400. 10 instances, load imbalance factor 0.8 (accounting for Nginx/K8s uneven distribution), rate limit threshold = 1400 × 10 × 0.8 = 11200 QPS.

For a complete capacity planning methodology, see Related: SLO Design — From Business Goals to Technical Metrics — align rate limit thresholds with SLO targets.

Summary

Microservice rate limiting, circuit breaking, and degradation is not a “install and forget” component — it’s a traffic governance system that requires continuous tuning. Returning to my experience managing 50+ microservices, here are the core principles:

Rate limiting: Token bucket at the gateway to allow bursts, Sentinel at the service layer for fine-grained control. Distributed limiting with Redis + Lua for atomicity. Thresholds must be based on load test data — never guess.

Circuit breaking: Resilience4j (Java) or gobreaker (Go) for application-layer, Envoy/Istio for mesh-layer. The key parameters are failureRateThreshold and slowCallDurationThreshold — the former determines sensitivity, the latter defines what counts as slow. minimumNumberOfCalls must be set to avoid small-sample false positives.

Degradation: Separate core and non-core paths. Degradation strategies must be pre-tested, with switches managed through config center. The return is a “second-best result,” not an error code.

Coordination: Multi-layer defense from outside in — gateway limiting, service limiting, call breaking, resource protection, business degradation. Timeouts decrease from outer to inner. Retries must be idempotent with jitter.

Observability: Circuit breaker state changes must trigger alerts. Rate limit rejection rate, circuit trip count, and degradation trigger count all need monitoring dashboards and alert rules.

One final lesson: after deploying rate limiting, circuit breaking, and degradation, you must run chaos engineering exercises — deliberately kill a service and see if your defense system works as expected. There’s a world of difference between paper plans and real failures. Don’t wait until 3 AM when alerts wake you up to discover your circuit breaker config is broken. For systematic approaches to failure drills, see Related: SRE Incident Preparedness and Drills.

References & Acknowledgments

This article referenced the following materials during writing. Thanks to the original authors for their contributions:

Circuit Breaker Design Pattern — GitHub Topics, a collection of mainstream circuit breaker open-source projects including Sentinel, Resilience4j, gobreaker, etc.
Spring Boot Microservice Circuit Breaking: 2026 Latest Guide — CSDN, provided detailed explanation of Resilience4j circuit breaker state machine model
Timeout, Retry, and Circuit Breaking in Microservices — CSDN, provided analysis of timeout configuration and retry storms
Degradation, Rate Limiting, and Graceful Failover for LLM Integration in Microservices — CSDN, provided multi-layer defense architecture design ideas
Resilience4j Introduction — cnblogs, provided comparison and configuration of Resilience4j modules
Performance Modeling of Microservices with Circuit Breakers using Stochastic Petri Nets — IEEE, academic paper studying quantified impact of circuit breakers on microservice performance, measured 71.4% reduction in request discard rate
Self-Adaptive Circuit Breaker Open-State Interval for Enhancing Resiliency of Microservices — IEEE, academic paper proposing adaptive circuit breaker cooldown time

Overview#

Avalanche Effect: Why Microservices Fail Harder Than Monoliths#

Fault Propagation Path#

Three Typical Failure Modes#

Rate Limiting: Caging the Traffic#

Rate Limiting Algorithm Selection#

Go Token Bucket Implementation#

Distributed Rate Limiting: When Single-Node Isn’t Enough#

Sentinel Rate Limiting in Practice#

Circuit Breaking: Installing a Fuse for Failures#

Circuit Breaker State Machine#

Resilience4j Circuit Breaker Configuration#

Go Circuit Breaker Implementation#

Envoy Proxy-Layer Circuit Breaking#

Degradation: Making the System Fail Gracefully#

Degradation Strategy Classification#

Design Principles#

Degradation Implementation Example#

Configurable Degradation Switches#

Coordinating All Three: Production Traffic Governance Architecture#

Multi-Layer Defense Responsibilities#

Timeout Hierarchy#

Production Pitfall Records#

Pitfall 1: Traffic Surge After Circuit Recovery#

Pitfall 2: Retry Amplification#

Pitfall 3: Sentinel Rules Not Persisted#

Pitfall 4: No Alerting on Circuit Breaker State Changes#

Pitfall 5: Envoy Ejection Causing Insufficient Instances#

Tool Comparison#

Monitoring and Observability#

Capacity Assessment and Load Testing#

Testing Methods#

Capacity Formula#

Summary#

References & Acknowledgments#