淘客系统的容灾演练与恢复：Java Chaos Monkey模拟节点故障下的服务降级与快速切换实践

淘客系统的容灾演练与恢复：Java Chaos Monkey模拟节点故障下的服务降级与快速切换实践

大家好，我是 微赚淘客系统3.0 的研发者省赚客！

高可用是淘客系统的核心要求。为验证在数据库宕机、缓存失效或服务实例崩溃等异常场景下系统的自愈能力，我们引入 Chaos Monkey 思想，结合 Spring Boot Actuator 与自研故障注入模块，在预发环境主动制造故障，并通过 Sentinel 与多活架构实现服务降级与快速切换。

Chaos Monkey 故障注入模块设计

我们基于 Spring Boot 开发轻量级 Chaos Agent，支持运行时注入延迟、异常或服务中断：

packagejuwatech.cn.chaos;importorg.springframework.beans.factory.annotation.Value;importorg.springframework.stereotype.Component;importjava.util.Random;@ComponentpublicclassChaosInjector{@Value("${chaos.enabled:false}")privatebooleanchaosEnabled;@Value("${chaos.failure.rate:0.1}")privatedoublefailureRate;privatefinalRandomrandom=newRandom();publicvoidmaybeThrowException(StringserviceName){if(!chaosEnabled)return;if(random.nextDouble()<failureRate){juwatech.cn.util.AsyncLogger.logAsync("Chaos injected: killing "+serviceName);thrownewRuntimeException("Simulated failure in "+serviceName);}}publicvoidmaybeDelay(longmaxDelayMs){if(!chaosEnabled)return;if(random.nextDouble()<failureRate){longdelay=random.nextInt((int)maxDelayMs);try{Thread.sleep(delay);}catch(InterruptedExceptione){Thread.currentThread().interrupt();}}}}

在关键服务中嵌入故障点：

packagejuwatech.cn.service;importjuwatech.cn.chaos.ChaosInjector;importorg.springframework.stereotype.Service;@ServicepublicclassCouponRemoteService{privatefinalChaosInjectorchaosInjector;publicCouponRemoteService(ChaosInjectorchaosInjector){this.chaosInjector=chaosInjector;}publicStringfetchCouponFromRemote(StringitemId){// 模拟调用第三方接口前注入故障chaosInjector.maybeThrowException("coupon-remote-api");chaosInjector.maybeDelay(2000);// 实际调用逻辑（略）return"CPN_"+itemId;}}

Sentinel 服务降级规则配置

当远程服务不可用时，自动触发 fallback：

packagejuwatech.cn.fallback;importcom.alibaba.csp.sentinel.slots.block.BlockException;publicclassCouponFallback{publicstaticStringremoteFallback(StringitemId,BlockExceptionex){juwatech.cn.util.AsyncLogger.logAsync("Fallback triggered for item: "+itemId);return"DEFAULT_COUPON_5YUAN";// 返回兜底优惠券}publicstaticStringdbFallback(StringuserId,BlockExceptionex){return"[]";// 返回空列表}}

通过 Sentinel 注解绑定资源与降级策略：

packagejuwatech.cn.service;importcom.alibaba.csp.sentinel.annotation.SentinelResource;importjuwatech.cn.fallback.CouponFallback;importorg.springframework.stereotype.Service;@ServicepublicclassCouponQueryService{@SentinelResource(value="fetchCouponFromRemote",blockHandler="remoteFallback",blockHandlerClass=CouponFallback.class)publicStringqueryCoupon(StringitemId){returnnewCouponRemoteService(newjuwatech.cn.chaos.ChaosInjector()).fetchCouponFromRemote(itemId);}}

动态加载降级规则（启动时初始化）：

packagejuwatech.cn.config;importcom.alibaba.csp.sentinel.slots.block.degrade.DegradeRule;importcom.alibaba.csp.sentinel.slots.block.degrade.DegradeRuleManager;importorg.springframework.context.annotation.Configuration;importjavax.annotation.PostConstruct;importjava.util.Collections;@ConfigurationpublicclassSentinelDegradeConfig{@PostConstructpublicvoidinitDegradeRules(){DegradeRulerule=newDegradeRule("fetchCouponFromRemote").setGrade(com.alibaba.csp.sentinel.slots.block.degrade.DegradeRule.RT)// 基于响应时间.setCount(200)// 超过200ms视为慢调用.setTimeWindow(10)// 熔断10秒.setMinRequestAmount(5)// 最小请求数.setSlowRatioThreshold(0.5);// 慢调用比例阈值DegradeRuleManager.loadRules(Collections.singletonList(rule));}}

多活数据库快速切换机制

主库故障时，自动切换至只读副本。我们封装数据源路由逻辑：

packagejuwatech.cn.datasource;importorg.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;publicclassRoutingDataSourceextendsAbstractRoutingDataSource{@OverrideprotectedObjectdetermineCurrentLookupKey(){returnDataSourceContext.getDataSourceType();}}

上下文管理：

packagejuwatech.cn.datasource;publicclassDataSourceContext{privatestaticfinalThreadLocal<String>contextHolder=newThreadLocal<>();publicstaticvoidsetMaster(){contextHolder.set("master");}publicstaticvoidsetSlave(){contextHolder.set("slave");}publicstaticStringgetDataSourceType(){returncontextHolder.get()!=null?contextHolder.get():"master";}}

在数据库操作前检测主库健康状态：

packagejuwatech.cn.service;importjuwatech.cn.datasource.DataSourceContext;importorg.springframework.stereotype.Service;@ServicepublicclassOrderService{privatefinaljuwatech.cn.db.HealthCheckerhealthChecker;publicOrderService(juwatech.cn.db.HealthCheckerhealthChecker){this.healthChecker=healthChecker;}publicvoidcreateOrder(StringorderId){if(!healthChecker.isMasterDbHealthy()){DataSourceContext.setSlave();// 临时切到从库（仅限查询）thrownewIllegalStateException("Master DB down, write operation blocked");}DataSourceContext.setMaster();// 执行写入}}

健康检查实现：

packagejuwatech.cn.db;importorg.springframework.jdbc.core.JdbcTemplate;importorg.springframework.stereotype.Component;@ComponentpublicclassHealthChecker{privatefinalJdbcTemplatemasterJdbcTemplate;publicHealthChecker(JdbcTemplatemasterJdbcTemplate){this.masterJdbcTemplate=masterJdbcTemplate;}publicbooleanisMasterDbHealthy(){try{masterJdbcTemplate.queryForObject("SELECT 1",Integer.class);returntrue;}catch(Exceptione){juwatech.cn.util.AsyncLogger.logAsync("Master DB health check failed: "+e.getMessage());returnfalse;}}}

演练流程与指标验证

1. 启动 Chaos Monkey（chaos.enabled=true）
2. 触发高频查券请求
3. 监控 Prometheus 指标：
   • sentinel_block_requests_total（降级次数）
   • http_server_requests_seconds_count{status="500"}（错误率）
4. 验证 Kibana 中 fallback 日志是否正常记录
5. 模拟主库宕机，观察数据源切换日志与业务影响范围

通过上述机制，系统在节点故障下仍能提供基础服务能力，保障核心链路可用性。

本文著作权归 微赚淘客系统3.0 研发团队，转载请注明出处！