Recently, I encountered some production problems involving hystrix. I want to understand the underlying principle. Hystrix makes a lot of use of Rxjava's responsive programming. It's a little hard to understand because you don't understand Rxjava.
Basic preparation
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<properties>
<maven.compiler.source>8</maven.compiler.source>
<maven.compiler.target>8</maven.compiler.target>
<spring-boot.version>2.3.12.RELEASE</spring-boot.version>
<spring-cloud.version>Hoxton.SR11</spring-cloud.version>
</properties>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-netflix-hystrix</artifactId>
</dependency>
<dependencyManagement>
<dependencies>
<!--Import dependent pom file-->
<!--First in order, first introduced-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
<version>${spring-boot.version}</version>
<type>pom</type>
<scope>provided</scope>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>${spring-cloud.version}</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
</project>
Enable hystrix
//Add on main class
@EnableHystrix
Let's look at the source code
@Target({ElementType.TYPE})
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@EnableCircuitBreaker
public @interface EnableHystrix {
}
final@EnableHystrix Inherited@EnableCircuitBreaker
@Target(ElementType.TYPE)
@Retention(RetentionPolicy.RUNTIME)
@Documented
@Inherited
@Import(EnableCircuitBreakerImportSelector.class)
public @interface EnableCircuitBreaker {
}
According to the previous model, we must analyze the enablercircuitbreaker importselector. As for how to execute @ Import, we won't explain it too much here. We've talked about it before. If you're interested, see here: the specific code is in configurationclassparser String [] importclassnames = selector in processimports selectImports(currentSourceClass.getMetadata());
Import HystrixCircuitBreakerConfiguration through enablercircuitbreakermportselector
//The priority is not particularly high. After all, it is the aspect of runtime
@Order(Ordered.LOWEST_PRECEDENCE - 100)
public class EnableCircuitBreakerImportSelector
extends SpringFactoryImportSelector<EnableCircuitBreaker> {
@Override
protected boolean isEnabled() {
//It is on by default
return getEnvironment().getProperty("spring.cloud.circuit.breaker.enabled",
Boolean.class, Boolean.TRUE);
}
}
After Import, the corresponding org. Class will be loaded from the configuration file through selectImports springframework. cloud. client. circuitbreaker. EnableCircuitBreaker
public abstract class SpringFactoryImportSelector<T>
implements DeferredImportSelector, BeanClassLoaderAware, EnvironmentAware {
//Resolve generic T to this annotationClass
protected SpringFactoryImportSelector() {
this.annotationClass = (Class<T>) GenericTypeResolver.resolveTypeArgument(this.getClass(), SpringFactoryImportSelector.class);
}
@Override
public String[] selectImports(AnnotationMetadata metadata) {
//Through springfactoriesloader Loadfactorynames from spring Get org. Org from factories springframework. cloud. client. circuitbreaker. Configuration corresponding to enablercircuitbreaker
List<String> factories = new ArrayList<>(new LinkedHashSet<>(SpringFactoriesLoader.loadFactoryNames(this.annotationClass, this.beanClassLoader)));
}
}
Finally, the corresponding is 2.2 8.RELEASE/spring-cloud-netflix-hystrix-2.2. 8.RELEASE. jar!/ META-INF/spring. factories
spring. Contents of the factories file
org.springframework.boot.autoconfigure.EnableAutoConfiguration=\ org.springframework.cloud.netflix.hystrix.HystrixAutoConfiguration,\ org.springframework.cloud.netflix.hystrix.HystrixCircuitBreakerAutoConfiguration,\ org.springframework.cloud.netflix.hystrix.ReactiveHystrixCircuitBreakerAutoConfiguration,\ org.springframework.cloud.netflix.hystrix.security.HystrixSecurityAutoConfiguration org.springframework.cloud.client.circuitbreaker.EnableCircuitBreaker=\ org.springframework.cloud.netflix.hystrix.HystrixCircuitBreakerConfiguration
In @ SpringBootApplication, there is a reference to @ EnableAutoConfiguration, so the HystrixAutoConfiguration classes will be loaded upon startup
@SpringBootConfiguration
@EnableAutoConfiguration
@ComponentScan(excludeFilters = { @Filter(type = FilterType.CUSTOM, classes = TypeExcludeFilter.class),
@Filter(type = FilterType.CUSTOM, classes = AutoConfigurationExcludeFilter.class) })
public @interface SpringBootApplication {}
Analysis of HystrixCircuitBreakerConfiguration
@Configuration(proxyBeanMethods = false)
public class HystrixCircuitBreakerConfiguration {
//The core is this section
@Bean
public HystrixCommandAspect hystrixCommandAspect() {
return new HystrixCommandAspect();
}
....
}
In the HystrixCommandAspect, the two annotations, HystrixCommand and HystrixCollapser, are mainly processed
@Aspect
public class HystrixCommandAspect {
static {
//Instantiate two factories of two annotations through static methods
META_HOLDER_FACTORY_MAP = ImmutableMap.<HystrixPointcutType, MetaHolderFactory>builder()
.put(HystrixPointcutType.COMMAND, new CommandMetaHolderFactory())
.put(HystrixPointcutType.COLLAPSER, new CollapserMetaHolderFactory())
.build();
}
//Define pointcut annotation HystrixCommand
@Pointcut("@annotation(com.netflix.hystrix.contrib.javanica.annotation.HystrixCommand)")
public void hystrixCommandAnnotationPointcut() {
}
//Define pointcut annotation HystrixCollapser
@Pointcut("@annotation(com.netflix.hystrix.contrib.javanica.annotation.HystrixCollapser)")
public void hystrixCollapserAnnotationPointcut() {
}
@Around("hystrixCommandAnnotationPointcut() || hystrixCollapserAnnotationPointcut()")
public Object methodsAnnotatedWithHystrixCommand(final ProceedingJoinPoint joinPoint) throws Throwable {
//Get target method
Method method = getMethodFromTarget(joinPoint);
Validate.notNull(method, "failed to get method from joinPoint: %s", joinPoint);
//Only these two annotation annotation methods are processed
if (method.isAnnotationPresent(HystrixCommand.class) && method.isAnnotationPresent(HystrixCollapser.class)) {
throw new IllegalStateException("method cannot be annotated with HystrixCommand and HystrixCollapser " +
"annotations at the same time");
}
//Get the factory of different implementations of MetaHolderFactory
MetaHolderFactory metaHolderFactory = META_HOLDER_FACTORY_MAP.get(HystrixPointcutType.of(method));
//Get the metadata, method signature, parameters, etc. of the target method
MetaHolder metaHolder = metaHolderFactory.create(joinPoint);
/**
* Create a processor CommandCollapser or GenericCommand (synchronous) or genericobservable command (asynchronous)
* GenericCommand There are many super in the, and finally through the hystrix command builder factory getInstance(). Create (metaholder) builds a HystrixCommandBuilder as the parameter of genericcommand
* new GenericCommand From super to AbstractHystrixCommand,
* AbstractHystrixCommand From super to HystrixCommand,
* HystrixCommand Finally, the AbstractCommand is passed all the way
* Analyze it in AbstractCommand later
*/
HystrixInvokable invokable = HystrixCommandFactory.getInstance().create(metaHolder);
//Infer the execution type from the return value
ExecutionType executionType = metaHolder.isCollapserAnnotationPresent() ?
metaHolder.getCollapserExecutionType() : metaHolder.getExecutionType();
//Return results
Object result;
try {
//Do not use observable mode
if (!metaHolder.isObservable()) {
//Execute execute
result = CommandExecutor.execute(invokable, executionType, metaHolder);
} else {
result = executeObservable(invokable, executionType, metaHolder);
}
} catch (HystrixBadRequestException e) {
throw e.getCause();
} catch (HystrixRuntimeException e) {
throw hystrixRuntimeExceptionToThrowable(metaHolder, e);
}
return result;
}
//Creation of MetaHolder when HystrixCommand
private static class CommandMetaHolderFactory extends MetaHolderFactory {
@Override
public MetaHolder create(Object proxy, Method method, Object obj, Object[] args, final ProceedingJoinPoint joinPoint) {
//Get annotation HystrixCommand
HystrixCommand hystrixCommand = method.getAnnotation(HystrixCommand.class);
//Infer the task type according to the returned results, and you can know which method to execute
ExecutionType executionType = ExecutionType.getExecutionType(method.getReturnType());
MetaHolder.Builder builder = metaHolderBuilder(proxy, method, obj, args, joinPoint);
if (isCompileWeaving()) {
builder.ajcMethod(getAjcMethodFromTarget(joinPoint));
}
//There are not many parameters here. The most important one is the hystrix command. What did you add to the annotation
return builder.defaultCommandKey(method.getName())
.hystrixCommand(hystrixCommand)
.observableExecutionMode(hystrixCommand.observableExecutionMode()) //Execution mode
.executionType(executionType) //Execution mode
.observable(ExecutionType.OBSERVABLE == executionType)
.build();
}
}
}
//In the enumerated ExecutionType class
public static ExecutionType getExecutionType(Class<?> type) {
if (Future.class.isAssignableFrom(type)) {
return ExecutionType.ASYNCHRONOUS;
} else if (Observable.class.isAssignableFrom(type)) {
return ExecutionType.OBSERVABLE;
} else {
return ExecutionType.SYNCHRONOUS;
}
}
Let's focus on the analysis of synchronization. Through the code, we can see that HystrixInvokable is GenericCommand. In synchronization, let's look at commandexecutor execute(invokable, executionType, metaHolder)
public class CommandExecutor {
public static Object execute(HystrixInvokable invokable, ExecutionType executionType, MetaHolder metaHolder) throws RuntimeException {
Validate.notNull(invokable);
Validate.notNull(metaHolder);
switch (executionType) {
case SYNCHRONOUS: {
//Focus on synchronization. First convert the GenericCommand to HystrixExecutable, and then execute
return castToExecutable(invokable, executionType).execute();
}
case ASYNCHRONOUS: {
HystrixExecutable executable = castToExecutable(invokable, executionType);
if (metaHolder.hasFallbackMethodCommand()
&& ExecutionType.ASYNCHRONOUS == metaHolder.getFallbackExecutionType()) {
return new FutureDecorator(executable.queue());
}
return executable.queue();
}
case OBSERVABLE: {
HystrixObservable observable = castToObservable(invokable);
return ObservableExecutionMode.EAGER == metaHolder.getObservableExecutionMode() ? observable.observe() : observable.toObservable();
}
default:
throw new RuntimeException("unsupported execution type: " + executionType);
}
}
}
Let's look at class relations first
Turn up the GenericCommand layer by layer, and finally locate that the HystrixCommand has an execute()
HystrixCommand
public abstract class HystrixCommand<R> extends AbstractCommand<R> implements HystrixExecutable<R>, HystrixInvokableInfo<R>, HystrixObservable<R> {
//Synchronous execution
public R execute() {
try {
//Via queue() Get() to execute synchronously
return queue().get();
} catch (Exception e) {
throw Exceptions.sneakyThrow(decomposeException(e));
}
}
//Asynchronous execution. When to get() is determined by the caller. When to get() will be blocked
public Future<R> queue() {
//The core processing is finally located in toObservable() in AbstractCommand
final Future<R> delegate = toObservable().toBlocking().toFuture();
final Future<R> f = new Future<R>() {
.....
@Override
public R get() throws InterruptedException, ExecutionException {
return delegate.get();
}
@Override
public R get(long timeout, TimeUnit unit) throws InterruptedException, ExecutionException, TimeoutException {
return delegate.get(timeout, unit);
}
};
//After special processing, it has been executed, and get() will not be blocked
if (f.isDone()) {
try {
f.get();
return f;
} catch (Exception e) {
...
}
}
return f;
}
}
Let's look at the official flow chart: (source address: https://github.com/Netflix/Hystrix/wiki/How-it-Works )Through the code, we can also see that it is here
AbstractCommand
abstract class AbstractCommand<R> implements HystrixInvokableInfo<R>, HystrixObservable<R> {
// A new GenericCommand finally comes here, and the parameters are wrapped here, just from the GenericCommand
protected AbstractCommand(HystrixCommandGroupKey group, HystrixCommandKey key, HystrixThreadPoolKey threadPoolKey, HystrixCircuitBreaker circuitBreaker, HystrixThreadPool threadPool,
HystrixCommandProperties.Setter commandPropertiesDefaults, HystrixThreadPoolProperties.Setter threadPoolPropertiesDefaults,
HystrixCommandMetrics metrics, TryableSemaphore fallbackSemaphore, TryableSemaphore executionSemaphore,
HystrixPropertiesStrategy propertiesStrategy, HystrixCommandExecutionHook executionHook) {
//Set group
this.commandGroup = initGroupKey(group);
//Normally, it is the method name. If it is not set, it is the class
this.commandKey = initCommandKey(key, getClass());
//Optimization is made here, and the assembled CommandProperties are cached in the ConcurrentHashMap (if it is a dynamic generation method overflow?)
this.properties = initCommandProperties(this.commandKey, propertiesStrategy, commandPropertiesDefaults);
//Initialize the key of the thread pool. If it is null, use groupkey name()
this.threadPoolKey = initThreadPoolKey(threadPoolKey, this.commandGroup, this.properties.executionIsolationThreadPoolKeyOverride().get());
//Initialize metrics to commandKey cache
this.metrics = initMetrics(metrics, this.commandGroup, this.threadPoolKey, this.commandKey, this.properties);
//Initializing the circuit breaker is also cached with commandKey
this.circuitBreaker = initCircuitBreaker(this.properties.circuitBreakerEnabled().get(), circuitBreaker, this.commandGroup, this.commandKey, this.properties, this.metrics);
//Initialize the thread pool, and create a cache if it is available. It is also the commandKey dimension. Finally, it is through concurrency strategy Getthreadpool (threadpoolkey, properties) creation
this.threadPool = initThreadPool(threadPool, this.threadPoolKey, threadPoolPropertiesDefaults);
//Strategies from plugins
this.eventNotifier = HystrixPlugins.getInstance().getEventNotifier();
this.concurrencyStrategy = HystrixPlugins.getInstance().getConcurrencyStrategy();
HystrixMetricsPublisherFactory.createOrRetrievePublisherForCommand(this.commandKey, this.commandGroup, this.metrics, this.circuitBreaker, this.properties);
this.executionHook = initExecutionHook(executionHook);
this.requestCache = HystrixRequestCache.getInstance(this.commandKey, this.concurrencyStrategy);
this.currentRequestLog = initRequestLog(this.properties.requestLogEnabled().get(), this.concurrencyStrategy);
/* fallback semaphore override if applicable */
this.fallbackSemaphoreOverride = fallbackSemaphore;
/* execution semaphore override if applicable */
this.executionSemaphoreOverride = executionSemaphore;
}
//Core processing logic
public Observable<R> toObservable() {
//Get the current object
final AbstractCommand<R> _cmd = this;
/**
* terminateCommandCleanup Callback function for interrupt command cleanup
* unsubscribeCommandCleanup Cleanup callback function for unsubscribe command
* applyHystrixSemantics Create Observable with execution logic for post subscription callback
* wrapWithAllOnNextHooks onNext Hook for execution
* fireOnCompletedHook Hook triggered after completion
*/
final Func0<Observable<R>> applyHystrixSemantics = new Func0<Observable<R>>() {
@Override
public Observable<R> call() {
//No subscription not executed
if (commandState.get().equals(CommandState.UNSUBSCRIBED)) {
return Observable.never();
}
//implement
return applyHystrixSemantics(_cmd);
}
};
//Hook after onNext execution
final Func1<R, R> wrapWithAllOnNextHooks = new Func1<R, R>() {
@Override
public R call(R r) {
R afterFirstApplication = r;
try {
afterFirstApplication = executionHook.onComplete(_cmd, r);
} catch (Throwable hookEx) {
logger.warn("Error calling HystrixCommandExecutionHook.onComplete", hookEx);
}
try {
return executionHook.onEmit(_cmd, afterFirstApplication);
} catch (Throwable hookEx) {
logger.warn("Error calling HystrixCommandExecutionHook.onEmit", hookEx);
return afterFirstApplication;
}
}
};
//Create Observable through defer
return Observable.defer(new Func0<Observable<R>>() {
//To create an observer, the execution of the observer must be triggered by subscribe()
@Override
public Observable<R> call() {
//The command status is judged through cas, and the initial value is NOT_STARTED, if it's not not NOT_STARTED indicates that another thread has been executed, so an exception is thrown
if (!commandState.compareAndSet(CommandState.NOT_STARTED, CommandState.OBSERVABLE_CHAIN_CREATED)) {
IllegalStateException ex = new IllegalStateException("This instance can only be executed once. Please instantiate a new instance.");
throw new HystrixRuntimeException(FailureType.BAD_REQUEST_EXCEPTION, _cmd.getClass(), getLogMessagePrefix() + " command executed multiple times - this is not permitted.", ex, null);
}
//Command start time
commandStartTimestamp = System.currentTimeMillis();
if (properties.requestLogEnabled().get()) {
// Log whatever happens
if (currentRequestLog != null) {
currentRequestLog.addExecutedCommand(_cmd);
}
}
//Whether to allow request caching,
final boolean requestCacheEnabled = isRequestCachingEnabled();
//The CacheKey needs to be set @ CacheKey. In the HystrixCacheKeyGenerator, the method and the value of the specified key will be spliced
final String cacheKey = getCacheKey();
//If the request cache is allowed, it is obtained from the cache first, and the final cache uses ConcurrentHashMap
if (requestCacheEnabled) {
HystrixCommandResponseFromCache<R> fromCache = (HystrixCommandResponseFromCache<R>) requestCache.get(cacheKey);
if (fromCache != null) {
isResponseFromCache = true;
return handleRequestCacheHitAndEmitValues(fromCache, _cmd);
}
}
//Create Observable through defer
Observable<R> hystrixObservable = Observable.defer(applyHystrixSemantics).map(wrapWithAllOnNextHooks);
Observable<R> afterCache;
// When the cache is turned on, put it back into the cache
if (requestCacheEnabled && cacheKey != null) {
// Wrap the hystrixObservable and put it into the cache
HystrixCachedObservable<R> toCache = HystrixCachedObservable.from(hystrixObservable, _cmd);
HystrixCommandResponseFromCache<R> fromCache = (HystrixCommandResponseFromCache<R>) requestCache.putIfAbsent(cacheKey, toCache);
} else {
afterCache = hystrixObservable;
}
return afterCache
.doOnTerminate(terminateCommandCleanup) //Interrupt command cleanup
.doOnUnsubscribe(unsubscribeCommandCleanup) // Clean up without subscription
.doOnCompleted(fireOnCompletedHook);//Complete triggered hook
}
});
}
private Observable<R> applyHystrixSemantics(final AbstractCommand<R> _cmd) {
//Start execution
executionHook.onStart(_cmd);
//The circuit breaker is allowed to execute. There are many logics in it. See the following explanation for details
if (circuitBreaker.allowRequest()) {
//Semaphore processing (if you configure the semaphore processing mechanism, if you use thread pool, use TryableSemaphoreNoOp, and the requested semaphore is true)
final TryableSemaphore executionSemaphore = getExecutionSemaphore();
final AtomicBoolean semaphoreHasBeenReleased = new AtomicBoolean(false);
//Release semaphore after execution
final Action0 singleSemaphoreRelease = new Action0() {
@Override
public void call() {
if (semaphoreHasBeenReleased.compareAndSet(false, true)) {
executionSemaphore.release();
}
}
};
final Action1<Throwable> markExceptionThrown = new Action1<Throwable>() {
@Override
public void call(Throwable t) {
eventNotifier.markEvent(HystrixEventType.EXCEPTION_THROWN, commandKey);
}
};
//Application semaphore
if (executionSemaphore.tryAcquire()) {
try {
//Set call time
executionResult = executionResult.setInvocationStartTime(System.currentTimeMillis());
return executeCommandAndObserve(_cmd) //Execute the command, this is the core
.doOnError(markExceptionThrown) //exception handling
.doOnTerminate(singleSemaphoreRelease)//When interrupted, the semaphore is released
.doOnUnsubscribe(singleSemaphoreRelease);//Do not subscribe or release semaphores
} catch (RuntimeException e) {
return Observable.error(e);
}
} else {
//Execute Fallback directly if the semaphore is not applied
return handleSemaphoreRejectionViaFallback();
}
} else {
//If the circuit breaker is not allowed to execute, the Fallback method is directly executed
return handleShortCircuitViaFallback();
}
}
private Observable<R> executeCommandAndObserve(final AbstractCommand<R> _cmd) {
//Hystrixrequestcontext is the context of a hystrix thread. It can pass some content to prevent data loss in the asynchronous process
final HystrixRequestContext currentRequestContext = HystrixRequestContext.getContextForCurrentThread();
/**
* It also defines a bunch of actions and Func S
* markEmits For onNext callback
* markOnCompleted Used to complete post-processing and sample a SUCCESS event
* handleFallback,There are returned results for processing fallbacks,
* setRequestContext Set request context
* markEmits,markOnCompleted,handleFallback Will produce an event
*/
Observable<R> execution;
//Execute command
if (properties.executionTimeoutEnabled().get()) {
//When the timeout is configured, a hystrixobservatabletimeoutoperator will be added, and the timeout event will be sent here
execution = executeCommandWithSpecifiedIsolation(_cmd).lift(new HystrixObservableTimeoutOperator<R>(_cmd));
} else {
execution = executeCommandWithSpecifiedIsolation(_cmd);
}
return execution.doOnNext(markEmits)
.doOnCompleted(markOnCompleted)
.onErrorResumeNext(handleFallback)
.doOnEach(setRequestContext);
}
private Observable<R> executeCommandWithSpecifiedIsolation(final AbstractCommand<R> _cmd) {
//The real execution logic distinguishes between thread pool and semaphore
if (properties.executionIsolationStrategy().get() == ExecutionIsolationStrategy.THREAD) {
//
return Observable.defer(new Func0<Observable<R>>() {
@Override
public Observable<R> call() {
executionResult = executionResult.setExecutionOccurred();
if (!commandState.compareAndSet(CommandState.OBSERVABLE_CHAIN_CREATED, CommandState.USER_CODE_EXECUTED)) {
return Observable.error(new IllegalStateException("execution attempted while in state : " + commandState.get().name()));
}
metrics.markCommandStart(commandKey, threadPoolKey, ExecutionIsolationStrategy.THREAD);
if (isCommandTimedOut.get() == TimedOutStatus.TIMED_OUT) {
return Observable.error(new RuntimeException("timed out before executing run()"));
}
//Set thread status, count, and execute
if (threadState.compareAndSet(ThreadState.NOT_USING_THREAD, ThreadState.STARTED)) {
//count
HystrixCounters.incrementGlobalConcurrentThreads();
threadPool.markThreadExecution();
endCurrentThreadExecutingCommand = Hystrix.startCurrentThreadExecutingCommand(getCommandKey());
executionResult = executionResult.setExecutedInThread();
try {
executionHook.onThreadStart(_cmd);
executionHook.onRunStart(_cmd);
executionHook.onExecutionStart(_cmd);
return getUserExecutionObservable(_cmd);
} catch (Throwable ex) {
return Observable.error(ex);
}
} else {
return Observable.error(new RuntimeException("unsubscribed before executing run()"));
}
}
}).doOnTerminate(new Action0() {
@Override
public void call() {
//exception handling
if (threadState.compareAndSet(ThreadState.STARTED, ThreadState.TERMINAL)) {
handleThreadEnd(_cmd);
}
}
}).doOnUnsubscribe(new Action0() {
@Override
public void call() {
//Unsubscribed processing
if (threadState.compareAndSet(ThreadState.STARTED, ThreadState.UNSUBSCRIBED)) {
handleThreadEnd(_cmd);
}
}
}).subscribeOn(threadPool.getScheduler(new Func0<Boolean>() {
@Override
public Boolean call() {
return properties.executionIsolationThreadInterruptOnTimeout().get() && _cmd.isCommandTimedOut.get() == TimedOutStatus.TIMED_OUT;
}
}));
} else {
//Execution of semaphores
return Observable.defer(new Func0<Observable<R>>() {
@Override
public Observable<R> call() {
executionResult = executionResult.setExecutionOccurred();
if (!commandState.compareAndSet(CommandState.OBSERVABLE_CHAIN_CREATED, CommandState.USER_CODE_EXECUTED)) {
return Observable.error(new IllegalStateException("execution attempted while in state : " + commandState.get().name()));
}
metrics.markCommandStart(commandKey, threadPoolKey, ExecutionIsolationStrategy.SEMAPHORE);
// semaphore isolated
// store the command that is being run
endCurrentThreadExecutingCommand = Hystrix.startCurrentThreadExecutingCommand(getCommandKey());
try {
executionHook.onRunStart(_cmd);
executionHook.onExecutionStart(_cmd);
return getUserExecutionObservable(_cmd); //the getUserExecutionObservable method already wraps sync exceptions, so this shouldn't throw
} catch (Throwable ex) {
//If the above hooks throw, then use that as the result of the run method
return Observable.error(ex);
}
}
});
}
}
}
Circuit breaker
public interface HystrixCircuitBreaker {
//Initialization implementation
public static class Factory {
public static HystrixCircuitBreaker getInstance(HystrixCommandKey key, HystrixCommandGroupKey group, HystrixCommandProperties properties, HystrixCommandMetrics metrics) {
// If there is in the cache, it will be returned directly
HystrixCircuitBreaker previouslyCached = circuitBreakersByCommand.get(key.name());
if (previouslyCached != null) {
return previouslyCached;
}
// The first time I came in, it was initialized and cached in the map without
HystrixCircuitBreaker cbForCommand = circuitBreakersByCommand.putIfAbsent(key.name(), new HystrixCircuitBreakerImpl(key, group, properties, metrics));
if (cbForCommand == null) {
// this means the putIfAbsent step just created a new one so let's retrieve and return it
return circuitBreakersByCommand.get(key.name());
} else {
return cbForCommand;
}
}
}
//Circuit breaker implementation
static class HystrixCircuitBreakerImpl implements HystrixCircuitBreaker {
//The configuration of hystrix is in hystrix commandproperties
@Override
public boolean allowRequest() {
if (properties.circuitBreakerForceOpen().get()) {
//Forcibly open the circuit breaker and return directly. At this time, it will blow
return false;
}
//Verify again with the circuit breaker closed
if (properties.circuitBreakerForceClosed().get()) {
isOpen();
//No matter what the result this time, at least what I come in is on. After it is set to off, I have to continue to execute
return true;
}
return !isOpen() || allowSingleTest();
}
@Override
public boolean isOpen() {
//If it is open, intercept and return true directly
if (circuitOpen.get()) {
return true;
}
/**
* HealthCounts Stored in is the number of requests during a sliding window.
* totalCount The total number of requests includes: (failure + success + timeout + threadPoolRejected + semaphore rejected)
* errorCount Exception data removal success is failure
* errorPercentage Exception percentage = errorCount/totalCount *100;
*/
HealthCounts health = metrics.getHealthCounts();
//If the total number of requests is less than the configured value, it will not be intercepted. The configured value is hystrix Corresponding comandkey circuitBreaker. The default value of requestvolumthreshold is 20
if (health.getTotalRequests() < properties.circuitBreakerRequestVolumeThreshold().get()) {
return false;
}
//If the exception rate is less than the configured value, it will not be intercepted. The configured value is hystrix Corresponding comandkey circuitBreaker. The default value of errorthresholdpercentage is 50, that is, if there are 50% exceptions, it will be blown
if (health.getErrorPercentage() < properties.circuitBreakerErrorThresholdPercentage().get()) {
return false;
} else {
// If the abnormal ratio is high, set the circuit breaker to open
if (circuitOpen.compareAndSet(false, true)) {
// Set the opening time of the circuit breaker
circuitOpenedOrLastTestedTime.set(System.currentTimeMillis());
return true;
} else {
//This shows that this thread has not been set successfully. Other threads have opened the circuit breaker and returned directly
return true;
}
}
}
}
}
}
- During initialization, cache the HystrixCircuitBreaker through the ConcurrentHashMap to improve performance.
The three states of the circuit breaker are transformed as follows:
The circuit breaker is initially in the Closed state. If the call continues to make an error or timeout, the circuit breaker will enter the Open state and make a fusing request. All calls in the subsequent period of time will trigger a fallback
-
Open status: it is requested not to call the current service. The internal set clock is generally MTTR (mean failure processing time). When it is opened up to the set clock, it will enter the semi fusing state
-
Closed: when the circuit breaker is closed, it will not make a partial request to the service
-
Half Open: call the current service according to the rules. If the request is successful and meets the rules, it is considered that the current service is restored to normal and closed
Look at the official flow chart: https://github.com/Netflix/Hystrix/wiki/How-it-Works There are pictures and explanations
-
The dimension of circuit breaker is commandKey
-
The circuit breaker is open and directly fused
-
During the window period, if the requested quantity is greater than the set value, it will fuse (the default is more than 20 requests in 10 seconds)
-
If the abnormal rate is greater than the configured value, the fuse will be blown (the default is more than 50% failure in 10 seconds)
-
After a period of time, the circuit breaker is in the broken state and will allow a request to come in. If successful, the circuit breaker will close
PS: the configuration properties of the hystrixcommand are in class: HystrixCommandProperties
Last up and down logic diagram
At the same time, in the jar package of hystrix core, com netflix. hystrix. metric. Under consumer, there are many consumption streams of HystrixEvent, which implement different flow limiting methods according to the configuration, including sliding window and token bucket. These streams will be started when HystrixCommandMetrics is instantiated. If you want to understand this section, you can take a look at the hystrix configuration stream and hystrix command metrics. We will also analyze this in the later stage.
The process is straightened out, but the whole process is bumpy. There are layers inside. If you want to really understand its underlying operation, you need to have a good understanding of Rxjava and share it after follow-up research.