Start at http://otzh.ml

Preface

Hystrix is no longer maintained, but a successful open source project is always worth learning. When you first look at the source code of Hystrix, you will find a bunch of Action,Function logic, which is actually the characteristic of RxJava - Responsive Programming. Introduction Unfamiliar students can go to see Hystrix first. This article will briefly introduce Hystrix, and then according to demo combined with source code to understand the implementation process of Hystrix.

A Brief Introduction to Hystrix

1. What is Hystrix?

   Hystrix is a delay and fault-tolerant library designed to isolate access points to remote systems, services and third-party libraries, stop cascading failures and resilient recovery in complex distributed systems where errors are inevitable.

2. Core concepts

   1. Command command

      command is the entry of Hystrix. For users, we just need to create corresponding commands and wrap the interfaces that need to be protected. We don't need to pay attention to the logic later. After deep integration with Spring, we can also use annotations to make it more development-friendly.

   2. Circuit Breaker Circuit Breaker

      Circuit breaker is a concept derived from the electrical field. It has the functions of overload, short circuit and under-voltage protection, and the ability to protect lines and power supply. In Hystrix, when a request exceeds a certain proportion of response failure, hystrix intercepts the request, guarantees the stability of the service and prevents cascading snow between services. The possibility of collapse.

   3. Isolation isolation strategy

      Isolation strategy is the highlight of Hystrix design, using Bulkhead mode Hystrix has two isolation strategies: thread pool isolation and semaphore isolation.

      

3. Running process of Hystrix

   Official How it Works It has a detailed introduction to the process, and the diagram is clear. I believe that after reading the flow chart, we can have a certain understanding of the operation process.

   

A Command Execution

HystrixCommand demo = new HystrixCommand<String>(HystrixCommandGroupKey.Factory.asKey("demo-group")) {
            @Override
            protected String run() {
                return "Hello World~";
            }
        };
demo.execute();

Execution process

Flow chart

This is the official link for a full call. In the demo above, we call the execute method directly, so the call path is execute () - > queue () - > toObservable() - > toBlocking () - > toFuture () - > get (). The core logic is actually in toObservable().

HystrixCommand.java

execute

The execute method returns the result for a synchronous call and handles the exception. queue is called internally.

// Synchronized call execution
public R execute() {
  try {
    // queue() returns an object of type Future, so here is blocking get
    return queue().get();
  } catch (Exception e) {
    throw decomposeException(e);
  }
}

queue

queueThe first line of code completes the core subscription logic.

1. toObservable() Generated Hystrix Of Observable object
2. take Observable Convert to BlockingObservable Blocking control data transmission
3. toFuture Implementing pairs BlockingObservable Subscription

public Future<R> queue() {
  // Focus on this line of code
  // Completed the creation and subscription of Observable
  // toBlocking() is to convert Observable to Blocking Observable, which can block the sending of data.
  final Future<R> delegate = toObservable().toBlocking().toFuture();

  final Future<R> f = new Future<R>() {
    // Because the Future returned by toObservable().toBlocking().toFuture() if interrupted,
    // There is no interruption to the current thread, so the Future returned here is repackaged to handle the exception logic
    ...
  }

  // Judge if it's over and throw an exception directly
  if (f.isDone()) {
    try {
      f.get();
      return f;
    } catch (Exception e) {
            // Eliminate this judgment
    }
  }

  return f;
}

BlockingObservable.java

// Packed Observable
private final Observable<? extends T> o;

// toBlocking() calls this static method to simply wrap the source Observable as a Blocking Observable
public static <T> BlockingObservable<T> from(final Observable<? extends T> o) {
  return new BlockingObservable<T>(o);
}

public Future<T> toFuture() {
  return BlockingOperatorToFuture.toFuture((Observable<T>)o);
}

BlockingOperatorToFuture.java

ReactiveX's Interpretation of to Future

The toFuture operator applies to the BlockingObservable subclass, so in order to use it, you must first convert your source Observable into a BlockingObservable by means of either the BlockingObservable.from method or the Observable.toBlocking operator.

toFuture only works on BlockingObservable, so there's also the operation above that you want to convert to BlockingObservable.

// This operation converts the source Observable to Future that returns a single data item
public static <T> Future<T> toFuture(Observable<? extends T> that) {
      // CountDownLatch judges whether or not it is complete
    final CountDownLatch finished = new CountDownLatch(1);
      // Store execution results
    final AtomicReference<T> value = new AtomicReference<T>();
      // Storing error results
    final AtomicReference<Throwable> error = new AtomicReference<Throwable>();

      // The single() method can limit Observable to send only one piece of data
      // Illegal ArgumentException is thrown if there are more than one data
      // If there is no data to send, NoSuchElementException will be thrown
    @SuppressWarnings("unchecked")
    final Subscription s = ((Observable<T>)that).single().subscribe(new Subscriber<T>() {
                // Observable returned by single() can be processed as a standard
        @Override
        public void onCompleted() {
            finished.countDown();
        }

        @Override
        public void onError(Throwable e) {
            error.compareAndSet(null, e);
            finished.countDown();
        }

        @Override
        public void onNext(T v) {
            // "single" guarantees there is only one "onNext"
            value.set(v);
        }
    });
        
      // Finally, the data returned by Subscription is encapsulated into Future and the corresponding logic is implemented.
    return new Future<T>() {
            // You can view the source code
    };

}

AbstractCommand.java

AbstractCommand is the place where toObservable is implemented. It belongs to the core logic of Hystrix. It has long code and can be eaten with the flow chart of method invocation. ToObservable mainly completes the logic of caching and creating Observable and requestLog. When Observable is first created, the applyHystrix Semantics method is the semantic implementation of Hystrix. Jump and watch.

tips: There are many actions and functions in the following sections. They are very similar. They all have call methods, but the difference is that Function has return values, while Action does not. The number followed by the method represents several entries. Func0/Func3 means that there are no entries and three entries.

toObservable

The toObservable code is long and layered, so the next block is written one by one. Its logic and the article begins with Hystrix flow chart It's exactly the same.

public Observable<R> toObservable() {
    final AbstractCommand<R> _cmd = this;
      // Many Action s and Function s are omitted here, most of which are functions for cleaning up, so let's talk about them when we use them.
  
      // defer, as mentioned in the previous introduction to rxjava, is a creative operator that generates a new Observable each time a subscription is made, and what we really need in the callback method is the Observable.
    return Observable.defer(new Func0<Observable<R>>() {
        @Override
        public Observable<R> call() {
              
                        // Check the status of the command to ensure that it is executed only once
            if (!commandState.compareAndSet(CommandState.NOT_STARTED, CommandState.OBSERVABLE_CHAIN_CREATED)) {
                IllegalStateException ex = new IllegalStateException("This instance can only be executed once. Please instantiate a new instance.");
                //TODO make a new error type for this
                throw new HystrixRuntimeException(FailureType.BAD_REQUEST_EXCEPTION, _cmd.getClass(), getLogMessagePrefix() + " command executed multiple times - this is not permitted.", ex, null);
            }

            commandStartTimestamp = System.currentTimeMillis();
                        // properties are all attributes of the current command
              // The command currently executed is saved when the request log is allowed to be recorded
            if (properties.requestLogEnabled().get()) {
                // log this command execution regardless of what happened
                if (currentRequestLog != null) {
                    currentRequestLog.addExecutedCommand(_cmd);
                }
            }
                        
              // Whether request caching is enabled
            final boolean requestCacheEnabled = isRequestCachingEnabled();
              // Get the cache key
            final String cacheKey = getCacheKey();

            // After opening the cache, try to extract it from the cache
            if (requestCacheEnabled) {
                HystrixCommandResponseFromCache<R> fromCache = (HystrixCommandResponseFromCache<R>) requestCache.get(cacheKey);
                if (fromCache != null) {
                    isResponseFromCache = true;
                    return handleRequestCacheHitAndEmitValues(fromCache, _cmd);
                }
            }
              // When the request cache is not opened, normal logic is executed
            Observable<R> hystrixObservable =
                          // Here we create the Observable we need through defer
                    Observable.defer(applyHystrixSemantics)
                                          // The default execution hook is empty, so we skipped it here.
                            .map(wrapWithAllOnNextHooks);
          
            // After you get the final encapsulated Observable, put it in the cache
            if (requestCacheEnabled && cacheKey != null) {
                // wrap it for caching
                HystrixCachedObservable<R> toCache = HystrixCachedObservable.from(hystrixObservable, _cmd);
                HystrixCommandResponseFromCache<R> fromCache = (HystrixCommandResponseFromCache<R>) requestCache.putIfAbsent(cacheKey, toCache);
                if (fromCache != null) {
                    // another thread beat us so we'll use the cached value instead
                    toCache.unsubscribe();
                    isResponseFromCache = true;
                    return handleRequestCacheHitAndEmitValues(fromCache, _cmd);
                } else {
                    // we just created an ObservableCommand so we cast and return it
                    afterCache = toCache.toObservable();
                }
            } else {
                afterCache = hystrixObservable;
            }

            return afterCache
                          // Operation at termination
                    .doOnTerminate(terminateCommandCleanup)     // perform cleanup once (either on normal terminal state (this line), or unsubscribe (next line))
                          // Operation on unsubscribe
                    .doOnUnsubscribe(unsubscribeCommandCleanup) // perform cleanup once
                          // Operation at completion
                    .doOnCompleted(fireOnCompletedHook);
        }
    }
                     

handleRequestCacheHitAndEmitValues

Cache Hit Processing

private Observable<R> handleRequestCacheHitAndEmitValues(final HystrixCommandResponseFromCache<R> fromCache, final AbstractCommand<R> _cmd) {
        try {
              // There are a lot of hooks in Hystrix. If you want to do secondary development, you can use these hooks to achieve perfect monitoring.
            executionHook.onCacheHit(this);
        } catch (Throwable hookEx) {
            logger.warn("Error calling HystrixCommandExecutionHook.onCacheHit", hookEx);
        }   
  // Assign cached results to the current command
    return fromCache.toObservableWithStateCopiedInto(this)
                    // doOnTerminate, or doOnUnsubscribe, or doOnError, as you will see later, refers to the operation after responding to onTerminate/onUnsubscribe/onError, i.e. registering an action-graceful processing logic on the life cycle of Observable
            .doOnTerminate(new Action0() {
                @Override
                public void call() {
                      // Different processing for different final states of commands
                    if (commandState.compareAndSet(CommandState.OBSERVABLE_CHAIN_CREATED, CommandState.TERMINAL)) {
                        cleanUpAfterResponseFromCache(false); //user code never ran
                    } else if (commandState.compareAndSet(CommandState.USER_CODE_EXECUTED, CommandState.TERMINAL)) {
                        cleanUpAfterResponseFromCache(true); //user code did run
                    }
                }
            })
            .doOnUnsubscribe(new Action0() {
                @Override
                public void call() {
                      // Different processing for different final states of commands
                    if (commandState.compareAndSet(CommandState.OBSERVABLE_CHAIN_CREATED, CommandState.UNSUBSCRIBED)) {
                        cleanUpAfterResponseFromCache(false); //user code never ran
                    } else if (commandState.compareAndSet(CommandState.USER_CODE_EXECUTED, CommandState.UNSUBSCRIBED)) {
                        cleanUpAfterResponseFromCache(true); //user code did run
                    }
                }
            });
}       

applyHystrixSemantics

Because the main purpose of this article is to talk about the execution process, the failure backoff and circuit breaker related will be reserved for future articles.

final Func0<Observable<R>> applyHystrixSemantics = new Func0<Observable<R>>() {
    @Override
    public Observable<R> call() {
          // Return Observable that does not send data without subscribing
        if (commandState.get().equals(CommandState.UNSUBSCRIBED)) {
              // No data or notification is sent
            return Observable.never();
        }
        return applyHystrixSemantics(_cmd);
    }
};

private Observable<R> applyHystrixSemantics(final AbstractCommand<R> _cmd) {
    // A hook that marks the beginning of execution
  // If an exception is thrown in the hook, it will fail quickly without fallback processing.
  executionHook.onStart(_cmd);

  /* determine if we're allowed to execute */
  // Circuit Breaker Core Logic: Determining whether or not to allow execution (TODO)
  if (circuitBreaker.allowRequest()) {
    // Hystrix's own signal wheel is not used under juc. The official explanation is that Juc's emphore implementation is too complex and has no ability to dynamically adjust the size of the signal. In short, it does not meet the demand.
    // Getting different Tryable Semphores according to different isolation strategies (thread pool isolation/semaphore isolation)
    final TryableSemaphore executionSemaphore = getExecutionSemaphore();
    // Semaphore Release Logo
    final AtomicBoolean semaphoreHasBeenReleased = new AtomicBoolean(false);
    
    // Action to release semaphores
    final Action0 singleSemaphoreRelease = new Action0() {
      @Override
      public void call() {
        if (semaphoreHasBeenReleased.compareAndSet(false, true)) {
          executionSemaphore.release();
        }
      }
    };

    // exception handling
    final Action1<Throwable> markExceptionThrown = new Action1<Throwable>() {
      @Override
      public void call(Throwable t) {
        // Hystrix Event Notifier is a plug-in of hystrix. Different events send different notifications. The default is empty implementation.
        eventNotifier.markEvent(HystrixEventType.EXCEPTION_THROWN, commandKey);
      }
    };
        
    // Thread pool isolated Tryable Semphore is always true
    if (executionSemaphore.tryAcquire()) {
      try {
        /* used to track userThreadExecutionTime */
        // Execution Result is a result information encapsulation of a command execution
        // The start time is set here to record the life cycle of the command, and other attributes are set in during execution.
        executionResult = executionResult.setInvocationStartTime(System.currentTimeMillis());
        return executeCommandAndObserve(_cmd)
          // Dealing with Error Reporting
          .doOnError(markExceptionThrown)
          // Release upon termination
          .doOnTerminate(singleSemaphoreRelease)
          // Release when unsubscribed
          .doOnUnsubscribe(singleSemaphoreRelease);
      } catch (RuntimeException e) {
        return Observable.error(e);
      }
    } else {
      // When tryAcquire fails, it fallback, TODO
      return handleSemaphoreRejectionViaFallback();
    }
  } else {
    // Circuit Breaker Short Circuit (Deny Request) fallback Processing TODO
    return handleShortCircuitViaFallback();
  }
}

executeCommandAndObserve

/**
 * Where to execute the run method
 */
private Observable<R> executeCommandAndObserve(final AbstractCommand<R> _cmd) {
      // Get the current context
    final HystrixRequestContext currentRequestContext = HystrixRequestContext.getContextForCurrentThread();

      // Action response when sending data
    final Action1<R> markEmits = new Action1<R>() {
        @Override
        public void call(R r) {
              // If onNext needs to be reported, do the following
            if (shouldOutputOnNextEvents()) {
                  // result tag
                executionResult = executionResult.addEvent(HystrixEventType.EMIT);
                  // notice
                eventNotifier.markEvent(HystrixEventType.EMIT, commandKey);
            }
              // commandIsScalar is a place I don't understand, and I haven't found a good explanation online.
              // This method is an abstract method, with HystrixCommand implementation returning true.HystrixObservableCommand returning false.
            if (commandIsScalar()) {
                  // time consuming
                long latency = System.currentTimeMillis() - executionResult.getStartTimestamp();
                  // notice
                eventNotifier.markCommandExecution(getCommandKey(), properties.executionIsolationStrategy().get(), (int) latency, executionResult.getOrderedList());
                eventNotifier.markEvent(HystrixEventType.SUCCESS, commandKey);
                executionResult = executionResult.addEvent((int) latency, HystrixEventType.SUCCESS);
                  // Circuit Breaker Marking Successful (Feedback from Half-Open Circuit Breaker to Decide whether to Close Circuit Breaker)
                circuitBreaker.markSuccess();
            }
        }
    };

    final Action0 markOnCompleted = new Action0() {
        @Override
        public void call() {
            if (!commandIsScalar()) {
                            // Similar to markEmits
            }
        }
    };

      // Logic of Failure Retreat
    final Func1<Throwable, Observable<R>> handleFallback = new Func1<Throwable, Observable<R>>() {
        @Override
        public Observable<R> call(Throwable t) {
          // It's not the point that's skipped.
        }
    };

      // Processing of request context
    final Action1<Notification<? super R>> setRequestContext = new Action1<Notification<? super R>>() {
        @Override
        public void call(Notification<? super R> rNotification) {
            setRequestContextIfNeeded(currentRequestContext);
        }
    };

    Observable<R> execution;
      // If there is a timeout limit, the wrapped Observable will be converted to TimeOut-enabled
    if (properties.executionTimeoutEnabled().get()) {
          // Packing different Observable s according to different isolation strategies
        execution = executeCommandWithSpecifiedIsolation(_cmd)
                      // lift is a basic operator in rxjava that converts Observable into another Observable
                      // Packaged as Observable with timeout restrictions
                .lift(new HystrixObservableTimeoutOperator<R>(_cmd));
    } else {
        execution = executeCommandWithSpecifiedIsolation(_cmd);
    }

    return execution.doOnNext(markEmits)
            .doOnCompleted(markOnCompleted)
            .onErrorResumeNext(handleFallback)
            .doOnEach(setRequestContext);
}

executeCommandWithSpecifiedIsolation

Create different execution Observable based on different isolation strategies

private Observable<R> executeCommandWithSpecifiedIsolation(final AbstractCommand<R> _cmd) {
    if (properties.executionIsolationStrategy().get() == ExecutionIsolationStrategy.THREAD) {
        // mark that we are executing in a thread (even if we end up being rejected we still were a THREAD execution and not SEMAPHORE)
        return Observable.defer(new Func0<Observable<R>>() {
            @Override
            public Observable<R> call() {
                  // Because the source code is too long, here we only focus on the normal process, need to understand in detail can see the source code.
                if (threadState.compareAndSet(ThreadState.NOT_USING_THREAD, ThreadState.STARTED)) {
                    try {
                        return getUserExecutionObservable(_cmd);
                    } catch (Throwable ex) {
                        return Observable.error(ex);
                    }
                } else {
                    //command has already been unsubscribed, so return immediately
                    return Observable.error(new RuntimeException("unsubscribed before executing run()"));
                }
            }})
        .doOnTerminate(new Action0() {})
        .doOnUnsubscribe(new Action0() {})
        // Specifying execution on a thread is an important concept of thread scheduling in rxjava
        .subscribeOn(threadPool.getScheduler(new Func0<Boolean>() {
        }));
    } else { // Semaphore isolation strategy
        return Observable.defer(new Func0<Observable<R>>() {
                        // Logic is roughly the same as thread pool
        });
    }
}

getUserExecutionObservable

Get the logic that the user executes

private Observable<R> getUserExecutionObservable(final AbstractCommand<R> _cmd) {
    Observable<R> userObservable;

    try {
          // getExecutionObservable is an abstract method, implemented by HystrixCommand itself.
        userObservable = getExecutionObservable();
    } catch (Throwable ex) {
        // the run() method is a user provided implementation so can throw instead of using Observable.onError
        // so we catch it here and turn it into Observable.error
        userObservable = Observable.error(ex);
    }
        // Observable for other transits
    return userObservable
            .lift(new ExecutionHookApplication(_cmd))
            .lift(new DeprecatedOnRunHookApplication(_cmd));
}

lift operator

Lift can be converted into a new Observable, which is very similar to an agent. It can send data to the original Observable when subscribing, and then return it to the subscriber after processing by itself. The bottom layer of Map/FlatMap operator is actually implemented by lift.

getExecutionObservable

@Override
final protected Observable<R> getExecutionObservable() {
  return Observable.defer(new Func0<Observable<R>>() {
    @Override
    public Observable<R> call() {
      try {
        // The just operator is the Observable that is executed directly.
        // The run method is the business logic we implement: Hello World~
        return Observable.just(run());
      } catch (Throwable ex) {
        return Observable.error(ex);
      }
    }
  }).doOnSubscribe(new Action0() {
    @Override
    public void call() {
         // When executing a subscription, the execution thread is recorded as the current thread, and we can interrupt if necessary.
      executionThread.set(Thread.currentThread());
    }
  });
}

summary

I hope I can fill the buried pits one by one: fault tolerance mechanism, metrics, circuit breakers and so on.

Reference resources

1. Hystrix How it Works
2. ReactiveX official website
3. Ruan Yifeng: Chinese Technical Document Writing Standard
4. Analysis of RxJava lift principle