Apache ShenYu Is an asynchronous, high-performance, cross language, responsive API gateway.
In ShenYu gateway, the registration center is used to register the client information with ShenYu admin, and the admin synchronizes the information to the gateway through data synchronization. The gateway completes traffic filtering through these data. Client information mainly includes interface information and URI information.
This paper analyzes the source code based on shenyu-2.4.1. Please refer to the introduction of the official website Client access principle .
1. Principle of registry
When the client starts, it reads the interface information and uri information, and sends the data to Shenyu admin through the specified registration type.
The registration center in the figure requires the user to specify which registration type to use. ShenYu currently supports Http, Zookeeper, Etcd, Consul and Nacos for registration. Please refer to Client access configuration .
ShenYu introduced the Disruptor in the principle design of the registry. The Disruptor queue decouples data and operation, which is conducive to expansion. If too many registration requests lead to registration exceptions, it also has the function of data buffering.
As shown in the figure, the registry is divided into two parts. One is the registry client, which handles client data reading. The other is the register server of the registry server, which handles data writing from the load processing server (i.e. Shenyu admin). Send and receive data by specifying the registration type.
- Client: Generally speaking, it is a micro service, which can be spring MVC, spring cloud, dubbo, grpc, etc.
- Register client: Registry client, which reads the client interface and uri information.
- Disruptor: data and operation are decoupled, and data buffer is used.
- Register server: the registry server, here is Shenyu admin, which receives data, writes to the database, and sends data synchronization events.
- Registration type: specify the registration type and complete data registration. Currently, Http, Zookeeper, Etcd, Consul and Nacos are supported.
This paper analyzes the use of Http for registration, so the specific processing flow is as follows:
At the client, after the data is out of the queue, it transmits the data through http. At the server, it provides the corresponding interface to receive the data, and then writes it to the queue.
2. Client registration process
After the client starts, read the attribute information according to the relevant configuration, and then write it to the queue. Officially provided shenyu-examples-http As an example, start source code analysis. The official example is a microservice built by springboot. Please refer to the official website for the relevant configuration of the registration center Client access configuration .
2.1 load configuration and read attributes
First, use a figure to connect the following registry client initialization process:
We analyze the registration through http, so the following configuration is required:
shenyu:
register:
registerType: http
serverLists: http://localhost:9095
client:
http:
props:
contextPath: /http
appName: http
port: 8189
isFull: false
The meaning of each attribute is as follows:
- registerType: service registration type, fill in http.
- serverList: when registering for HTTP, fill in the address of Shenyu admin project. Note that http: / /, and multiple addresses are separated by English commas.
- Port: the startup port of your project. Currently, springmvc/tars/grpc needs to be filled in.
- contextPath: the routing prefix of your mvc project in shenyu gateway, such as / order, / product, etc. the gateway will route according to your prefix.
- appName: your app name. If it is not configured, it will take the value of spring.application.name by default.
- isFull: set true to represent your entire service, and false to represent some of your controller s; Currently applicable to spring MVC / spring cloud.
After the project is started, the configuration file will be loaded first, the attribute information will be read, and the corresponding Bean will be generated.
The first Configuration file read is shenyusprinmvcclientconfiguration, which is the http registered Configuration class of shenyu client. It is represented as a Configuration class through @ Configuration, and other Configuration classes are introduced through @ ImportAutoConfiguration. Create spring mvcclientbeanpostprocessor, which mainly processes metadata. Create a ContextRegisterListener, which mainly processes URI information.
/**
* shenyu Client http registration configuration class
*/
@Configuration
@ImportAutoConfiguration(ShenyuClientCommonBeanConfiguration.class)
public class ShenyuSpringMvcClientConfiguration {
//Create spring mvcclientbeanpostprocessor, which mainly processes metadata
@Bean
public SpringMvcClientBeanPostProcessor springHttpClientBeanPostProcessor(final ShenyuClientConfig clientConfig,final ShenyuClientRegisterRepository shenyuClientRegisterRepository) {
return new SpringMvcClientBeanPostProcessor(clientConfig.getClient().get(RpcTypeEnum.HTTP.getName()), shenyuClientRegisterRepository);
}
// Create a ContextRegisterListener, which mainly processes URI information
@Bean
public ContextRegisterListener contextRegisterListener(final ShenyuClientConfig clientConfig) {
return new ContextRegisterListener(clientConfig.getClient().get(RpcTypeEnum.HTTP.getName()));
}
}
ShenyuClientCommonBeanConfiguration is a general configuration class of shenyu client, which will create a bean common to the registry client.
- Create ShenyuClientRegisterRepository through the factory class.
- Create ShenyuRegisterCenterConfig and read the shenyu.register property configuration.
- Create ShenyuClientConfig and read the shenyu.client property configuration.
/**
* shenyu Client general configuration class
*/
@Configuration
public class ShenyuClientCommonBeanConfiguration {
// Create ShenyuClientRegisterRepository through the factory class.
@Bean
public ShenyuClientRegisterRepository shenyuClientRegisterRepository(final ShenyuRegisterCenterConfig config) {
return ShenyuClientRegisterRepositoryFactory.newInstance(config);
}
// Create ShenyuRegisterCenterConfig and read the shenyu.register property configuration
@Bean
@ConfigurationProperties(prefix = "shenyu.register")
public ShenyuRegisterCenterConfig shenyuRegisterCenterConfig() {
return new ShenyuRegisterCenterConfig();
}
// Create ShenyuClientConfig and read the property configuration of shenyu.client
@Bean
@ConfigurationProperties(prefix = "shenyu")
public ShenyuClientConfig shenyuClientConfig() {
return new ShenyuClientConfig();
}
}
2.2 HttpClientRegisterRepository for registration
The ShenyuClientRegisterRepository generated in the above configuration file is the specific implementation of client registration. It is an interface, and its implementation classes are as follows.
- HttpClientRegisterRepository: register through http;
- ConsulClientRegisterRepository: register through Consul;
- EtcdClientRegisterRepository: register through Etcd;
- NacosClientRegisterRepository: register through nacos;
- Zookeeper clientregisterrepository is registered through zookeeper.
The specific method is realized by SPI loading. The implementation logic is as follows:
/**
* Load ShenyuClientRegisterRepository
*/
public final class ShenyuClientRegisterRepositoryFactory {
private static final Map<String, ShenyuClientRegisterRepository> REPOSITORY_MAP = new ConcurrentHashMap<>();
/**
* Create ShenyuClientRegisterRepository
*/
public static ShenyuClientRegisterRepository newInstance(final ShenyuRegisterCenterConfig shenyuRegisterCenterConfig) {
if (!REPOSITORY_MAP.containsKey(shenyuRegisterCenterConfig.getRegisterType())) {
// Load through SPI, and the type is determined by registerType
ShenyuClientRegisterRepository result = ExtensionLoader.getExtensionLoader(ShenyuClientRegisterRepository.class).getJoin(shenyuRegisterCenterConfig.getRegisterType());
//perform an initialization operation
result.init(shenyuRegisterCenterConfig);
ShenyuClientShutdownHook.set(result, shenyuRegisterCenterConfig.getProps());
REPOSITORY_MAP.put(shenyuRegisterCenterConfig.getRegisterType(), result);
return result;
}
return REPOSITORY_MAP.get(shenyuRegisterCenterConfig.getRegisterType());
}
}
The loading type is specified by registerType, that is, the type specified in the configuration file:
shenyu:
register:
registerType: http
serverLists: http://localhost:9095
We specify http, so we will load HttpClientRegisterRepository. After the object is created successfully, the initialization method init() executed is as follows:
@Join
public class HttpClientRegisterRepository implements ShenyuClientRegisterRepository {
@Override
public void init(final ShenyuRegisterCenterConfig config) {
this.serverList = Lists.newArrayList(Splitter.on(",").split(config.getServerLists()));
}
// Omit other logic temporarily
}
Read the serverLists in the configuration file, that is, the address of sheenyu admin, to prepare for subsequent data transmission. Class annotation @ Join is used to load SPI.
SPI, fully known as Service Provider Interface, is a built-in service discovery function in JDK and a dynamic replacement discovery mechanism.
shenyu-spi It is the SPI extension implementation customized by Apache ShenYu gateway. The design and implementation principle refer to Dubbo's SPI extension implementation .
2.3 SpringMvcClientBeanPostProcessor for Constructing Metadata
Create SpringMvcClientBeanPostProcessor, which is responsible for the construction and registration of metadata. Its constructor logic is as follows:
/**
* spring mvc Post processor of client bean
*/
public class SpringMvcClientBeanPostProcessor implements BeanPostProcessor {
/**
* Instantiation by constructor
*/
public SpringMvcClientBeanPostProcessor(final PropertiesConfig clientConfig,
final ShenyuClientRegisterRepository shenyuClientRegisterRepository) {
// Read configuration properties
Properties props = clientConfig.getProps();
// Obtain port information and verify
int port = Integer.parseInt(props.getProperty(ShenyuClientConstants.PORT));
if (port <= 0) {
String errorMsg = "http register param must config the port must > 0";
LOG.error(errorMsg);
throw new ShenyuClientIllegalArgumentException(errorMsg);
}
// Get appName
this.appName = props.getProperty(ShenyuClientConstants.APP_NAME);
// Get contextPath
this.contextPath = props.getProperty(ShenyuClientConstants.CONTEXT_PATH);
// Verify appName and contextPath
if (StringUtils.isBlank(appName) && StringUtils.isBlank(contextPath)) {
String errorMsg = "http register param must config the appName or contextPath";
LOG.error(errorMsg);
throw new ShenyuClientIllegalArgumentException(errorMsg);
}
// Get isFull
this.isFull = Boolean.parseBoolean(props.getProperty(ShenyuClientConstants.IS_FULL, Boolean.FALSE.toString()));
// Start event Publishing
publisher.start(shenyuClientRegisterRepository);
}
// Other logic is omitted for the time being
@Override
public Object postProcessAfterInitialization(@NonNull final Object bean, @NonNull final String beanName) throws BeansException {
// Other logic is omitted for the time being
}
}
In the constructor, you mainly read the attribute information and then verify it.
shenyu:
client:
http:
props:
contextPath: /http
appName: http
port: 8189
isFull: false
Finally, publisher.start() is executed to start event publishing and prepare for registration.
- ShenyuClientRegisterEventPublisher
ShenyuClientRegisterEventPublisher is implemented through singleton mode, mainly generating metadata and URI subscribers (later used for Data Publishing), and then starting the Disruptor queue. A common method publishEvent() is provided to publish events and send data to the Disruptor queue.
public class ShenyuClientRegisterEventPublisher {
// private variable
private static final ShenyuClientRegisterEventPublisher INSTANCE = new ShenyuClientRegisterEventPublisher();
private DisruptorProviderManage providerManage;
private RegisterClientExecutorFactory factory;
/**
* Expose static methods
*
* @return ShenyuClientRegisterEventPublisher instance
*/
public static ShenyuClientRegisterEventPublisher getInstance() {
return INSTANCE;
}
/**
* Start Method execution
*
* @param shenyuClientRegisterRepository shenyuClientRegisterRepository
*/
public void start(final ShenyuClientRegisterRepository shenyuClientRegisterRepository) {
// Create client registration factory class
factory = new RegisterClientExecutorFactory();
// Add metadata subscriber
factory.addSubscribers(new ShenyuClientMetadataExecutorSubscriber(shenyuClientRegisterRepository));
// Add URI subscriber
factory.addSubscribers(new ShenyuClientURIExecutorSubscriber(shenyuClientRegisterRepository));
// Start the Disruptor queue
providerManage = new DisruptorProviderManage(factory);
providerManage.startup();
}
/**
* Publish events and send data to the Disruptor queue
*
* @param <T> the type parameter
* @param data the data
*/
public <T> void publishEvent(final T data) {
DisruptorProvider<Object> provider = providerManage.getProvider();
provider.onData(f -> f.setData(data));
}
}
The constructor logic of Spring mvcclientbeanpostprocessor has been analyzed, mainly reading the property configuration, creating metadata and URI subscribers, and starting the Disruptor queue. Note that it implements the BeanPostProcessor, an interface provided by Spring. In the life cycle of a Bean, the postprocessor's postProcessAfterInitialization() method will be executed before it is actually used.
- postProcessAfterInitialization() method
As a post processor, the function of spring mvcclientbeanpost processor is to read metadata in annotations and register with admin.
// Post Processors
public class SpringMvcClientBeanPostProcessor implements BeanPostProcessor {
// Other logic is omitted
// Post processor: read the metadata in the annotation and register with admin
@Override
public Object postProcessAfterInitialization(@NonNull final Object bean, @NonNull final String beanName) throws BeansException {
// The configuration attribute, if isFull=true, indicates that the entire microservice is registered
if (isFull) {
return bean;
}
// Gets the Controller annotation of the current bean
Controller controller = AnnotationUtils.findAnnotation(bean.getClass(), Controller.class);
// Get the RequestMapping annotation of the current bean
RequestMapping requestMapping = AnnotationUtils.findAnnotation(bean.getClass(), RequestMapping.class);
// If this bean is an interface
if (controller != null || requestMapping != null) {
// Get the shenyusprinmvcclient annotation of the current bean
ShenyuSpringMvcClient clazzAnnotation = AnnotationUtils.findAnnotation(bean.getClass(), ShenyuSpringMvcClient.class);
String prePath = "";
//If there is no shenyusprinmvcclient annotation, it returns, indicating that this interface does not need to be registered
if (Objects.isNull(clazzAnnotation)) {
return bean;
}
//If the path attribute in the shenyusprinmvcclient annotation includes *, it means that the entire interface is registered
if (clazzAnnotation.path().indexOf("*") > 1) {
// Build metadata and send registration events
publisher.publishEvent(buildMetaDataDTO(clazzAnnotation, prePath));
return bean;
}
prePath = clazzAnnotation.path();
// Gets all methods of the current bean
final Method[] methods = ReflectionUtils.getUniqueDeclaredMethods(bean.getClass());
// traversal method
for (Method method : methods) {
// Get the annotation shenyusprinmvcclient on the current method
ShenyuSpringMvcClient shenyuSpringMvcClient = AnnotationUtils.findAnnotation(method, ShenyuSpringMvcClient.class);
// If there is an annotation shenyusprinmvcclient on the method, it means that the method needs to be registered
if (Objects.nonNull(shenyuSpringMvcClient)) {
// Build metadata and send registration events
publisher.publishEvent(buildMetaDataDTO(shenyuSpringMvcClient, prePath));
}
}
}
return bean;
}
// Construct metadata
private MetaDataRegisterDTO buildMetaDataDTO(final ShenyuSpringMvcClient shenyuSpringMvcClient, final String prePath) {
// contextPath context name
String contextPath = this.contextPath;
// appName app name
String appName = this.appName;
// Path register path
String path;
if (StringUtils.isEmpty(contextPath)) {
path = prePath + shenyuSpringMvcClient.path();
} else {
path = contextPath + prePath + shenyuSpringMvcClient.path();
}
// desc description information
String desc = shenyuSpringMvcClient.desc();
// ruleName is the rule name. If it is not filled in, it is consistent with path
String configRuleName = shenyuSpringMvcClient.ruleName();
String ruleName = StringUtils.isBlank(configRuleName) ? path : configRuleName;
// Building metadata
return MetaDataRegisterDTO.builder()
.contextPath(contextPath)
.appName(appName)
.path(path)
.pathDesc(desc)
.rpcType(RpcTypeEnum.HTTP.getName())
.enabled(shenyuSpringMvcClient.enabled())
.ruleName(ruleName)
.registerMetaData(shenyuSpringMvcClient.registerMetaData())
.build();
}
}
In the post processor, the configuration attribute needs to be read. If isFull=true, it means that the whole microservice is registered. Obtain the Controller annotation, RequestMapping annotation and shenyusprinmvcclient annotation of the current bean, and judge whether the current bean is an interface by reading these annotation information? Does the interface need to be registered? Does the method need to be registered? Then build metadata according to the attributes in shenyusprinmvcclient annotation, and finally publish the event through publisher.publishEvent() for registration.
The Controller annotation and RequestMapping annotation are provided by Spring, which you should be familiar with, but please repeat them. Shenyusprinmvcclient annotation is provided by Apache ShenYu to register the Spring MVC client. Its definition is as follows:
/**
* shenyu Client interface, used on methods or classes
*/
@Retention(RetentionPolicy.RUNTIME)
@Target({ElementType.TYPE, ElementType.METHOD})
public @interface ShenyuSpringMvcClient {
// Path register path
String path();
// ruleName rule name
String ruleName() default "";
// desc description information
String desc() default "";
// Enabled is enabled
boolean enabled() default true;
// registerMetaData registration metadata
boolean registerMetaData() default false;
}
Its use is as follows:
- Register the entire interface
@RestController
@RequestMapping("/test")
@ShenyuSpringMvcClient(path = "/test/**") // Indicates that the entire interface is registered
public class HttpTestController {
//......
}
- Register current method
@RestController
@RequestMapping("/order")
@ShenyuSpringMvcClient(path = "/order")
public class OrderController {
/**
* Save order dto.
*
* @param orderDTO the order dto
* @return the order dto
*/
@PostMapping("/save")
@ShenyuSpringMvcClient(path = "/save", desc = "Save order") // Register current method
public OrderDTO save(@RequestBody final OrderDTO orderDTO) {
orderDTO.setName("hello world save order");
return orderDTO;
}
- publisher.publishEvent() publishes a registration event
This method will send data to the Disruptor queue. More details about the Disruptor queue will not be introduced here, which will not affect the analysis and registration process.
After the data is sent, consumers in the Disruptor queue will process the data and consume it.
- QueueConsumer consumption data
QueueConsumer is a consumer, which implements the WorkHandler interface, and its creation process is in the provider manage. Startup () logic. The WorkHandler interface is the data consumption interface of the disruptor. Only one method is onEvent().
package com.lmax.disruptor;
public interface WorkHandler<T> {
void onEvent(T var1) throws Exception;
}
QueueConsumer rewrites the onEvent() method. The main logic is to generate consumption tasks and then execute them in the process pool.
/**
*
* Queue consumer
*/
public class QueueConsumer<T> implements WorkHandler<DataEvent<T>> {
// Other logic is omitted
@Override
public void onEvent(final DataEvent<T> t) {
if (t != null) {
// Create queue consumption task through factory
QueueConsumerExecutor<T> queueConsumerExecutor = factory.create();
// Save data
queueConsumerExecutor.setData(t.getData());
// help gc
t.setData(null);
// Put it in the process pool to perform consumption tasks
executor.execute(queueConsumerExecutor);
}
}
}
Queueconsumerexecution is a task executed in the thread pool. It implements the Runnable interface. There are two specific implementation classes:
- RegisterClientConsumerExecutor: client consumer executor;
- RegisterServerConsumerExecutor: the server-side consumer executor.
As the name suggests, one is responsible for processing client tasks and the other is responsible for processing server tasks (the server is admin, which is analyzed below).
- RegisterClientConsumerExecutor consumer executor
The rewritten run() logic is as follows:
public final class RegisterClientConsumerExecutor extends QueueConsumerExecutor<DataTypeParent> {
//......
@Override
public void run() {
// get data
DataTypeParent dataTypeParent = getData();
// Call the corresponding processor for processing according to the data type
subscribers.get(dataTypeParent.getType()).executor(Lists.newArrayList(dataTypeParent));
}
}
Call different processors to perform corresponding tasks according to different data types. There are two data types. One is metadata, which records the client registration information. One is URI data, which records client service information.
//data type
public enum DataType {
// metadata
META_DATA,
// URI data
URI,
}
- ExecutorSubscriber#executor() executor subscriber
Executor subscribers are also divided into two categories: one is to process metadata and the other is to process URI s. There are two on the client side and two on the server side, so there are four in total.
- ShenyuClientMetadataExecutorSubscriber#executor()
The metadata processing logic on the client side is: go through the metadata information and call the interface method persistInterface() to publish the data.
public class ShenyuClientMetadataExecutorSubscriber implements ExecutorTypeSubscriber<MetaDataRegisterDTO> {
//......
@Override
public DataType getType() {
return DataType.META_DATA; // metadata
}
@Override
public void executor(final Collection<MetaDataRegisterDTO> metaDataRegisterDTOList) {
for (MetaDataRegisterDTO metaDataRegisterDTO : metaDataRegisterDTOList) {
// Call the interface method persistInterface() to publish the data
shenyuClientRegisterRepository.persistInterface(metaDataRegisterDTO);
}
}
}
- ShenyuClientRegisterRepository#persistInterface()
ShenyuClientRegisterRepository is an interface used to represent client data registration. At present, there are five implementation classes, and each represents a registration method.
- ConsulClientRegisterRepository: realize client registration through Consul;
- EtcdClientRegisterRepository: realize client registration through Etcd;
- HttpClientRegisterRepository: realize client registration through Http;
- NacosClientRegisterRepository: realize client registration through Nacos;
- Zookeeper clientregisterrepository: realize client registration through zookeeper;
As can be seen from the figure, the loading of the registry is completed through SPI. As mentioned earlier, in the client general configuration file, the specific class loading is completed by specifying the properties in the configuration file.
/**
* Load ShenyuClientRegisterRepository
*/
public final class ShenyuClientRegisterRepositoryFactory {
private static final Map<String, ShenyuClientRegisterRepository> REPOSITORY_MAP = new ConcurrentHashMap<>();
/**
* Create ShenyuClientRegisterRepository
*/
public static ShenyuClientRegisterRepository newInstance(final ShenyuRegisterCenterConfig shenyuRegisterCenterConfig) {
if (!REPOSITORY_MAP.containsKey(shenyuRegisterCenterConfig.getRegisterType())) {
// Load through SPI, and the type is determined by registerType
ShenyuClientRegisterRepository result = ExtensionLoader.getExtensionLoader(ShenyuClientRegisterRepository.class).getJoin(shenyuRegisterCenterConfig.getRegisterType());
//perform an initialization operation
result.init(shenyuRegisterCenterConfig);
ShenyuClientShutdownHook.set(result, shenyuRegisterCenterConfig.getProps());
REPOSITORY_MAP.put(shenyuRegisterCenterConfig.getRegisterType(), result);
return result;
}
return REPOSITORY_MAP.get(shenyuRegisterCenterConfig.getRegisterType());
}
}
The source code analysis in this paper is based on Http registration, so we first analyze HttpClientRegisterRepository, and then analyze other registration methods later.
Registering through http is very simple, that is, calling the tool class to send http requests. The registration metadata and URI are the same method called, doRegister(). Just specify the interface and type.
- /Shenyu client / register metadata: the interface provided by the server is used to register metadata.
- /Shenyu client / register URI: the interface provided by the server is used to register the URI.
@Join
public class HttpClientRegisterRepository implements ShenyuClientRegisterRepository {
// The interface provided by the server is used to register metadata
private static final String META_PATH = "/shenyu-client/register-metadata";
// The interface provided by the server is used to register the URI
private static final String URI_PATH = "/shenyu-client/register-uri";
//Registration URI
@Override
public void persistURI(final URIRegisterDTO registerDTO) {
doRegister(registerDTO, URI_PATH, Constants.URI);
}
//Registration interface (i.e. metadata information)
@Override
public void persistInterface(final MetaDataRegisterDTO metadata) {
doRegister(metadata, META_PATH, META_TYPE);
}
// Register
private <T> void doRegister(final T t, final String path, final String type) {
// Traverse the list of admin services (admin may be a cluster)
for (String server : serverList) {
try {
// Call the tool class to send an http request
RegisterUtils.doRegister(GsonUtils.getInstance().toJson(t), server + path, type);
return;
} catch (Exception e) {
LOGGER.error("register admin url :{} is fail, will retry", server);
}
}
}
}
After serializing the data, send the data through OkHttp.
public final class RegisterUtils {
//......
// Send data via OkHttp
public static void doRegister(final String json, final String url, final String type) throws IOException {
String result = OkHttpTools.getInstance().post(url, json);
if (Objects.equals(SUCCESS, result)) {
LOGGER.info("{} client register success: {} ", type, json);
} else {
LOGGER.error("{} client register error: {} ", type, json);
}
}
}
So far, the logic analysis of the client registering metadata through http is finished. Summary: construct metadata by reading custom annotation information, send the data to the Disruptor queue, then consume the data from the queue, put the consumer into the thread pool for execution, and finally send an http request to admin.
The source code analysis process of the client metadata registration process is completed, and the flow chart is described as follows:
2.4 build ContextRegisterListener of URI
Create a ContextRegisterListener, which is responsible for the construction and registration of client URI data. Its creation is completed in the configuration file.
@Configuration
@ImportAutoConfiguration(ShenyuClientCommonBeanConfiguration.class)
public class ShenyuSpringMvcClientConfiguration {
// ......
// Create ContextRegisterListener
@Bean
public ContextRegisterListener contextRegisterListener(final ShenyuClientConfig clientConfig) {
return new ContextRegisterListener(clientConfig.getClient().get(RpcTypeEnum.HTTP.getName()));
}
}
ContextRegisterListener implements the ApplicationListener interface and rewrites the onApplicationEvent() method. When a Spring event occurs, the method will execute.
// Implements the ApplicationListener interface
public class ContextRegisterListener implements ApplicationListener<ContextRefreshedEvent> {
//......
//Instantiation through constructor
public ContextRegisterListener(final PropertiesConfig clientConfig) {
// Read shenyu.client.http configuration information
Properties props = clientConfig.getProps();
// isFull registers the entire service
this.isFull = Boolean.parseBoolean(props.getProperty(ShenyuClientConstants.IS_FULL, Boolean.FALSE.toString()));
// contextPath context path
String contextPath = props.getProperty(ShenyuClientConstants.CONTEXT_PATH);
this.contextPath = contextPath;
if (isFull) {
if (StringUtils.isBlank(contextPath)) {
String errorMsg = "http register param must config the contextPath";
LOG.error(errorMsg);
throw new ShenyuClientIllegalArgumentException(errorMsg);
}
this.contextPath = contextPath + "/**";
}
// Port client port information
int port = Integer.parseInt(props.getProperty(ShenyuClientConstants.PORT));
// appName app name
this.appName = props.getProperty(ShenyuClientConstants.APP_NAME);
// host information
this.host = props.getProperty(ShenyuClientConstants.HOST);
this.port = port;
}
// This method executes when a context refresh event ContextRefreshedEvent occurs
@Override
public void onApplicationEvent(@NonNull final ContextRefreshedEvent contextRefreshedEvent) {
//Ensure that the contents of the method are executed only once
if (!registered.compareAndSet(false, true)) {
return;
}
// If isFull=true, it means to register the whole service, build metadata and register
if (isFull) {
publisher.publishEvent(buildMetaDataDTO());
}
// Build URI data and register
publisher.publishEvent(buildURIRegisterDTO());
}
// Build URI data
private URIRegisterDTO buildURIRegisterDTO() {
String host = IpUtils.isCompleteHost(this.host) ? this.host : IpUtils.getHost(this.host);
return URIRegisterDTO.builder()
.contextPath(this.contextPath)
.appName(appName)
.host(host)
.port(port)
.rpcType(RpcTypeEnum.HTTP.getName())
.build();
}
// Building metadata
private MetaDataRegisterDTO buildMetaDataDTO() {
String contextPath = this.contextPath;
String appName = this.appName;
return MetaDataRegisterDTO.builder()
.contextPath(contextPath)
.appName(appName)
.path(contextPath)
.rpcType(RpcTypeEnum.HTTP.getName())
.enabled(true)
.ruleName(contextPath)
.build();
}
}
The constructor mainly reads the attribute configuration.
onApplicationEvent() method is executed when a Spring event occurs. The parameter here is ContextRefreshedEvent, which indicates the context refresh event. When the Spring container is ready, execute the logic here: if isFull=true, it means registering the whole service, building metadata and registering. The post processor Spring mvcclientbeanpostprocessor analyzed above does not handle the case of isFull=true, so it is processed here. Then build the URI data and register.
ContextRefreshedEvent is a Spring built-in event. This event is triggered when the ApplicationContext is initialized or refreshed. This can also occur using the refresh() method in the ConfigurableApplicationContext interface. Initialization here means that all beans are successfully loaded, post-processing beans are detected and activated, all singleton beans are pre instantiated, and the ApplicationContext container is ready and available.
The registration logic is completed through publisher.publishEvent(). It has been analyzed before: write data to the Disruptor queue, consume data from it, and finally process it through the ExecutorSubscriber.
- ExecutorSubscriber#executor()
There are two types of executor subscribers: one is processing metadata and the other is processing URI. There are two on the client side and two on the server side, so there are four in total.
Here is the registration URI information, so the execution class is shenyuclienturiexecursubscriber.
- ShenyuClientURIExecutorSubscriber#executor()
The main logic is to traverse the URI data set and realize data registration through the persistURI() method.
public class ShenyuClientURIExecutorSubscriber implements ExecutorTypeSubscriber<URIRegisterDTO> {
//......
@Override
public DataType getType() {
return DataType.URI; //The data type is URI
}
// Register URI data
@Override
public void executor(final Collection<URIRegisterDTO> dataList) {
for (URIRegisterDTO uriRegisterDTO : dataList) {
Stopwatch stopwatch = Stopwatch.createStarted();
while (true) {
try (Socket ignored = new Socket(uriRegisterDTO.getHost(), uriRegisterDTO.getPort())) {
break;
} catch (IOException e) {
long sleepTime = 1000;
// maybe the port is delay exposed
if (stopwatch.elapsed(TimeUnit.SECONDS) > 5) {
LOG.error("host:{}, port:{} connection failed, will retry",
uriRegisterDTO.getHost(), uriRegisterDTO.getPort());
// If the connection fails for a long time, Increase sleep time
if (stopwatch.elapsed(TimeUnit.SECONDS) > 180) {
sleepTime = 10000;
}
}
try {
TimeUnit.MILLISECONDS.sleep(sleepTime);
} catch (InterruptedException ex) {
ex.printStackTrace();
}
}
}
//Add a hook and stop the client gracefully
ShenyuClientShutdownHook.delayOtherHooks();
// Registration URI
shenyuClientRegisterRepository.persistURI(uriRegisterDTO);
}
}
}
The while(true) loop in the code is to ensure that the client has been successfully started and can be connected through host and port.
The following logic is: add a hook function to stop the client gracefully.
Data registration is implemented through the persistURI() method. The whole logic has also been analyzed before. Finally, http is initiated to Shenyu admin through OkHttp client, and the URI is registered through http.
After the analysis, the registration logic of the client is analyzed. The constructed metadata and URI data are sent to the Disruptor queue, where they are consumed, read the data, and send the data to the admin through http.
The source code analysis of the client URI registration process is completed, and the flow chart is as follows:
3. Server registration process
3.1 registration interface ShenyuHttpRegistryController
According to the previous analysis, the server provides two registered interfaces:
- /Shenyu client / register metadata: the interface provided by the server is used to register metadata.
- /Shenyu client / register URI: the interface provided by the server is used to register the URI.
These two interfaces are located in ShenyuHttpRegistryController, which implements the ShenyuServerRegisterRepository interface and is the implementation class registered by the server. It is marked with @ Join, indicating loading through SPI.
// shenuyu client interface
@RequestMapping("/shenyu-client")
@Join
public class ShenyuHttpRegistryController implements ShenyuServerRegisterRepository {
private ShenyuServerRegisterPublisher publisher;
@Override
public void init(final ShenyuServerRegisterPublisher publisher, final ShenyuRegisterCenterConfig config) {
this.publisher = publisher;
}
// Registration metadata
@PostMapping("/register-metadata")
@ResponseBody
public String registerMetadata(@RequestBody final MetaDataRegisterDTO metaDataRegisterDTO) {
publish(metaDataRegisterDTO);
return ShenyuResultMessage.SUCCESS;
}
// Registration URI
@PostMapping("/register-uri")
@ResponseBody
public String registerURI(@RequestBody final URIRegisterDTO uriRegisterDTO) {
publish(uriRegisterDTO);
return ShenyuResultMessage.SUCCESS;
}
// Publish registration events
private <T> void publish(final T t) {
publisher.publish(Collections.singletonList(t));
}
}
After the two registration interfaces get the data, they call the publish() method to publish the data to the Disruptor queue.
- ShenyuServerRegisterRepository interface
ShenyuServerRegisterRepository interface is a service registration interface. It has five implementation classes, indicating that there are five registration methods:
- ConsulServerRegisterRepository: register through Consul;
- EtcdServerRegisterRepository: register through Etcd;
- NacosServerRegisterRepository: register through Nacos;
- ShenyuHttpRegistryController: register through Http;
- Zookeeper server registerreposition: register through zookeeper.
The specific method is specified through the configuration file, and then loaded through SPI.
Configure the registration method in the application.yml file in Shenyu admin. registerType specifies the registration type. When registering with http, serverLists does not need to be filled in. For more configuration instructions, please refer to the official website Client access configuration .
shenyu:
register:
registerType: http
serverLists:
- RegisterCenterConfiguration load configuration
After introducing related dependency and attribute configuration, the configuration file will be loaded first when Shenyu admin is started. The configuration file class related to the registry is RegisterCenterConfiguration.
// Registry configuration class
@Configuration
public class RegisterCenterConfiguration {
// Read configuration properties
@Bean
@ConfigurationProperties(prefix = "shenyu.register")
public ShenyuRegisterCenterConfig shenyuRegisterCenterConfig() {
return new ShenyuRegisterCenterConfig();
}
//Create ShenyuServerRegisterRepository for server-side registration
@Bean
public ShenyuServerRegisterRepository shenyuServerRegisterRepository(final ShenyuRegisterCenterConfig shenyuRegisterCenterConfig, final List<ShenyuClientRegisterService> shenyuClientRegisterService) {
// 1. Get the registration type from the configuration attribute
String registerType = shenyuRegisterCenterConfig.getRegisterType();
// 2. Load the implementation class with SPI method by registering the type
ShenyuServerRegisterRepository registerRepository = ExtensionLoader.getExtensionLoader(ShenyuServerRegisterRepository.class).getJoin(registerType);
// 3. Get publisher and write data to the Disruptor queue
RegisterServerDisruptorPublisher publisher = RegisterServerDisruptorPublisher.getInstance();
// 4. Register Service, rpctype - > registerservice
Map<String, ShenyuClientRegisterService> registerServiceMap = shenyuClientRegisterService.stream().collect(Collectors.toMap(ShenyuClientRegisterService::rpcType, e -> e));
// 5. Preparation for event release
publisher.start(registerServiceMap);
// 6. Initialization of registration
registerRepository.init(publisher, shenyuRegisterCenterConfig);
return registerRepository;
}
}
Two bean s are generated in the configuration class:
-
shenyuRegisterCenterConfig: read the attribute configuration;
-
shenyuServerRegisterRepository: used for server registration.
During the process of creating shenyuServerRegisterRepository, a series of preparations were also made:
-
1. Get the registration type from the configuration attribute.
-
2. Register the type and load the implementation class with SPI method: for example, if the specified type is http, ShenyuHttpRegistryController will be loaded.
-
3. Get publisher and write data to the Disruptor queue.
-
4. Register Service, rpctype - > registerservice: get the registered Service. Each rpc has a corresponding Service. The client of this article is built through springboot, which belongs to http type. There are other client types: dubbo, Spring Cloud, gRPC, etc.
-
5. Preparations for event Publishing: add server metadata and URI subscribers to process data. And start the Disruptor queue.
-
6. Registration initialization: http type registration initialization is to save publisher.
-
RegisterServerDisruptorPublisher#publish()
The publisher whose server writes data to the Disruptor queue is built through the singleton mode.
public class RegisterServerDisruptorPublisher implements ShenyuServerRegisterPublisher {
//Private property
private static final RegisterServerDisruptorPublisher INSTANCE = new RegisterServerDisruptorPublisher();
//Expose static methods to get instances
public static RegisterServerDisruptorPublisher getInstance() {
return INSTANCE;
}
//Prepare for event publishing, add server metadata and URI subscribers, and process data. And start the Disruptor queue.
public void start(final Map<String, ShenyuClientRegisterService> shenyuClientRegisterService) {
//Server registered factory
factory = new RegisterServerExecutorFactory();
//Add URI data subscriber
factory.addSubscribers(new URIRegisterExecutorSubscriber(shenyuClientRegisterService));
//Add metadata subscriber
factory.addSubscribers(new MetadataExecutorSubscriber(shenyuClientRegisterService));
//Start the Disruptor queue
providerManage = new DisruptorProviderManage(factory);
providerManage.startup();
}
// Write data to queue
@Override
public <T> void publish(final T data) {
DisruptorProvider<Object> provider = providerManage.getProvider();
provider.onData(f -> f.setData(data));
}
@Override
public void close() {
providerManage.getProvider().shutdown();
}
}
The loading of the configuration file can be regarded as the initialization process of the registry server, which is described as follows:
3.2 consumption data QueueConsumer
In the previous section, we analyzed the over consumption data of the client-side disruptor queue. The same logic applies to the server, except that the executor of the task has changed.
QueueConsumer is a consumer, which implements the WorkHandler interface, and its creation process is in the provider manage. Startup () logic. The WorkHandler interface is the data consumption interface of the disruptor. Only one method is onEvent().
package com.lmax.disruptor;
public interface WorkHandler<T> {
void onEvent(T var1) throws Exception;
}
QueueConsumer rewrites the onEvent() method. The main logic is to generate consumption tasks and then execute them in the process pool.
/**
*
* Queue consumer
*/
public class QueueConsumer<T> implements WorkHandler<DataEvent<T>> {
// Other logic is omitted
@Override
public void onEvent(final DataEvent<T> t) {
if (t != null) {
// Create queue consumption task through factory
QueueConsumerExecutor<T> queueConsumerExecutor = factory.create();
// Save data
queueConsumerExecutor.setData(t.getData());
// help gc
t.setData(null);
// Put it in the process pool to perform consumption tasks
executor.execute(queueConsumerExecutor);
}
}
}
Queueconsumerexecution is a task executed in the thread pool. It implements the Runnable interface. There are two specific implementation classes:
- RegisterClientConsumerExecutor: client consumer executor;
- RegisterServerConsumerExecutor: the server-side consumer executor.
As the name suggests, one is responsible for handling client tasks and the other is responsible for handling server tasks.
- RegisterServerConsumerExecutor#run()
RegisterServerConsumerExecutor is a service-side consumer executor. It indirectly implements the Runnable interface through queueconsumereexecutor and rewrites the run() method.
public final class RegisterServerConsumerExecutor extends QueueConsumerExecutor<List<DataTypeParent>> {
// ...
@Override
public void run() {
//Get the data from the disruptor queue
List<DataTypeParent> results = getData();
// data verification
results = results.stream().filter(data -> isValidData(data)).collect(Collectors.toList());
if (CollectionUtils.isEmpty(results)) {
return;
}
//Perform actions based on type
getType(results).executor(results);
}
// Get subscribers by type
private ExecutorSubscriber getType(final List<DataTypeParent> list) {
DataTypeParent result = list.get(0);
return subscribers.get(result.getType());
}
}
- ExecutorSubscriber#executor()
There are two types of executor subscribers: one is processing metadata and the other is processing URI. There are two on the client side and two on the server side, so there are four in total.
- MetadataExecutorSubscriber#executor()
If it is registration metadata, it is implemented through MetadataExecutorSubscriber#executor(): get the registration Service according to the type and call register().
public class MetadataExecutorSubscriber implements ExecutorTypeSubscriber<MetaDataRegisterDTO> {
//......
@Override
public DataType getType() {
return DataType.META_DATA; // Metadata Type
}
@Override
public void executor(final Collection<MetaDataRegisterDTO> metaDataRegisterDTOList) {
// History metadata list
for (MetaDataRegisterDTO metaDataRegisterDTO : metaDataRegisterDTOList) {
// Get registered Service by type
ShenyuClientRegisterService shenyuClientRegisterService = this.shenyuClientRegisterService.get(metaDataRegisterDTO.getRpcType());
Objects.requireNonNull(shenyuClientRegisterService);
// Register metadata and lock it to ensure sequential execution and prevent concurrency errors
synchronized (ShenyuClientRegisterService.class) {
shenyuClientRegisterService.register(metaDataRegisterDTO);
}
}
}
}
- URIRegisterExecutorSubscriber#executor()
If it is registration metadata, it is implemented through uriregisterexecutiorsubscriber#executor(): build URI data, find Service according to registration type, and realize registration through registerURI method.
public class URIRegisterExecutorSubscriber implements ExecutorTypeSubscriber<URIRegisterDTO> {
//......
@Override
public DataType getType() {
return DataType.URI; // URI data type
}
@Override
public void executor(final Collection<URIRegisterDTO> dataList) {
if (CollectionUtils.isEmpty(dataList)) {
return;
}
// Build a URI data type and register it through the registerURI method
findService(dataList).ifPresent(service -> {
Map<String, List<URIRegisterDTO>> listMap = buildData(dataList);
listMap.forEach(service::registerURI);
});
}
// Find Service by type
private Optional<ShenyuClientRegisterService> findService(final Collection<URIRegisterDTO> dataList) {
return dataList.stream().map(dto -> shenyuClientRegisterService.get(dto.getRpcType())).findFirst();
}
}
- ShenyuClientRegisterService#register()
ShenyuClientRegisterService is a registration method interface. It has multiple implementation classes:
- AbstractContextPathRegisterService: an abstract class that handles some public logic;
- AbstractShenyuClientRegisterServiceImpl: abstract class that handles some public logic;
- ShenyuClientRegisterDivideServiceImpl: the divide class handles http registration types;
- ShenyuClientRegisterDubboServiceImpl: dubbo class, which handles dubbo registration types;
- ShenyuClientRegisterGrpcServiceImpl: gRPC class, which handles gRPC registration types;
- ShenyuClientRegisterMotanServiceImpl: Motan class, which handles Motan registration types;
- ShenyuClientRegisterSofaServiceImpl: Sofa class, which handles the Sofa registration type;
- ShenyuClientRegisterSpringCloudServiceImpl: spring cloud class, which handles spring cloud registration types;
- ShenyuClientRegisterTarsServiceImpl: Tars class, which handles Tars registration types;
As can be seen from the above, each microservice has a corresponding registration implementation class. The source code analysis of this paper is provided officially shenyu-examples-http For example, it belongs to http registration type, so the registration implementation class of metadata and URI data is ShenyuClientRegisterDivideServiceImpl:
- register(): register metadata
public String register(final MetaDataRegisterDTO dto) {
// 1. Register selector information
String selectorHandler = selectorHandler(dto);
String selectorId = selectorService.registerDefault(dto, PluginNameAdapter.rpcTypeAdapter(rpcType()), selectorHandler);
// 2. Registration rule information
String ruleHandler = ruleHandler();
RuleDTO ruleDTO = buildRpcDefaultRuleDTO(selectorId, dto, ruleHandler);
ruleService.registerDefault(ruleDTO);
// 3. Registration metadata information
registerMetadata(dto);
// 4. Register contextPath
String contextPath = dto.getContextPath();
if (StringUtils.isNotEmpty(contextPath)) {
registerContextPath(dto);
}
return ShenyuResultMessage.SUCCESS;
}
The whole registration logic can be divided into four steps:
- 1. Register selector information
- 2. Registration rule information
- 3. Registration metadata information
- 4. Register contextPath
On the admin side, you need to build selectors, rules, metadata and ContextPath through the metadata information of the client. The specific registration process and details are related to rpc types. We will not continue to track down. For the logical analysis of the registry, tracking here is enough.
The source code analysis of the service side metadata registration process is completed, and the flow chart is described as follows:
- registerURI(): register URI data
public String registerURI(final String selectorName, final List<URIRegisterDTO> uriList) {
if (CollectionUtils.isEmpty(uriList)) {
return "";
}
// Does the corresponding selector exist
SelectorDO selectorDO = selectorService.findByNameAndPluginName(selectorName, PluginNameAdapter.rpcTypeAdapter(rpcType()));
if (Objects.isNull(selectorDO)) {
return "";
}
// Handling handler information in selectors
String handler = buildHandle(uriList, selectorDO);
selectorDO.setHandle(handler);
SelectorData selectorData = selectorService.buildByName(selectorName, PluginNameAdapter.rpcTypeAdapter(rpcType()));
selectorData.setHandle(handler);
// Update records in the database
selectorService.updateSelective(selectorDO);
// Publish event
eventPublisher.publishEvent(new DataChangedEvent(ConfigGroupEnum.SELECTOR, DataEventTypeEnum.UPDATE, Collections.singletonList(selectorData)));
return ShenyuResultMessage.SUCCESS;
}
After getting the URI data, admin mainly updates the handler information in the selector, writes it to the database, and finally publishes the event notification gateway. The logic of the notification gateway is completed by the data synchronization operation, which has been analyzed in the previous article and will not be repeated.
The source code analysis of the server URI registration process is completed, which is described as follows:
So far, the server registration process has been analyzed. It mainly receives the registration information of the client through the external interface, writes it to the Disruptor queue, consumes data from it, and updates the selector, rule, metadata and selector handler of admin according to the received metadata and URI data.
4. Summary
This paper mainly analyzes the source code of http registration module in Apache ShenYu gateway. The main knowledge points involved are summarized as follows:
- The registration center is to register the client information with admin to facilitate traffic filtering;
- http registration is to register the client metadata information and URI information to admin;
- The access of http service is identified by annotation @ shenyusprinmvcclient;
- The registration information is built mainly through Spring's post processor BeanPostProcessor and application listener;
- The loading of registration type is completed through SPI;
- The purpose of introducing the Disruptor queue is to decouple data from operations and buffer data.
- The implementation of the registry adopts interface oriented programming, using template method, singleton, observer and other design patterns.