Typical usage scenarios of zookeeper

Posted by wwedude on Wed, 10 Nov 2021 13:15:27 +0100

Zookeeper unfair lock / fair lock / shared lock

Zookeeper distributed lock practice

mutex

Unfair lock


In the above implementation, when the concurrency problem is serious, the performance will decline seriously. The main reason is that all connections are listening to the same node. When the server detects a deletion event, it should notify all connections. All connections receive events at the same time and compete concurrently again. This is herding. This locking method is the specific implementation of unfair locking: how to avoid it? Let's look at the following method.

Fair lock


As mentioned above, with the help of temporary sequential nodes, concurrent competitive locks of multiple nodes at the same time can be avoided, and the pressure on the server can be relieved. In this way, all lock requests are queued and locked, which is the specific implementation of fair lock

Read write lock

The previous two locking methods have a common characteristic, that is, they are mutually exclusive locks. Only one request can be occupied at a time. If a large number of concurrent requests come up, the performance will decline sharply. All requests have to be locked. Is it true that all requests need to be locked? The answer is No. for example, if the data has not been modified, it does not need to be locked, but if the request to read the data has not been read, what should we do if a write request comes at this time? Someone is already reading data. You can't write data at this time, otherwise the data will be incorrect. The write request cannot be executed until all the previous read locks are released. Therefore, it is necessary to add an ID (read lock) to the read request to let the write request know that the data cannot be modified at this time. Otherwise the data will be inconsistent. If someone is already writing data, another request to write data is not allowed, which will also lead to data inconsistency. Therefore, all write requests need to be added with a write lock to avoid writing shared data at the same time.

Problems caused by read-write locks

  • Inconsistent read-write concurrency

  • Double write inconsistency

Principle of read-write lock

case

Case scenario analysis

Case realization

Configuration class

package com.tuling.distributedlock.zkcfg;

import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.retry.ExponentialBackoffRetry;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;


@Configuration
public class CuratorCfg {
    @Bean(initMethod = "start")
    public CuratorFramework curatorFramework(){
        RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3);
        CuratorFramework client = CuratorFrameworkFactory.newClient("192.168.1.104:2181", retryPolicy);
        return client;
    }
}

Controller class

package com.tuling.distributedlock;

import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.recipes.locks.InterProcessMutex;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.web.bind.annotation.PostMapping;
import org.springframework.web.bind.annotation.RestController;

@RestController
public class TestController {


    @Autowired
    private OrderService orderService;

    @Value("${server.port}")
    private String port;
    @Autowired
    CuratorFramework curatorFramework;

    @PostMapping("/stock/deduct")
    public Object reduceStock(Integer id) throws Exception {
        try {
            orderService.reduceStock(id);

        } catch (Exception e) {
            if (e instanceof RuntimeException) {
                throw e;
            }
        } 
        return "ok:" + port;
    }
}

Service class

package com.tuling.distributedlock;

import com.tuling.distributedlock.entity.Order;
import com.tuling.distributedlock.entity.Product;
import com.tuling.distributedlock.mapper.OrderMapper;
import com.tuling.distributedlock.mapper.ProductMapper;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Service;
import org.springframework.transaction.annotation.Transactional;

import java.util.UUID;
import java.util.concurrent.TimeUnit;


@Service
public class OrderService {

    @Autowired
    private ProductMapper productMapper;

    @Autowired
    private OrderMapper orderMapper;



    @Transactional
     public void reduceStock(Integer id){
        // 1. Obtain inventory
        Product product = productMapper.getProduct(id);
        // Simulate time-consuming business processing
        sleep( 500); // Other business processing

        if (product.getStock() <=0 ) {
            throw new RuntimeException("out of stock");
        }
        // 2. Inventory reduction
        int i = productMapper.deductStock(id);
        if (i==1){
            Order order = new Order();
            order.setUserId(UUID.randomUUID().toString());
            order.setPid(id);
            orderMapper.insert(order);
        }else{
            throw new RuntimeException("deduct stock fail, retry.");
        }

    }

    /**
     * Simulate time-consuming business processing
     * @param wait
     */
   public void sleep(long  wait){
       try {
           TimeUnit.MILLISECONDS.sleep( wait );
       } catch (InterruptedException e) {
           e.printStackTrace();
       }
   }

}

Global exception handling class

package com.tuling.distributedlock;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.http.HttpStatus;
import org.springframework.web.bind.annotation.*;

import java.util.HashMap;
import java.util.Map;


@ControllerAdvice
public class ExceptionHandlerController {


    @ExceptionHandler
    @ResponseStatus(value = HttpStatus.BAD_REQUEST)
    @ResponseBody
    public Object exceptionHandler(RuntimeException e){

        Map<String,Object> result=new HashMap<>(  );
        result.put( "status","error" );
        result.put( "message",e.getMessage() );
        return result;
    }


}

package com.tuling.distributedlock.mapper;

import com.tuling.distributedlock.entity.Product;
import org.apache.ibatis.annotations.Param;
import org.apache.ibatis.annotations.Select;
import org.apache.ibatis.annotations.Update;


public interface ProductMapper {

    @Select(" select * from product where id=#{id}  ")
    Product getProduct(@Param("id") Integer id);

    @Update(" update product set stock=stock-1    where id=#{id}  ")
    int deductStock(@Param("id") Integer id);
}

package com.tuling.distributedlock.mapper;

import com.tuling.distributedlock.entity.Order;
import org.apache.ibatis.annotations.Insert;
import org.apache.ibatis.annotations.Options;

public interface OrderMapper {

    @Options(useGeneratedKeys = true,keyColumn = "id",keyProperty = "id")
    @Insert(" insert into `order`(user_id,pid) values(#{userId},#{pid}) ")
    int insert(Order order);
}


package com.tuling.distributedlock.entity;


public class Product {
    private Integer id;
    private String productName;
    private Integer stock;
    private Integer version;

    public Integer getId() {
        return id;
    }

    public void setId(Integer id) {
        this.id = id;
    }

    public String getProductName() {
        return productName;
    }

    public void setProductName(String productName) {
        this.productName = productName;
    }

    public Integer getStock() {
        return stock;
    }

    public void setStock(Integer stock) {
        this.stock = stock;
    }

    public Integer getVersion() {
        return version;
    }

    public void setVersion(Integer version) {
        this.version = version;
    }
}


package com.tuling.distributedlock.entity;


public class Order {
  private Integer id;
  private Integer pid;
  private String userId;

    public Integer getId() {
        return id;
    }

    public void setId(Integer id) {
        this.id = id;
    }

    public Integer getPid() {
        return pid;
    }

    public void setPid(Integer pid) {
        this.pid = pid;
    }

    public String getUserId() {
        return userId;
    }

    public void setUserId(String userId) {
        this.userId = userId;
    }
}


package com.tuling;

import org.mybatis.spring.annotation.MapperScan;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;


@MapperScan("com.tuling.distributedlock.mapper")
@SpringBootApplication
public class DistributedlockApplication {

	public static void main(String[] args) {
		SpringApplication.run(DistributedlockApplication.class, args);
	}

}

server.port=8080
spring.datasource.url=jdbc:mysql://localhost:3306/pro?serverTimezone=UTC&useSSL=false&useUnicode=true&characterEncoding=UTF-8&allowPublicKeyRetrieval=true
spring.datasource.username=root
spring.datasource.password=1234
spring.datasource.driver-class-name=com.mysql.jdbc.Driver
mybatis.configuration.map-underscore-to-camel-case=true

SET FOREIGN_KEY_CHECKS=0;

-- ----------------------------
-- Table structure for order
-- ----------------------------
DROP TABLE IF EXISTS `order`;
CREATE TABLE `order` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `pid` int(11) DEFAULT NULL,
  `user_id` varchar(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=1513 DEFAULT CHARSET=utf8mb4;


-- ----------------------------
-- Table structure for product
-- ----------------------------
DROP TABLE IF EXISTS `product`;
CREATE TABLE `product` (
  `id` int(11) NOT NULL AUTO_INCREMENT,
  `product_name` varchar(255) DEFAULT NULL,
  `stock` int(11) DEFAULT NULL,
  `version` int(11) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=utf8mb4;


INSERT INTO `product` VALUES ('1', 'New year's Day gift bag', '0', '0');

Case demonstration

Database table initialization


jmeter installation (stress test tool, simulated multithreading test)








nginx installation (load balancing, configured so that it can poll and call cluster instances)



Start the project and start the pressure test






At this time, it is obvious that 10 threads rush to buy 5 inventory goods. There are - 5 items left in the inventory, and there are 10 orders. There is a concurrency problem here

Add code related to distributed locks
<dependency>
	<groupId>mysql</groupId>
	<artifactId>mysql-connector-java</artifactId>
	<version>5.1.34</version>
	<scope>runtime</scope>
</dependency>
<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
	<groupId>org.mybatis.spring.boot</groupId>
	<artifactId>mybatis-spring-boot-starter</artifactId>
	<version>2.1.1</version>
</dependency>
<dependency>
	<groupId>org.springframework.boot</groupId>
	<artifactId>spring-boot-starter-test</artifactId>
	<scope>test</scope>
	<exclusions>
		<exclusion>
			<groupId>org.junit.vintage</groupId>
			<artifactId>junit-vintage-engine</artifactId>
		</exclusion>
	</exclusions>
</dependency>

 <!-- https://mvnrepository.com/artifact/org.apache.curator/curator-recipes -->
 <dependency>
	 <groupId>org.apache.curator</groupId>
	 <artifactId>curator-recipes</artifactId>
	 <version>5.0.0</version>
	 <exclusions>
		 <exclusion>
			 <groupId>org.apache.zookeeper</groupId>
			 <artifactId>zookeeper</artifactId>
		 </exclusion>
	 </exclusions>
 </dependency>
 <dependency>
	 <groupId>org.apache.zookeeper</groupId>
	 <artifactId>zookeeper</artifactId>
	 <version>3.5.8</version>
 </dependency>

@RestController
public class TestController {


    @Autowired
    private OrderService orderService;

    @Value("${server.port}")
    private String port;



    @Autowired
    CuratorFramework curatorFramework;

    @PostMapping("/stock/deduct")
    public Object reduceStock(Integer id) throws Exception {

        InterProcessMutex interProcessMutex = new InterProcessMutex(curatorFramework, "/product_" + id);

        try {
            // ...
            interProcessMutex.acquire();
            orderService.reduceStock(id);

        } catch (Exception e) {
            if (e instanceof RuntimeException) {
                throw e;
            }
        } finally {
            interProcessMutex.release();
        }
        return "ok:" + port;
    }


}




Analysis of zookeeper distributed lock source code





Application of Leader election in distributed scenario

Case demonstration

<dependency>
    <groupId>org.apache.curator</groupId>
    <artifactId>curator-recipes</artifactId>
    <version>5.0.0</version>
    <exclusions>
        <exclusion>
            <groupId>org.apache.zookeeper</groupId>
            <artifactId>zookeeper</artifactId>
        </exclusion>
    </exclusions>
</dependency>
 <dependency>
    <groupId>org.apache.zookeeper</groupId>
    <artifactId>zookeeper</artifactId>
    <version>3.5.8</version>
</dependency>

package zookeeper.leaderselector;

import org.apache.curator.RetryPolicy;
import org.apache.curator.framework.CuratorFramework;
import org.apache.curator.framework.CuratorFrameworkFactory;
import org.apache.curator.framework.recipes.leader.LeaderSelector;
import org.apache.curator.framework.recipes.leader.LeaderSelectorListener;
import org.apache.curator.framework.recipes.leader.LeaderSelectorListenerAdapter;
import org.apache.curator.retry.ExponentialBackoffRetry;

import java.util.concurrent.CountDownLatch;
import java.util.concurrent.TimeUnit;

public class LeaderSelectorDemo {
    private static  final  String CONNECT_STR="192.168.1.104:2181";
    private static RetryPolicy retryPolicy=new ExponentialBackoffRetry( 5*1000, 10 );
    private  static  CuratorFramework curatorFramework;
     private static CountDownLatch countDownLatch = new CountDownLatch(1);
     
    public static void main(String[] args) throws InterruptedException {
        String appName = System.getProperty("appName");

        CuratorFramework curatorFramework = CuratorFrameworkFactory.newClient(CONNECT_STR, retryPolicy);
        LeaderSelectorDemo.curatorFramework = curatorFramework;
        curatorFramework.start();

        LeaderSelectorListener listener = new LeaderSelectorListenerAdapter()
        {
            public void takeLeadership(CuratorFramework client) throws Exception
            {

                System.out.println(" I' m leader now . i'm , "+appName);

                TimeUnit.SECONDS.sleep(15);

            }
        };

        LeaderSelector selector = new LeaderSelector(curatorFramework, "/cachePreHeat_leader", listener);
        selector.autoRequeue();  // not required, but this is behavior that you will probably expect
        selector.start();
        countDownLatch.await();

    }
}



Source code introduction




The underlying source code also uses the principle of distributed lock

Spring Cloud Zookeeper registration center actual combat

Registry scenario analysis


What is a registration center that allows many services to register in zookeeper and write some of their service information, such as IP, port and more specific service information, to the zookeeper node, so that services in need can be directly obtained from zookeeper. How? At this time, we can define a unified name, such as user service. When all user services are started, a child node (temporary node) is created under the user service node. This child node should be unique, representing the unique identification of each service instance, For those that depend on user services, for example, order service can obtain all user service child nodes and all child node information (IP, port, etc.) through the parent node user service. After obtaining the data of the child nodes, order service can cache them, and then realize load balancing of a client, At the same time, you can also listen to the user service directory, so that if a new node joins or exits, the order service can receive a notification, so that the order service can re obtain all child nodes and update the data.

The child node of this user service is of type temporary node. The life cycle of the temporary node in Zookeeper is bound to the SESSION. If the SESSION times out, the corresponding node will be deleted. When deleted, Zookeeper will notify the client listening to the parent node of the node, so that the corresponding client can refresh the local cache again. When a new service is added, the corresponding client will also be notified to refresh the local cache. To achieve this goal, the client needs to register repeatedly to listen to the parent node. In this way, the automatic registration and automatic exit of the service are realized.

actual combat

Spring Cloud ecology also provides the implementation of Zookeeper registry. This project is called Spring Cloud Zookeeper
User center: User Service
Product center: Product Service
The user invokes the product service and realizes the load balancing of the client. The product service automatically joins the cluster and automatically exits the service.

Create user service

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.3.5.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>
	<groupId>com.example</groupId>
	<artifactId>user-center</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<name>user-center</name>
	<description>Demo project for Spring Boot</description>

	<properties>
		<java.version>1.8</java.version>
		<spring-cloud.version>Hoxton.SR8</spring-cloud.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-zookeeper-discovery</artifactId>
		</dependency>
		<!--<dependency>-->
			<!--<groupId>org.springframework.cloud</groupId>-->
			<!--<artifactId>spring-cloud-starter-openfeign</artifactId>-->
		<!--</dependency>-->



		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
			<optional>true</optional>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
			<exclusions>
				<exclusion>
					<groupId>org.junit.vintage</groupId>
					<artifactId>junit-vintage-engine</artifactId>
				</exclusion>
			</exclusions>
		</dependency>
	</dependencies>

	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>org.springframework.cloud</groupId>
				<artifactId>spring-cloud-dependencies</artifactId>
				<version>${spring-cloud.version}</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>

</project>

configuration file

spring.application.name=user-center
#zookeeper connection address
spring.cloud.zookeeper.connect-string=192.168.1.104:2181

The startup class is configured with load balancer @ LoadBalanced, which can poll and call multiple instances

package com.example.usercenter;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cloud.client.loadbalancer.LoadBalanced;
import org.springframework.context.annotation.Bean;
import org.springframework.web.client.RestTemplate;

@SpringBootApplication
public class UserCenterApplication {

    public static void main(String[] args) {
        SpringApplication.run(UserCenterApplication.class, args);
    }


    @Bean
    @LoadBalanced
    public RestTemplate restTemplate() {
        RestTemplate restTemplate = new RestTemplate();
         return restTemplate;
    }
}

Controller class

package com.example.usercenter;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.cloud.client.ServiceInstance;
import org.springframework.cloud.client.loadbalancer.LoadBalancerClient;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;
import org.springframework.web.client.RestTemplate;

@RestController
public class TestController {

    @Autowired
    private RestTemplate restTemplate;

    @Autowired
    private LoadBalancerClient loadBalancerClient;

    @GetMapping("/test")
    public String test() {
        return this.restTemplate.getForObject("http://product-center/getInfo", String.class);
    }

    @GetMapping("/lb")
    public String getLb(){
        ServiceInstance choose = loadBalancerClient.choose("product-center");
        String serviceId = choose.getServiceId();
        int port = choose.getPort();
        return serviceId + " : "+port;
    }
}

Create product service

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
	xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
	<modelVersion>4.0.0</modelVersion>
	<parent>
		<groupId>org.springframework.boot</groupId>
		<artifactId>spring-boot-starter-parent</artifactId>
		<version>2.3.5.RELEASE</version>
		<relativePath/> <!-- lookup parent from repository -->
	</parent>
	<groupId>com.example</groupId>
	<artifactId>product-center</artifactId>
	<version>0.0.1-SNAPSHOT</version>
	<name>product-center</name>
	<description>Demo project for Spring Boot</description>

	<properties>
		<java.version>1.8</java.version>
		<spring-cloud.version>Hoxton.SR8</spring-cloud.version>
	</properties>

	<dependencies>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-web</artifactId>
		</dependency>
		<dependency>
			<groupId>org.springframework.cloud</groupId>
			<artifactId>spring-cloud-starter-zookeeper-discovery</artifactId>
		</dependency>

		<dependency>
			<groupId>org.projectlombok</groupId>
			<artifactId>lombok</artifactId>
			<optional>true</optional>
		</dependency>
		<dependency>
			<groupId>org.springframework.boot</groupId>
			<artifactId>spring-boot-starter-test</artifactId>
			<scope>test</scope>
			<exclusions>
				<exclusion>
					<groupId>org.junit.vintage</groupId>
					<artifactId>junit-vintage-engine</artifactId>
				</exclusion>
			</exclusions>
		</dependency>
	</dependencies>

	<dependencyManagement>
		<dependencies>
			<dependency>
				<groupId>org.springframework.cloud</groupId>
				<artifactId>spring-cloud-dependencies</artifactId>
				<version>${spring-cloud.version}</version>
				<type>pom</type>
				<scope>import</scope>
			</dependency>
		</dependencies>
	</dependencyManagement>

	<build>
		<plugins>
			<plugin>
				<groupId>org.springframework.boot</groupId>
				<artifactId>spring-boot-maven-plugin</artifactId>
			</plugin>
		</plugins>
	</build>

</project>


spring.application.name=product-center
#zookeeper connection address
spring.cloud.zookeeper.connect-string=192.168.1.104:2181
#Register this service with zookeeper
spring.cloud.zookeeper.discovery.register=true
spring.cloud.zookeeper.session-timeout=30000

The startup class is also the Controller class

package com.example.productcenter;

import org.springframework.beans.factory.annotation.Value;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RestController;

@SpringBootApplication
@RestController
public class ProductCenterApplication {

	@Value("${server.port}")
	private String port;

	@Value( "${spring.application.name}" )
	private String name;

	@GetMapping("/getInfo")
	public String getServerPortAndName(){

		return  this.name +" : "+ this.port;
	}
	public static void main(String[] args) {
		SpringApplication.run(ProductCenterApplication.class, args);
	}

}


package com.example.productcenter;

import lombok.extern.slf4j.Slf4j;
import org.springframework.cloud.client.discovery.event.HeartbeatEvent;
import org.springframework.cloud.zookeeper.discovery.ZookeeperServiceWatch;
import org.springframework.context.ApplicationListener;
import org.springframework.stereotype.Component;

@Component
@Slf4j
public class HeartbeatEventListener implements ApplicationListener<HeartbeatEvent> {


    @Override
    public void onApplicationEvent(HeartbeatEvent event) {

        Object value = event.getValue();
        ZookeeperServiceWatch source = (ZookeeperServiceWatch)event.getSource();

        log.info(" event:source: {} ,event:value{}",source.getCache().getCurrentChildren("/services"),value.toString());
    }
}


package com.example.productcenter;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.ApplicationArguments;
import org.springframework.boot.ApplicationRunner;
import org.springframework.cloud.zookeeper.serviceregistry.ServiceInstanceRegistration;
import org.springframework.cloud.zookeeper.serviceregistry.ZookeeperRegistration;
import org.springframework.cloud.zookeeper.serviceregistry.ZookeeperServiceRegistry;
import org.springframework.stereotype.Component;

@Component
public class ApplicationRunner1 implements ApplicationRunner{

    @Autowired
    private ZookeeperServiceRegistry serviceRegistry;

    @Override
    public void run(ApplicationArguments args) throws Exception {
        ZookeeperRegistration registration = ServiceInstanceRegistration.builder()
                .defaultUriSpec()
                .address("anyUrl")
                .port(10)
                .name("/a/b/c/d/anotherservice")
                .build();
        this.serviceRegistry.register(registration);
    }
}

Case demonstration





Topics: Java Back-end

Hot Topics

©2025 Programmer Think