Background requirements

technical analysis

It is obviously impractical to take all the local databases out into a collection and then traverse and send http requests to verify the data. How much caching does it take to store 10W records? And it takes up a lot of system resources.
The batch processing data is based on the production-consumer model. The production consumer mode maintains a queue, where one thread adds data and the other takes data. The processing speed can be controlled by controlling the number of threads.
Idea: Page batch data, put it into a blocking queue Linked Blocking Queue, open another thread to fetch data from the queue, cycle the above process until all data processing is completed (data processing to the last page), the queue is empty.

code implementation

• Talk is cheap, show me code

public class CheckRecordWithChannel {

    private static final Logger logger = LoggerFactory.getLogger(CheckRecordWithChannel.class);

    //500 pieces of data
    private volatile BlockingQueue<WechatTransInfo> orderQueue = new LinkedBlockingQueue<>(500);

    public static final int PAGE_NUM = 200;

    //Production Completion Mark
    private boolean produceFlag = false;
    //Consumption Completion Mark
    private boolean cosumerFlag = false;
    //Number of current pages
    private int currentPageNum = 0;

    private CheckOrderBusinessHandle businessHandle;

    /**
     * Construction method
     * @param businessHandle Special business methods to be implemented
     */
    public CheckRecordWithChannel (CheckOrderBusinessHandle businessHandle) {
        this.businessHandle = businessHandle;
    }

    /**
     * Main Method
     * @return
     * @throws InterruptedException
     */
    public boolean checkOrderBusiness() throws InterruptedException {
        //Create thread pools, and the number of producers and consumers can be more than a few
        ExecutorService checkOrderService = Executors.newCachedThreadPool();
        LocalOrderProducer producer = this.new LocalOrderProducer();
        OrderConsumer consumer = this.new OrderConsumer();
        checkOrderService.submit(producer);
        checkOrderService.submit(consumer);
        while (true) {
            Thread.sleep(2000L);
            if (produceFlag && cosumerFlag && orderQueue.isEmpty()) {
                List<Runnable> shutdownList = checkOrderService.shutdownNow();
                logger.warn("------{} Page processing is completed.{} Thread stops running-----", currentPageNum,shutdownList);
                return true;
            }
        }
    }

    /**
     * Paging query data service
     */
    public Page<WechatTransInfo> queryOrderWithPage(int pageNum) {
        //Facilitate future expansion
        return businessHandle.queryOrderWithPage(this.currentPageNum++, pageNum);
    }

    /**
     * Production method, blocking method, put method
     * @param orderList
     * @return Whether there is data or not, false means there is no data
     * @throws InterruptedException Thread interrupted
     */
    public boolean produceOrder() throws InterruptedException
    {
        //Data identification
        boolean hasData = true;
        Page<Info> queryOrderPage = queryOrderWithPage(PAGE_NUM);
        if (null == queryOrderPage) {
            return false;
        }
        //Final query results
        if (!queryOrderPage.hasNextPage()) {
            hasData = false;
        }
        //Cyclic insertion of 200 pieces of data, queue full blocked waiting
        for (Info order : queryOrderPage.getContent()) {
            orderQueue.put(order);
        }
        return hasData;
    }

    /**
     * Consumer approach
     */
    public boolean cosumerOrder(Info orderInfo) {
        //Facilitate future expansion
        return businessHandle.checkOrderWithOrg(orderInfo);
    }

    /**
     * Producer
     * <p> Query the local database order and put it into orderIdQueue, and flag=true at the end of the query
     * @author 
     */
    class LocalOrderProducer implements Runnable {
        @Override
        public void run() {
            try {
                // If the database is not queried, continue production
                while (!produceFlag) {
                    Thread.sleep(2000L);  //Slow down producers
                    //If there is no data in the database
                    if(!produceOrder()) {
                        Thread.sleep(5000L);
                        produceFlag = true;
                    }
                }
                logger.debug("----producer was done---");
            } catch (InterruptedException e) {
                logger.error("--- producer thread was interrupted--{}-", e);
            }
        }

    }

    /**
     * Consumer
     * <p> Non-blocking consumption
     * @author 
     */
    class OrderConsumer implements Runnable {
        @Override
        public void run() {
            try {
                // First Blocking Data Retrieval
                Thread.sleep(5000L);
                Info orderInfo = null;
                //If the producer is still in production or the queue is not empty, then enter the continued consumption processing.
                while (!produceFlag || null != (orderInfo = orderQueue.poll())) {
                    //If the queue is empty, wait for 2s producer production
                    logger.debug("---is get order data, ? {} ---", null != orderInfo);
                    if (null == orderInfo) {
                        Thread.sleep(2000L);
                    }
                    cosumerOrder(orderInfo);
                    //orderInfo = orderQueue.poll();
                }

                if (produceFlag && orderQueue.isEmpty()) {
                    cosumerFlag = true;
                }
            } catch (InterruptedException e) {
                logger.error("---thread was interrupted--{}-", e);
            }

        }

    }

}

public interface CheckOrderBusinessHandle {

    /**
     * Query Local Database Business
     * <p> The database is queried sequentially and iteratively.
     * pageNum Is the number of queries per query
     * @return Returns the result of the query
     */
    Page<Info> queryOrderWithPage(int currentPageNum, int pageNum);

    /**
     * Processing business logic, updating local database
     * @param order  information
     */
    boolean checkOrderWithOrg(Info orderInfo);
}

1. The CheckRecordWithChannel constructor initializes the handle to handle specific business for production and consumption.
2. The two Runnable implementation classes are production and consumption threads, in which production is blocked, if the queue is full and the database is not queried, then blocked, until the database is fully queried and put into the queue, the thread exits put; consumption uses non-blocking poll, when the production is finished and the queue is empty, it exits poll;
3. Page is a paging bean in Spring framework.
4. LinkedBlockingQueue uses volatile modification to prevent data inconsistency during thread operation.
5. CheckOrder Business Handle is an interface that needs to be implemented according to business.

• Run call
  Implementing Specific Business Interface

public void checkOrderWith() {

        try {
            new CheckRecordWithChannel (new CheckOrderBusinessHandle() {

                @Override
                public Page<Info> queryOrderWithPage(int currentPageNum,
                        int pageNum) {
//TODO
                }

                @Override
                public boolean checkOrderWithOrg(Info orderInfo) {
//TODO
                }
            }).checkOrderBusiness();
        } catch (InterruptedException e) {
            logger.error("----check thread was stoped, {}----", e);
        }
    }