Business scenario
The scenario in the project needs to get the values of a batch of key s, because the get operation of redis (not just the get command) is blocked. If the cyclic value is taken, even in the intranet, it takes a lot of time. So I thought of the pipeline command of redis.
pipeline introduction
- Non pipeline: a request from the client and a response from the redis server, during which the client is blocked
- Pipeline: the pipeline command of redis allows the client to send multiple requests to the server in turn (redis clients, such as jedisCluster and lettuce, all implement the encapsulation of pipeline). In the process, there is no need to wait for the reply of the request, and then read the results at the end.
Stand alone version
The stand-alone version is relatively simple and can be obtained in batches
//Replace with a real redis instance
Jedis jedis = new Jedis();
//Get pipeline
Pipeline p = jedis.pipelined();
for (int i = 0; i < 10000; i++) {
p.get("key_" + i);
}
//Get results
List<Object> results = p.syncAndReturnAll();
Batch insert
String key = "key";
Jedis jedis = new Jedis();
Pipeline p = jedis.pipelined();
List<String> cacheData = .... //List of data to insert
for(String data: cacheData ){
p.hset(key, data);
}
p.sync();
jedis.close();
Cluster version
In fact, the problem is that Redis used in the project is a cluster, and the class used during initialization is JedisCluster instead of Jedis. After checking the JedisCluster documentation, I found that there is no pipelined() method to obtain the Pipeline object like Jedis. Solution:
The Redis cluster specification says that the key space of the Redis cluster is divided into 16384 slots, and the maximum number of nodes in the cluster is also 16384. Each master node is responsible for processing part of the 16384 hash slots. When we say that a cluster is in a "stable" state, it means that the cluster is not performing reconfiguration, and each hash slot is processed by only one node. Therefore, you can know the slot number corresponding to the key according to the key to be inserted, and then find the corresponding Jedis from the cluster through the slot number. The specific implementation is as follows:
//When the jedis cluster is initialized, the code of how to obtain the HostAndPort collection is not written Set<HostAndPort> nodes = ..... JedisCluster jedisCluster = new JedisCluster(nodes); Map<String, JedisPool> nodeMap = jedisCluster.getClusterNodes(); String anyHost = nodeMap.keySet().iterator().next(); //The getSlotHostMap method is shown below TreeMap<Long, String> slotHostMap = getSlotHostMap(anyHost);
private static TreeMap<Long, String> getSlotHostMap(String anyHostAndPortStr) {
TreeMap<Long, String> tree = new TreeMap<Long, String>();
String parts[] = anyHostAndPortStr.split(":");
HostAndPort anyHostAndPort = new HostAndPort(parts[0], Integer.parseInt(parts[1]));
try{
Jedis jedis = new Jedis(anyHostAndPort.getHost(), anyHostAndPort.getPort());
List<Object> list = jedis.clusterSlots();
for (Object object : list) {
List<Object> list1 = (List<Object>) object;
List<Object> master = (List<Object>) list1.get(2);
String hostAndPort = new String((byte[]) master.get(0)) + ":" + master.get(1);
tree.put((Long) list1.get(0), hostAndPort);
tree.put((Long) list1.get(1), hostAndPort);
}
jedis.close();
}catch(Exception e){
}
return tree;
}
The above steps can be completed during initialization. You do not need to call every time. Both nodeMap and slotHostMap are defined as static variables.
//Get slot number int slot = JedisClusterCRC16.getSlot(key); //Get the corresponding Jedis object Map.Entry<Long, String> entry = slotHostMap.lowerEntry(Long.valueOf(slot)); Jedis jedis = nodeMap.get(entry.getValue()).getResource();
It is suggested that the above operations can be encapsulated into a static method. For example, it is named public static Jedis getJedisByKey(String key). This means that in the cluster, the Jedis object corresponding to the key is obtained through the key. In this way, the above jedis.pipelined(); Then you can insert in batches. The following is a relatively complete package
import redis.clients.jedis.*;
import redis.clients.jedis.exceptions.JedisMovedDataException;
import redis.clients.jedis.exceptions.JedisRedirectionException;
import redis.clients.util.JedisClusterCRC16;
import redis.clients.util.SafeEncoder;
import java.io.Closeable;
import java.lang.reflect.Field;
import java.util.*;
import java.util.function.BiConsumer;
public class JedisClusterPipeline extends PipelineBase implements Closeable {
/**
* Used to get jedisclusterinfo cache
*/
private JedisSlotBasedConnectionHandler connectionHandler;
/**
* Get the connection according to the hash value
*/
private JedisClusterInfoCache clusterInfoCache;
/**
* You can also inherit JedisCluster and JedisSlotBasedConnectionHandler to provide access interfaces
* JedisCluster Inherited from BinaryJedisCluster
* In BinaryJedisCluster, the connectionHandler property is protected, so reflection is required
*
*
* The jedisclusterinfo cache attribute is in the JedisClusterConnectionHandler, but this class is an abstract class,
* But it has an implementation class, JedisSlotBasedConnectionHandler
*/
private static final Field FIELD_CONNECTION_HANDLER;
private static final Field FIELD_CACHE;
static {
FIELD_CONNECTION_HANDLER = getField(BinaryJedisCluster.class, "connectionHandler");
FIELD_CACHE = getField(JedisClusterConnectionHandler.class, "cache");
}
/**
* Store the Client corresponding to each command in order
*/
private Queue<Client> clients = new LinkedList<>();
/**
* For caching connections
* jedis cache used in pipeline process once
*/
private Map<JedisPool, Jedis> jedisMap = new HashMap<>();
/**
* Is there data in the cache
*/
private boolean hasDataInBuf = false;
/**
* Generate the corresponding JedisClusterPipeline according to the jedisCluster instance
* If you get the pipeline in this way to operate, you must call close() to close the pipeline
* Call the pipelineXX method in this class without close(), but it is recommended to call close() in finally
* @param
* @return
*/
public static JedisClusterPipeline pipelined(JedisCluster jedisCluster) {
JedisClusterPipeline pipeline = new JedisClusterPipeline();
pipeline.setJedisCluster(jedisCluster);
return pipeline;
}
public JedisClusterPipeline() {
}
public void setJedisCluster(JedisCluster jedis) {
connectionHandler = getValue(jedis, FIELD_CONNECTION_HANDLER);
clusterInfoCache = getValue(connectionHandler, FIELD_CACHE);
}
/**
* Refresh the cluster information, and call when the cluster information changes
* @param
* @return
*/
public void refreshCluster() {
connectionHandler.renewSlotCache();
}
/**
* Read all data synchronously. Compared with syncAndReturnAll(), sync() just does not deserialize the data
*/
public void sync() {
innerSync(null);
}
/**
* Reads all data synchronously and returns a list in command order
*
* @return Returns all data in the order of the command
*/
public List<Object> syncAndReturnAll() {
List<Object> responseList = new ArrayList<>();
innerSync(responseList);
return responseList;
}
@Override
public void close() {
clean();
clients.clear();
for (Jedis jedis : jedisMap.values()) {
if (hasDataInBuf) {
flushCachedData(jedis);
}
jedis.close();
}
jedisMap.clear();
hasDataInBuf = false;
}
private void flushCachedData(Jedis jedis) {
try {
jedis.getClient().getAll();
} catch (RuntimeException ex) {
}
}
@Override
protected Client getClient(String key) {
byte[] bKey = SafeEncoder.encode(key);
return getClient(bKey);
}
@Override
protected Client getClient(byte[] key) {
Jedis jedis = getJedis(JedisClusterCRC16.getSlot(key));
Client client = jedis.getClient();
clients.add(client);
return client;
}
private Jedis getJedis(int slot) {
JedisPool pool = clusterInfoCache.getSlotPool(slot);
// Get Jedis from cache according to pool
Jedis jedis = jedisMap.get(pool);
if (null == jedis) {
jedis = pool.getResource();
jedisMap.put(pool, jedis);
}
hasDataInBuf = true;
return jedis;
}
public static void pipelineSetEx(String[] keys, String[] values, int[] exps,JedisCluster jedisCluster) {
operate(new Command() {
@Override
public List execute() {
JedisClusterPipeline p = pipelined(jedisCluster);
for (int i = 0, len = keys.length; i < len; i++) {
p.setex(keys[i], exps[i], values[i]);
}
return p.syncAndReturnAll();
}
});
}
public static List<Map<String, String>> pipelineHgetAll(String[] keys,JedisCluster jedisCluster) {
return operate(new Command() {
@Override
public List execute() {
JedisClusterPipeline p = pipelined(jedisCluster);
for (int i = 0, len = keys.length; i < len; i++) {
p.hgetAll(keys[i]);
}
return p.syncAndReturnAll();
}
});
}
public static List<Boolean> pipelineSismember(String[] keys, String members,JedisCluster jedisCluster) {
return operate(new Command() {
@Override
public List execute() {
JedisClusterPipeline p = pipelined(jedisCluster);
for (int i = 0, len = keys.length; i < len; i++) {
p.sismember(keys[i], members);
}
return p.syncAndReturnAll();
}
});
}
public static <O> List pipeline(BiConsumer<O, JedisClusterPipeline> function, O obj,JedisCluster jedisCluster) {
return operate(new Command() {
@Override
public List execute() {
JedisClusterPipeline jcp = JedisClusterPipeline.pipelined(jedisCluster);
function.accept(obj, jcp);
return jcp.syncAndReturnAll();
}
});
}
private void innerSync(List<Object> formatted) {
HashSet<Client> clientSet = new HashSet<>();
try {
for (Client client : clients) {
// In fact, it is not necessary to parse the result data when calling sync(). However, if the get method is not called, the application does not know that an error such as jedismoveddateexception occurs, so it needs to call get() to trigger an error.
// In fact, if the data attribute of the Response can be obtained directly, the time for parsing the data can be saved. However, it does not provide a corresponding method. To obtain the data attribute, you have to use reflection instead of reflection, so that's it
Object data = generateResponse(client.getOne()).get();
if (null != formatted) {
formatted.add(data);
}
// The same size means that all client s have been added, so there is no need to call the add method
if (clientSet.size() != jedisMap.size()) {
clientSet.add(client);
}
}
} catch (JedisRedirectionException jre) {
if (jre instanceof JedisMovedDataException) {
// if MOVED redirection occurred, rebuilds cluster's slot cache,
// recommended by Redis cluster specification
refreshCluster();
}
throw jre;
} finally {
if (clientSet.size() != jedisMap.size()) {
// All client s that have not been executed should be flushed to prevent the following commands from being polluted after being put back into the connection pool
for (Jedis jedis : jedisMap.values()) {
if (clientSet.contains(jedis.getClient())) {
continue;
}
flushCachedData(jedis);
}
}
hasDataInBuf = false;
close();
}
}
private static Field getField(Class<?> cls, String fieldName) {
try {
Field field = cls.getDeclaredField(fieldName);
field.setAccessible(true);
return field;
} catch (NoSuchFieldException | SecurityException e) {
throw new RuntimeException("cannot find or access field '" + fieldName + "' from " + cls.getName(), e);
}
}
@SuppressWarnings({"unchecked" })
private static <T> T getValue(Object obj, Field field) {
try {
return (T)field.get(obj);
} catch (IllegalArgumentException | IllegalAccessException e) {
throw new RuntimeException(e);
}
}
private static <T> T operate(Command command) {
try {
return command.execute();
} catch (Exception e) {
throw new RuntimeException(e);
}
}
interface Command {
/**
* Specific execution command
*
* @param <T>
* @return
*/
<T> T execute();
}
}
Use examples
public Object testPipelineOperate() {
// String[] keys = {"dylan1","dylan2"};
// String[] values = {"dylan1-v1","dylan2-v2"};
// int[] exps = {100,200};
// JedisClusterPipeline.pipelineSetEx(keys, values, exps, jedisCluster);
long start = System.currentTimeMillis();
List<String> keyList = new ArrayList<>();
for (int i = 0; i < 1000; i++) {
keyList.add(i + "");
}
// List<String> pipeline = JedisClusterPipeline.pipeline(this::getValue, keyList, jedisCluster);
// List<String> pipeline = JedisClusterPipeline.pipeline(this::getHashValue, keyList, jedisCluster);
String[] keys = {"dylan-test1", "dylan-test2"};
List<Map<String, String>> all = JedisClusterPipeline.pipelineHgetAll(keys, jedisCluster);
long end = System.currentTimeMillis();
System.out.println("testPipelineOperate cost:" + (end-start));
return Response.success(all);
}