1: Background
- Implement a multi-dimensional ranking list (a natural week is a cycle), considering score and time dimensions. When the scores are the same, the earlier you get this ranking, the higher the ranking
- You need to listen to the original data. There are three actions: received, read and passed. Statistics of various data indicators according to three actions
- The number of received, viewed and marked by the user in the current natural week
- Carry out multi condition filtering according to the three actions, and prepare copywriting tips under various conditions
2: Scheme design
- For the definition of natural week, you can refer to the implementation of snowflake algorithm. By designing A fixed and immutable benchmark start date A, A date B is transformed into the number of weeks X from benchmark date A as the number of cycles
- For the implementation of leaderboard, we can use ZSet of Redis. key: fixed ID + fixed base date A + weeks from fixed base date A x value: user ID score: you can refer to the implementation of snowflake algorithm
- Because score has two dimensions: score and time, 64 bit long is used for data integration
- score 64 bit: the first bit can be 0 by default, which is used as reserved bit
- score 64 bits: the score can occupy 23 bits, representing the maximum score: 8388608 (2 ^ 23)
- score 64 bits: the timestamp (in milliseconds) from the current time to base date C can occupy 40 bits, which means it can last 34 years (2 ^ 40). Because the ranking is in reverse order, the benchmark date C must be the time after 34 this year. In this way, when calculating the timestamp difference Y, the larger the difference Y, the higher the ranking
- Such a score is spliced: 0 (identification bit) + 00000 00000 (real score bit) + 00000 00000 00000 00000 00000 00000 (timestamp difference C). Because the real score weight is more important than the time stamp, the real score is at the top
- For score assignment, optimistic lock + ZADD + LUA can be considered to avoid coverage update, resulting in incorrect score
- For monitoring raw data, the observer mode + thread isolation implementation can be considered. Based on the opening and closing principle, high cohesion and low coupling make the business more bright
- Multiple conditions are filtered for the data sources of the three actions to obtain the personalized copy belonging to each user, which can be realized by considering the responsibility chain. Based on the opening and closing principle, each filter condition has an implementation class. When conditions are added, reduced or changed, you can flexibly change only the current filter implementation class, which can achieve the lowest impact, high reuse and low coupling.
3: Concrete implementation
Implementation of redis Zset Score:
Preparation of basic score format:
/** * Maximum score of scoring position 8388608 */ private static final int SCORE = 23; /** * Timestamp: 34 years */ private static final int TIMESTAMP = 40; /** * Maximum score occupancy */ private static final long SCORE_MAX_SIZE = ~(-1L << SCORE); /** * Maximum timestamp occupancy */ private static final long TIME_STAMP_MAX_SIZE = ~(-1L << TIMESTAMP);
/**
* Get real score
* @param redisScore redis Store score
* @return
*/
public static BigDecimal getRealScore(Long redisScore) {
if (redisScore == null) {
return BigDecimal.ZERO;
}
long score = getRedisRealScore(redisScore);
return new BigDecimal(score).divide(BigDecimal.TEN, 2, BigDecimal.ROUND_HALF_UP);
}/**
* Obtain redis real score (expanded by 10 times)
* @param redisScore redis Store score
* @return
*/
public static long getRedisRealScore(Long redisScore) {
if (redisScore == null) {
return 0;
}
return redisScore >> TIMESTAMP & SCORE_MAX_SIZE;
}/**
* Calculate timestamp
* @param redisScore redis Store score
* @return
*/
public static long genTimeStamp(Long redisScore) {
if (redisScore == null) {
return 0;
}
return getFixedEndTimeStamp() - (redisScore & TIME_STAMP_MAX_SIZE);
}/**
* Calculate the added value
* @param score score
* @param betweenMs Phase difference millisecond
*/
public static Number incScoreValue(long score, long betweenMs) {
return ((score & SCORE_MAX_SIZE) << TIMESTAMP) | (betweenMs & TIME_STAMP_MAX_SIZE);
}/**
* Obtain the fixed time (benchmark starting value, do not change it)
*
* @return
*/
public static DateTime getFixedStartTime() {
return DateUtil.parse("2020-09-07 00:00:00", DatePattern.NORM_DATETIME_PATTERN);
}/**
* Get fixed timestamp (benchmark end value, never change)
*
* @return
*/
public static long getFixedEndTimeStamp() {
return DateUtil.offset(getFixedStartTime(), DateField.YEAR, 34).getTime();
}redis call:
// current time
Date now = new Date();
// Seconds difference
long betweenMs = fixedEndTimeStamp - currentTime.getTime();
// Get the ranking key corresponding to the number of periods in the industry
String weekRankingKey = MessageFormat.format("WEEK_RANKING:{0}:{2}", getFixedStartTime().toString(DatePattern.PURE_DATE_PATTERN), getFixedPeriod(now));
incrScore(weekRankingKey, String.valueOf(bUid), rankingScore, betweenMs);/**
* Set multi-dimensional sealing value
*
* @param key zset key
* @param value zset value
* @param getScore Score obtained this time
* @param betweenMs Milliseconds from fixed time
* @return
*/
private boolean incrScore(String key, String value, long getScore, long betweenMs) {
Long oldScore = null;
long newScore;
long totalScore;
do {
Double zScore = redisClient.zscore(key, value);
if (zScore != null) {
oldScore = zScore.longValue();
long redisRealScore = getRedisRealScore(oldScore);
totalScore = redisRealScore + getScore;
} else {
totalScore = getScore;
}
// Generate new value
newScore = incScoreValue(totalScore, betweenMs).longValue();
} while (!compareAndSetScore(key, value, oldScore, newScore));
return true;
}private static String LUA_SCRIPT = "if ( (ARGV[2] == '' or ARGV[2] == nil) and ((not (redis.call('zscore', KEYS[1], ARGV[1]))) ) or redis.call('zscore', KEYS[1], ARGV[1]) == ARGV[2]) \n" +
" then \n" +
"redis.call('zadd',KEYS[1],ARGV[3],ARGV[1])\n" +
" redis.call('EXPIRE', KEYS[1], tonumber(ARGV[4]))\n" +
" return 1\n" +
" else\n" +
" return 0\n" +
" end";
}/** * 1 Months */ public static final int EXPIRE_ONE_MONTH = 60 * 60 * 24 * 30;
/**
* CAS Set score
*
* @param key
* @param value
* @param oldScore
* @param newScore
* @return
*/
private boolean compareAndSetScore(String key, String value, Long oldScore, long newScore) {
Long execute = 0L;
try {
execute = redisClient.execute(workCallback -> {
List<String> args = new ArrayList<>();
args.add(value);
args.add(Convert.toStr(oldScore, ""));
args.add(Convert.toStr(newScore, ""));
args.add(String.valueOf(EXPIRE_ONE_MONTH));
return (Long) workCallback.eval(LUA_SCRIPT, Lists.newArrayList(key), args);
});
} catch (Exception e) {
log.error("compareAndSetScore Exception", e);
}
return execute == 1L;
}Observer mode + thread isolation listening
You can use the Observer and Observable built-in in java
public class ActionObservable extends Observable {
private ActionObservable() {
}
private static volatile ActionObservable actionObservable = null;
public static ActionObservable getInstance() {
if (actionObservable == null) {
synchronized (ActionObservable.class) {
if (actionObservable == null) {
actionObservable = new ActionObservable();
}
}
}
return actionObservable;
}
/**
* Initialize subscriber
*/
public void initLoginObserver() {
addObserver(new RankingObserver());
addObserver(new OwnerTitleObserver());
}
public void loginNoticeAll(Dto dto) {
setChanged();
notifyObservers(dto);
}
}public abstract class AbsObserver implements Observer {
private final Logger log = Logger.getLogger(AbsObserver.class);
private static ThreadPoolExecutor threadPoolExecutor;
static {
int nThreads = Runtime.getRuntime().availableProcessors() * 2 + 1;
ThreadPoolExecutor.CallerRunsPolicy callerRunsPolicy = new ThreadPoolExecutor.CallerRunsPolicy();
threadPoolExecutor = new ThreadPoolExecutor(nThreads, nThreads, 0L, TimeUnit.MILLISECONDS, new ArrayBlockingQueue<Runnable>(2048), callerRunsPolicy);
}
@Override
public void update(Observable o, Object arg) {
if (o instanceof ActionObservable) {
if (arg instanceof Dto) {
final Dto param = (Dto) arg;
try {
threadPoolExecutor.execute(() -> change(param));
} catch (Exception e) {
log.error("AbsObserver-change Exception param: " + JSON.toJSONString(param), e);
}
}
}
}
/**
* Execute after accepting subscription messages
*
* @param dto
*/
protected abstract void change(Dto dto);
}@Slf4j
public class OwnerTitleObserver extends AbsObserver {
@Override
protected void change(Dto dto) {
// Personalized copywriting statistics
}
}@Slf4j
public class RankingObserver extends AbsObserver {
@Override
protected void change(Dto dto) {
// Rank
}
}Multiple conditions are used to filter the responsibility chain for the data sources of the three actions
Bottom abstraction of responsibility chain
@Slf4j
public abstract class AbsFilter<T> {
/**
* Next processing chain
*/
protected AbsFilter nextFilter;
public AbsFilter setNextFilter(AbsFilter nextFilter) {
return this.nextFilter = nextFilter;
}
public void filter(T param) {
if (param == null) {
return;
}
int order = getOrder();
if (order == 1) {
boolean isDeal = handlerFirstBefore(param);
if (!isDeal) {
return;
}
}
boolean handlerRes = handler(param);
if (handlerRes) {
if (nextFilter != null) {
// Call next chain
nextFilter.filter(param);
}
} else {
handlerAfterFalse(param);
}
}
/**
* processing logic
*
* @param param
* @return
*/
abstract protected boolean handler(T param);
/**
* Preprocessing (only once)
* @param param
* @return Continue processing
*/
protected boolean handlerFirstBefore(T param) {
// Parameter verification can be performed
ValidateUtils.validate(param);
return true;
}
/**
* Post processing (only handle false returned by handler once)
* @param param
* @return
*/
protected void handlerAfterFalse(T param) {}
/**
* The smaller the user-defined sort, the higher it is, starting from 1
* @return
*/
protected abstract int getOrder();
}Responsibility chain business underlying abstraction
@Slf4j
public abstract class AbsOwnerTitleFilter extends AbsFilter<Dto> {
@Override
protected boolean handlerFirstBefore(Dto param) {
super.handlerFirstBefore(param);
return false;
}
protected void commonDeal(Dto dto) {
// Public processing
// Save personalized copy
}
@Override
protected int getOrder() {
return getCurrentOwnerTitleEnum().getOrder();
}
/**
* Get the personalized title of the current representative. The copywriting enumeration can be customized, including copywriting, sorting, type and other fields
*
* @return
*/
protected abstract OwnerTitleEnum getCurrentOwnerTitleEnum();
@Override
protected void handlerAfterFalse(Dto dto) {
commonDeal(param);
}
}Specific filter condition call (example)
@NoArgsConstructor(access = AccessLevel.PRIVATE)
public class OwnerTitleSevenFilter extends AbsOwnerTitleFilter {
@Override
protected boolean handler(Dto dto) {
// If false is returned, the link is completed and the next link is not called. Otherwise, continue to call the next link
return false;
}
@Override
protected OwnerTitleEnum getCurrentOwnerTitleEnum() {
return OwnerTitleEnum.SENEN;
}
}Call entry can be called
private OwnerTitleOneFilter ownerTitleOneFilter = Singleton.get(OwnerTitleOneFilter.class);
// Constructor initialization
private RankingServiceImpl() {
// Responsibility chain filtering
ownerTitleOneFilter
.setNextFilter(ownerTitleTwoFilter)
.setNextFilter(ownerTitleThreeFilter)
.setNextFilter(ownerTitleFourFilter)
.setNextFilter(ownerTitleFiveFilter)
.setNextFilter(ownerTitleSixFilter)
.setNextFilter(ownerTitleSevenFilter);
}Summary:
The main challenge of this demand is
- Multi dimensional sorting of redis zset. We can refer to the implementation of other frameworks. For example, we reuse some ideas of the snowflake algorithm this time. Therefore, we look at the source code. We look more at the ideas and architecture so that we can reuse them in other places rather than just reciting them.
- The effective use of design patterns can greatly reduce the coupling degree of the system. The reason why we don't want to write too much if else is very simple. It is for the sake of clear code and strong scalability. After all, we don't want to edit in a Shishan like code. More importantly, we write a new class for our own code editing, which can also reduce the occurrence of errors.