1. Spring Security
- brief introduction
Spring Security is a framework that focuses on providing identity authentication and authorization for Java applications. Its strength is that it can be easily extended to meet customized requirements - features
- Provide comprehensive and scalable support for identity authentication and authorization
- Prevent various attacks, such as session fixation attack, click hijacking, CSRF attack, etc
- Support integration with Servlet API, Spring MVC and other Web technologies
Related connections: Official website
1.1 brief analysis of spring security function implementation
The bottom layer of Spring Security uses 11 filters for permission control. If you don't log in, you can't even access the dispatcher servlet, let alone the Controller.
Filter and DispatcherServlet are Java EE standards. DispatcherServlet is defined and implemented by spring MVC. In essence, it still follows Java EE standards, while Interceptor and Controller are spring MVC's own.
Recommended website: Spring For All , here is the Chinese tutorial document of Spring Security, which is well written: Community Spring Security tutorial series from getting started to advanced
1.2 trial demo of spring security
Instead of using it on a real project, it's a simplified project (that is, extracting some functions) to experience it with spring security.
Directory structure of demo:
- Introduce dependency
<dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-security</artifactId> </dependency>
- Modify entity class User
In the original User class, inherit the UserDetails interface and override the following methods:
User class specific code:
package com.nowcoder.community.entity;
import org.springframework.security.core.GrantedAuthority;
import org.springframework.security.core.userdetails.UserDetails;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Date;
import java.util.List;
public class User implements UserDetails {
private int id;
private String username;
private String password;
private String salt;
private String email;
private int type;
private int status;
private String activationCode;
private String headerUrl;
private Date createTime;
public int getId() {
return id;
}
public void setId(int id) {
this.id = id;
}
public String getUsername() {
return username;
}
public void setUsername(String username) {
this.username = username;
}
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
public String getSalt() {
return salt;
}
public void setSalt(String salt) {
this.salt = salt;
}
public String getEmail() {
return email;
}
public void setEmail(String email) {
this.email = email;
}
public int getType() {
return type;
}
public void setType(int type) {
this.type = type;
}
public int getStatus() {
return status;
}
public void setStatus(int status) {
this.status = status;
}
public String getActivationCode() {
return activationCode;
}
public void setActivationCode(String activationCode) {
this.activationCode = activationCode;
}
public String getHeaderUrl() {
return headerUrl;
}
public void setHeaderUrl(String headerUrl) {
this.headerUrl = headerUrl;
}
public Date getCreateTime() {
return createTime;
}
public void setCreateTime(Date createTime) {
this.createTime = createTime;
}
@Override
public String toString() {
return "User{" +
"id=" + id +
", username='" + username + '\'' +
", password='" + password + '\'' +
", salt='" + salt + '\'' +
", email='" + email + '\'' +
", type=" + type +
", status=" + status +
", activationCode='" + activationCode + '\'' +
", headerUrl='" + headerUrl + '\'' +
", createTime=" + createTime +
'}';
}
// true: the account has not expired
@Override
public boolean isAccountNonExpired() {
return true;
}
// true: the account is not locked
@Override
public boolean isAccountNonLocked() {
return true;
}
// true: the voucher has not expired
@Override
public boolean isCredentialsNonExpired() {
return true;
}
// true: the account is available
@Override
public boolean isEnabled() {
return true;
}
@Override
public Collection<? extends GrantedAuthority> getAuthorities() {
List<GrantedAuthority> list = new ArrayList<>();
list.add(new GrantedAuthority() {
@Override
public String getAuthority() {
switch (type) {
case 1:
return "ADMIN";
default:
return "USER";
}
}
});
return list;
}
}
- Modify UserService class
UserService inherits the UserDetailsService interface and overrides the loadUserByUsername method
package com.nowcoder.community.service;
import com.nowcoder.community.dao.UserMapper;
import com.nowcoder.community.entity.User;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.security.core.userdetails.UserDetails;
import org.springframework.security.core.userdetails.UserDetailsService;
import org.springframework.security.core.userdetails.UsernameNotFoundException;
import org.springframework.stereotype.Service;
@Service
public class UserService implements UserDetailsService {
@Autowired
private UserMapper userMapper;
public User findUserByName(String username) {
return userMapper.selectByName(username);
}
@Override
public UserDetails loadUserByUsername(String username) throws UsernameNotFoundException {
return this.findUserByName(username);
}
}
- In the config package, create a SecurityConfig class to configure Spring Security
Authentication: authentication
Authorization: authorization
package com.nowcoder.community.config;
import com.nowcoder.community.entity.User;
import com.nowcoder.community.service.UserService;
import com.nowcoder.community.util.CommunityUtil;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.authentication.AuthenticationProvider;
import org.springframework.security.authentication.BadCredentialsException;
import org.springframework.security.authentication.UsernamePasswordAuthenticationToken;
import org.springframework.security.config.annotation.authentication.builders.AuthenticationManagerBuilder;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.builders.WebSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
import org.springframework.security.core.Authentication;
import org.springframework.security.core.AuthenticationException;
import org.springframework.security.core.userdetails.UsernameNotFoundException;
import org.springframework.security.crypto.password.Pbkdf2PasswordEncoder;
import org.springframework.security.web.authentication.AuthenticationFailureHandler;
import org.springframework.security.web.authentication.AuthenticationSuccessHandler;
import org.springframework.security.web.authentication.UsernamePasswordAuthenticationFilter;
import org.springframework.security.web.authentication.logout.LogoutSuccessHandler;
import org.springframework.security.web.authentication.rememberme.InMemoryTokenRepositoryImpl;
import javax.servlet.*;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
@Configuration
public class SecurityConfig extends WebSecurityConfigurerAdapter {
@Autowired
private UserService userService;
@Override
public void configure(WebSecurity web) throws Exception {
// Ignore access to static resources
web.ignoring().antMatchers("/resources/**");
}
// authentication : authentication
// AuthenticationManager: Core interface of authentication
// AuthenticationManagerBuilder: Tools for building AuthenticationManager objects
// ProviderManager: AuthenticationManager Interface
@Override
protected void configure(AuthenticationManagerBuilder auth) throws Exception {
// Built in authentication rules
// auth.userDetailsService(userService).passwordEncoder(new Pbkdf2PasswordEncoder("12345"));
// Custom authentication rules
// AuthenticationProvider: ProviderManager It holds a group of authenticationproviders, and each AuthenticationProvider is responsible for one kind of authentication
// Delegation mode: ProviderManager delegates authentication to AuthenticationProvider
auth.authenticationProvider(new AuthenticationProvider() {
// Authentication: The interface used to encapsulate authentication information. Different implementation classes represent different types of authentication information
@Override
public Authentication authenticate(Authentication authentication) throws AuthenticationException {
String username = authentication.getName();// User's incoming account
String password = (String) authentication.getCredentials();// Password passed in by user
User user = userService.findUserByName(username);
if (user == null) {
throw new UsernameNotFoundException("Account does not exist! ");
}
password = CommunityUtil.md5(password + user.getSalt());
if (!user.getPassword().equals(password)) {
throw new BadCredentialsException("Incorrect password! ");
}
// principal: Main information; credentials: certificate; authorities: authority;
return new UsernamePasswordAuthenticationToken(user, user.getPassword(), user.getAuthorities());
}
// What type of authentication does the current AuthenticationProvider support
@Override
public boolean supports(Class<?> aClass) {
// UsernamePasswordAuthenticationToken: Authentication Interface is a common implementation class
return UsernamePasswordAuthenticationToken.class.equals(aClass);
}
});
}
//authorization: to grant authorization
@Override
protected void configure(HttpSecurity http) throws Exception {
//super.configure(http);//Override this method
// Login related configuration
http.formLogin()
.loginPage("/loginpage")//See HomeController for the path to the login page
.loginProcessingUrl("/login")//For the path of login form submission, see login Form submission path in HTML page
//.successForwardUrl()//Where to jump when successful. But because we need to carry some parameters in addition to some logic, we use the following successHandler() will be more flexible.
//.failureForwardUrl()//Where to jump when you fail. Similarly.
.successHandler(new AuthenticationSuccessHandler() {
@Override
public void onAuthenticationSuccess(HttpServletRequest request, HttpServletResponse response, Authentication authentication) throws IOException, ServletException {
response.sendRedirect(request.getContextPath() + "/index");
}
})
.failureHandler(new AuthenticationFailureHandler() {
@Override
public void onAuthenticationFailure(HttpServletRequest request, HttpServletResponse response, AuthenticationException e) throws IOException, ServletException {
request.setAttribute("error", e.getMessage());
request.getRequestDispatcher("/loginpage").forward(request, response);
}
});
// Exit related configuration
http.logout()
.logoutUrl("/logout")
.logoutSuccessHandler(new LogoutSuccessHandler() {
@Override
public void onLogoutSuccess(HttpServletRequest request, HttpServletResponse response, Authentication authentication) throws IOException, ServletException {
response.sendRedirect(request.getContextPath() + "/index");
}
});
// Authorization configuration
http.authorizeRequests() //When a user does not log in, he does not have any permissions
.antMatchers("/letter").hasAnyAuthority("USER", "ADMIN") //As long as you have any of the permissions of "user" and "admin", you can access the "private letter" / letter "page
.antMatchers("/admin").hasAnyAuthority("ADMIN")
.and().exceptionHandling().accessDeniedPage("/denied");//If the permissions do not match, jump to the "/ denied" page
//The verification code should be processed before the account password. If the verification codes are wrong, you don't have to look at the account password. Therefore, add a Filter verification code before verifying the Filter of the account password
// Add Filter to process verification code
http.addFilterBefore(new Filter() {
@Override
public void doFilter(ServletRequest servletRequest, ServletResponse servletResponse, FilterChain filterChain) throws IOException, ServletException {
HttpServletRequest request = (HttpServletRequest) servletRequest;
HttpServletResponse response = (HttpServletResponse) servletResponse;
if (request.getServletPath().equals("/login")) {
String verifyCode = request.getParameter("verifyCode");
if (verifyCode == null || !verifyCode.equalsIgnoreCase("1234")) {
request.setAttribute("error", "Verification code error! ");
request.getRequestDispatcher("/loginpage").forward(request, response);
return; //If the verification code is incorrect, the request will not continue to execute downward
}
}
// Let the request continue down
filterChain.doFilter(request, response);
}
}, UsernamePasswordAuthenticationFilter.class); //The new Filter should be filtered before the UsernamePasswordAuthenticationFilter
/*
If "remember me" is checked, Spring Security will save a cookie in the browser, in which the user's user name is stored,
Then, turn off the browser / shut down. When accessing again next time, the browser will pass the cookie to the server, and the server will find out the user according to the user name and userService,
Then, the user will be stored in the SecurityContext through the SecurityContextHolder,
Then, when the user accesses the "/ index" page, the user name will be taken from the SecurityContext and displayed on the home page
*/
// Remember me
http.rememberMe()
.tokenRepository(new InMemoryTokenRepositoryImpl())//If you want to save the data to Redis / database, implement the TokenRepository interface yourself, and then tokenRepository(tokenRepository)
.tokenValiditySeconds(3600 * 24)//24 hours
.userDetailsService(userService);//Must have
}
}
- index.html page
<!DOCTYPE html>
<html lang="en" xmlns:th="http://www.thymeleaf.org">
<head>
<meta charset="UTF-8">
<title>home page</title>
</head>
<body>
<h1>Community home page</h1>
<!--Welcome message-->
<p th:if="${loginUser!=null}">
Welcome, <span th:text="${loginUser.username}"></span>!
</p>
<ul>
<li><a th:href="@{/discuss}">Post details</a></li>
<li><a th:href="@{/letter}">Private message list</a></li>
<li><a th:href="@{/loginpage}">Sign in</a></li>
<!--SpringSecurity It is specified that exit must be used post Request. first<li>yes get Request. the second<li>yes post Request, post Request must use form Form.--
<!--<li><a th:href="@{/loginpage}">sign out</a></li>-->
<li>
<form method="post" th:action="@{/logout}">
<a href="javascript:document.forms[0].submit();">sign out</a>
</form>
</li>
</ul>
</body>
</html>
- HomeController class
package com.nowcoder.community.controller;
import com.nowcoder.community.entity.User;
import org.springframework.security.core.context.SecurityContextHolder;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
@Controller
public class HomeController {
@RequestMapping(path = "/index", method = RequestMethod.GET)
public String getIndexPage(Model model) {
// After successful authentication, the results will be stored in the SecurityContext through the SecurityContextHolder
Object obj = SecurityContextHolder.getContext().getAuthentication().getPrincipal();
if (obj instanceof User) {
model.addAttribute("loginUser", obj);
}
return "/index";
}
@RequestMapping(path = "/discuss", method = RequestMethod.GET)
public String getDiscussPage() {
return "/site/discuss";
}
@RequestMapping(path = "/letter", method = RequestMethod.GET)
public String getLetterPage() {
return "/site/letter";
}
@RequestMapping(path = "/admin", method = RequestMethod.GET)
public String getAdminPage() {
return "/site/admin";
}
@RequestMapping(path = "/loginpage", method = {RequestMethod.GET, RequestMethod.POST})
public String getLoginPage() {
return "/site/login";
}
// Prompt page when access is denied
@RequestMapping(path = "/denied", method = RequestMethod.GET)
public String getDeniedPage() {
return "/error/404";
}
}
- login.html
<!DOCTYPE html>
<html lang="en" xmlns:th="http://www.thymeleaf.org">
<head>
<meta charset="UTF-8">
<title>Sign in</title>
</head>
<body>
<h1>Login community</h1>
<form method="post" th:action="@{/login}">
<p style="color:red;" th:text="${error}">
<!--Prompt information-->
</p>
<p>
account number:<input type="text" name="username" th:value="${param.username}">
</p>
<p>
password:<input type="password" name="password" th:value="${param.password}">
</p>
<p>
Verification Code:<input type="text" name="verifyCode"> <i>1234</i>
</p>
<p>
<input type="checkbox" name="remember-me"> Remember me
</p>
<p>
<input type="submit" value="Sign in">
</p>
</form>
</body>
</html>
2. Permission control
- Login check
- Before this case, there was an interceptor to realize login check. This is a simple permission management scheme. Now it is abandoned
- Authorization configuration
- Assign access rights to all requests contained in the current system (ordinary users, moderators, administrators)
- Certification scheme
- Bypass the Security authentication process and adopt the original authentication scheme of the system
- CSRF configuration
- The basic principle of preventing CSRF attack and the configuration related to forms and AJAx
This time, Spring Security is introduced into the actual project to make changes
Note:
Assistant V of Niuke course replied to Eric Lee
- Security provides two functions: authentication and authorization. We also demonstrated it in DEMO. When we applied it in the project, we did not use its authentication function, but used its authorization function alone. Therefore, we need to do some special processing on the authentication link to ensure the normal operation of authorization;
- All functions of Security are implemented based on Filter, and the implementation of Filter is earlier than that of Interceptor and Controller. You can refer to the Filter principle of Security http://www.spring4all.com/article/458 ；
- Our solution is to judge whether to log in in the Interceptor, and then artificially add the authentication results to the SecurityContextHolder. Note that since the Interceptor executes later than the Filter, the authentication depends on the Interceptor processing of the previous request. For example, I successfully logged in, and then requested to redirect to the home page. When accessing the home page, the authentication Filter does not work because the request does not require permission, and then the Interceptor is executed. At this time, the authentication result is added to the SecurityContextHolder. Then you can access / letter/list successfully, because in this request, the Filter judges that you have permission according to the authentication result just now;
- When exiting, you need to clean up the authentication results in the SecurityContextHolder, so that the Filter can correctly identify the user's permissions in the next request;
- In fact, there is no need to clean up the SecurityContextHolder in the afterCompletion in LoginTicketInterceptor. Delete this sentence.
Eric.Lee replied to the teaching assistant of Niuke course: for the next request, does Security find the corresponding user information and permissions saved in the SecurityContextHolder through the cookie carried in the user request?
2020-01-22 17:17:14
Assistant V: the bottom layer of SecurityContextHolder uses Session to store data by default, and Session depends on cookies
2020-02-10 12:14:55
2.1 login check
Before this case, there was an interceptor to realize login check. This is a simple permission management scheme. Now it is abandoned
Discard the interceptor settings in WebMvcConfig
2.2 authorization configuration
All requests contained in the current system are assigned access (ordinary users, moderators and administrators)
- In the CommunityConstant interface, add several attributes
/**
* Permissions: ordinary users
*/
String AUTHORITY_USER = "user";
/**
* Permissions: Administrator
*/
String AUTHORITY_ADMIN = "admin";
/**
* Permission: moderator
*/
String AUTHORITY_MODERATOR = "moderator";
- Create a new SecurityConfig class in the config package
package com.ateam.community.config;
import com.ateam.community.util.CommunityConstant;
import com.ateam.community.util.CommunityUtil;
import org.springframework.context.annotation.Configuration;
import org.springframework.security.access.AccessDeniedException;
import org.springframework.security.config.annotation.web.builders.HttpSecurity;
import org.springframework.security.config.annotation.web.builders.WebSecurity;
import org.springframework.security.config.annotation.web.configuration.WebSecurityConfigurerAdapter;
import org.springframework.security.core.AuthenticationException;
import org.springframework.security.web.AuthenticationEntryPoint;
import org.springframework.security.web.access.AccessDeniedHandler;
import javax.servlet.ServletException;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
import java.io.IOException;
import java.io.PrintWriter;
@Configuration
public class SecurityConfig extends WebSecurityConfigurerAdapter implements CommunityConstant {
@Override
public void configure(WebSecurity web) throws Exception {
web.ignoring().antMatchers("/resources/**");
}
@Override
protected void configure(HttpSecurity http) throws Exception {
// to grant authorization
http.authorizeRequests()
.antMatchers(
"/user/setting",
"/user/upload",
"/user/update/password",
"/discuss/add",
"/comment/add/**",
"/letter/**",
"/notice/**",
"/like",
"follow",
"/unfollow"
)
.hasAnyAuthority(
AUTHORITY_ADMIN,
AUTHORITY_MODERATOR,
AUTHORITY_USER
)
.antMatchers(
"/discuss/top",
"/discuss/wonderful"
)
.hasAnyAuthority(
AUTHORITY_MODERATOR
)
.antMatchers(
"/discuss/delete",
"/data/**",
"/actuator/**"
)
.hasAnyAuthority(
AUTHORITY_ADMIN
)
.anyRequest().permitAll() // Except for the above path, other requests are allowed
.and().csrf().disable(); //Disable CSRF
//The function of preventing csrf in Spring Security is canceled here, because the teacher is too lazy to change all asynchronous requests to have tocken, but if this function is available, it must be available everywhere
// Handling of insufficient authorization
http.exceptionHandling()
.authenticationEntryPoint(new AuthenticationEntryPoint() { // Processing without login
// Not logged in authenticationEntryPoint() is how to handle when there is no login
// Processing idea: the synchronous request jumps to the login page; An asynchronous request is returned by splicing a json string
@Override
public void commence(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse, AuthenticationException e) throws IOException, ServletException {
String xRequestedWith = httpServletRequest.getHeader("x-requested-with");//The fields in the corresponding header of the browser determine whether the request is synchronous or asynchronous
if ("XMLHttpRequest".equals(xRequestedWith)) {
// Is an asynchronous request
httpServletResponse.setContentType("application/plain;charset=utf-8");
PrintWriter writer = httpServletResponse.getWriter();
writer.write(CommunityUtil.getJSONString(403,"You haven't logged in yet"));
} else {
httpServletResponse.sendRedirect(httpServletRequest.getContextPath() + "/login");
}
}
})
.accessDeniedHandler(new AccessDeniedHandler() { // Handling of insufficient authorization
// Insufficient permissions accessDeniedHandler() is how to handle when the configuration permission is insufficient
@Override
public void handle(HttpServletRequest httpServletRequest, HttpServletResponse httpServletResponse, AccessDeniedException e) throws IOException, ServletException {
String xRequestedWith = httpServletRequest.getHeader("x-requested-with");
if ("XMLHttpRequest".equals(xRequestedWith)) {
// Is an asynchronous request
httpServletResponse.setContentType("application/plain;charset=utf-8");
PrintWriter writer = httpServletResponse.getWriter();
writer.write(CommunityUtil.getJSONString(403,"You do not have permission for this function!"));
} else {
httpServletResponse.sendRedirect(httpServletRequest.getContextPath() + "/denied");
}
}
});
// The Security bottom layer will intercept the / logout request by default for exit processing
// Override its default logic to execute our own exit code
http.logout().logoutUrl("securitylogout"); //Let it intercept a path we don't use
}
}
- Modify UserService
Add the following method:
public Collection<? extends GrantedAuthority> getAuthorities(int userId) {
User user = this.findUserById(userId);
ArrayList<GrantedAuthority> list = new ArrayList<>();
list.add(new GrantedAuthority() {
@Override
public String getAuthority() {
switch (user.getUserType()) {
case 1 :
return AUTHORITY_ADMIN;
case 2 :
return AUTHORITY_MODERATOR;
default:
return AUTHORITY_USER;
}
}
});
return list;
}
2.3 certification scheme
Bypass the Security authentication process and adopt the original authentication scheme of the system
Modify loginticketinterceptor Java, modified the methods: preHandle and afterCompletion
package com.ateam.community.controller.interceptor;
@Component
public class LoginTicketInterceptor implements HandlerInterceptor {
@Autowired
private UserService userService;
@Autowired
private HostHolder hostHolder;
@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
// Get credentials from cookie s
String ticket = CookieUtil.getValue(request, "ticket");
if (ticket != null) {
// Query voucher
LoginTicket loginTicket = userService.findLoginTicket(ticket);
// Check whether the voucher is valid
if (loginTicket != null && loginTicket.getStatus() == 0 && loginTicket.getExpired().after(new Date())) {
// Voucher query user
User user = userService.findUserById(loginTicket.getUserId());
// Hold user in this request
hostHolder.setUser(user);
// The results of user authentication are built and stored in the SecurityContext for Security authorization
Authentication authentication = new UsernamePasswordAuthenticationToken(
user, user.getPassword(), userService.getAuthorities(user.getId()));
SecurityContextHolder.setContext(new SecurityContextImpl(authentication));
}
}
return true;
}
@Override
public void postHandle(HttpServletRequest request, HttpServletResponse response, Object handler, ModelAndView modelAndView) throws Exception {
User user = hostHolder.getUser();
if (user != null && modelAndView != null) {
modelAndView.addObject("loginUser", user);
}
}
@Override
public void afterCompletion(HttpServletRequest request, HttpServletResponse response, Object handler, Exception ex) throws Exception {
hostHolder.clear();
//This sentence only exists in the logout() method of the LoginController class.
//You don't need this line of code
SecurityContextHolder.clearContext();
}
}
2.4 CSRF configuration
csrf means that other users access the server by obtaining cookie s and ticket s from the client. security can generate TOKEN data, which is hidden to prevent csrf attacks.
csrf attack principle and Spring Security solution.
- For form forms, Spring Security will automatically generate a token to prevent CSFR
- For asynchronous requests, you must write your own token to prevent CSFR
Example:
Introduce dependency
<!-- thymeleaf security-->
<dependency>
<groupId>org.thymeleaf.extras</groupId>
<artifactId>thymeleaf-extras-springsecurity5</artifactId>
</dependency>
Add the following code (the two annotated lines) to the page that needs to submit asynchronous requests
index.html
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1, shrink-to-fit=no">
<!-- When you access this page, a message is generated here CSRF token-->
<!-- <meta name="_csrf" th:content="${_csrf.token}">-->
<!-- <meta name="_csrf_header" th:content="${_csrf.headerName}">-->
<link rel="icon" href="https://static.nowcoder.com/images/logo_87_87.png"/>
<link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.3.1/css/bootstrap.min.css" crossorigin="anonymous">
<link rel="stylesheet" th:href="@{/css/global.css}" />
<title>ATeam-home page</title>
</head>
Corresponding JS, index js
// Before sending the AJAX request, set the CSRF token into the request header
// var token = $("meta[name='_csrf']").attr("content");
// var header = $("meta[name='_csrf_header']").attr("content");
// $(document).ajaxSend(function(e, xhr, options){
// xhr.setRequestHeader(header, token);
// });
Note:
Generally, all asynchronous requests need to be configured, otherwise it is unsafe and cannot pass.
In order to save trouble, teachers do not configure CSRF, so they need it when authorizing and().csrf().disable();
3. Top, refine and delete
- Function realization
- Click "top" to modify the type of post
- Click "refine" and "delete" to modify the status of the post
- Authority management
- Moderators can perform "top setting" and "refining"
- Administrators can perform delete operations
- Button display
- The moderator can see the "top" and "refine" buttons
- The administrator can see the delete button
2.1 function realization
- Data layer
Add the following methods in DiscussPostMapper under dao package
int updateDiscussType(int id, int discussType);
int updateStatus(int id, int status);
In mapper, discuss mapper XML, add corresponding SQL
<update id="updateDiscussType" >
update discuss_post
set discuss_type = #{discussType}
where id = #{id}
</update>
<update id="updateStatus">
update discuss_post
set status = #{status}
where id = #{id}
</update>
- Service layer
Add a new method in DiscussPostService class under service package
public int updateDiscussType(int id, int discussType) {
return discussPostMapper.updateDiscussType(id, discussType);
}
public int updateStatus(int id, int status) {
return discussPostMapper.updateStatus(id, status);
}
- View layer
In DiscussPostController under controller package, add a new method
// Topping
@RequestMapping(value = "/top", method = RequestMethod.POST)
@ResponseBody
public String setTop(int id) {
discussPostService.updateDiscussType(id,1);
// Trigger posting event
Event event = new Event()
.setTopic(TOPIC_PUBLISH)
.setUserId(hostHolder.getUser().getId())
.setEntityType(ENTITY_TYPE_POST)
.setEntityId(id);
eventProducer.fireEvent(event);
return CommunityUtil.getJSONString(0);
}
// Refining
@RequestMapping(value = "/wonderful", method = RequestMethod.POST)
@ResponseBody
public String setWonderful(int id) {
discussPostService.updateStatus(id,1);
// Trigger posting event
Event event = new Event()
.setTopic(TOPIC_PUBLISH)
.setUserId(hostHolder.getUser().getId())
.setEntityType(ENTITY_TYPE_POST)
.setEntityId(id);
eventProducer.fireEvent(event);
return CommunityUtil.getJSONString(0);
}
// delete
@RequestMapping(value = "/delete", method = RequestMethod.POST)
@ResponseBody
public String setDelete(int id) {
discussPostService.updateStatus(id,2);
// Trigger post deletion event
Event event = new Event()
.setTopic(TOPIC_DELETE)
.setUserId(hostHolder.getUser().getId())
.setEntityType(ENTITY_TYPE_POST)
.setEntityId(id);
eventProducer.fireEvent(event);
return CommunityUtil.getJSONString(0);
}
Consume the post deletion event in the EventConsumer class under the event package
// Consumption post deletion event
@KafkaListener(topics = {TOPIC_DELETE})
public void handleDeleteMessage(ConsumerRecord record) {
if (record == null || record.value() == null) {
logger.error("The content of the message is empty!");
return;
}
// Using fastjson to convert json string into Event object
Event event = JSONObject.parseObject(record.value().toString(), Event.class);
if (event == null) {
logger.error("Message format error!");
return;
}
elasticsearchService.deleteDiscussPost(event.getEntityId());
}
In discussions js file, bind three js click events for topping, refining and deleting.
$(function (){
$("#topBtn").click(setTop);
$("#wonderfulBtn").click(setWonderful);
$("#deleteBtn").click(setDelete);
});
// Topping
function setTop() {
$.post(
CONTEXT_PATH + "/discuss/top",
{"id":$("#postId").val()},
function(data) {
data = $.parseJSON(data);
if(data.code == 0) {
$("#topBtn").attr("disabled","disabled");
} else {
alert(data.msg);
}
}
);
}
// Refining
function setWonderful() {
$.post(
CONTEXT_PATH + "/discuss/wonderful",
{"id":$("#postId").val()},
function(data) {
data = $.parseJSON(data);
if(data.code == 0) {
$("#wonderfulBtn").attr("disabled","disabled");
} else {
alert(data.msg);
}
}
);
}
// delete
function setDelete() {
$.post(
CONTEXT_PATH + "/discuss/delete",
{"id":$("#postId").val()},
function(data) {
data = $.parseJSON(data);
if(data.code == 0) {
location.href = CONTEXT_PATH + "/index"; //Jump to home page
} else {
alert(data.msg);
}
}
);
}
2.2 authority management
Permission management includes two parts:
1. The server should deny access to this function to users without permission.
2. If the client wants to, the functions that the user does not have permission to access will not be displayed on the page.
Moderators can perform "top" and "refine" operations, and administrators can perform "delete" operations
Configure permissions under SecurityConfig.
.antMatchers(
"/discuss/top",// Topping
"/discuss/wonderful"// Refining
)
.hasAnyAuthority(
AUTHORITY_MODERATOR
)
.antMatchers(
"/discuss/delete",// delete
)
.hasAnyAuthority(
AUTHORITY_ADMIN
)
Moderators can execute the "top" and "refine" buttons, administrators can see the "delete" button, and other users can't see the above buttons
On the html page, add xmlns:sec=“ http://www.thymeleaf.org/extras/spring-security ”, introduce permission control at each button
Modify discussions detail HTML page
<!doctype html>
<html lang="en" xmlns:th="http://www.thymeleaf.org" xmlns:sec="http://www.thymeleaf.org/extras/spring-security">
<!--only moderator Only users with permissions can see this button:sec:authorize="hasAnyAuthority('moderator')"-->
<div class="float-right">
<input type="hidden" id="postId" th:value="${post.id}">
<button type="button" class="btn btn-danger btn-sm" id="topBtn"
th:disabled="${post.discussType==1}" sec:authorize="hasAnyAuthority('moderator')">Topping</button>
<button type="button" class="btn btn-danger btn-sm" id="wonderfulBtn"
th:disabled="${post.status==1}" sec:authorize="hasAnyAuthority('moderator')">Refining</button>
<button type="button" class="btn btn-danger btn-sm" id="deleteBtn"
th:disabled="${post.status==2}" sec:authorize="hasAnyAuthority('admin')">delete</button>
</div>
4. Redis advanced data type
- HyperLogLog
- A cardinality algorithm is used to complete the statistics of independent totals
- The occupied space is small. No matter how many data are counted, it only occupies 12K memory space
- The standard error is 0.81%
- Bitmap
- It is not an independent data structure, but actually a string
- It supports bitwise access to data, which can be regarded as a byte array
- Boolean values suitable for storing large amounts of continuous data
This section tests whether hyperlog and Bitmap are used or in RedisTests class under test package
package com.ateam.community;
import com.ateam.community.util.CommunityUtil;
import com.ateam.community.util.RedisKeyUtil;
import org.apache.kafka.common.protocol.types.Field;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.dao.DataAccessException;
import org.springframework.data.redis.connection.RedisConnection;
import org.springframework.data.redis.connection.RedisStringCommands;
import org.springframework.data.redis.core.*;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringRunner;
import javax.swing.*;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Calendar;
import java.util.Date;
With(SpringRunner.class)
@SpringBootTest
@ContextConfiguration(classes = CommunityApplication.class)//Configuration class
public class RedisTests {
@Autowired
private RedisTemplate redisTemplate;
/*HyperLogLog*/
// Count the independent total of 200000 duplicate data. Equivalent to 200000 visits, how many uv (unique visitor, independent IP: refers to independent users / independent visitors) do you want to count
// Count the total number of 200000 duplicate data
@Test
public void testHyperLogLog(){
String redis = "test:hll:01";
for (int i = 0; i < 100000; i++) {
redisTemplate.opsForHyperLogLog().add(redis,i);
}
for (int i = 0; i < 100000; i++) {
int r = (int) (Math.random() * 100000 + 1);
redisTemplate.opsForHyperLogLog().add(redis, r);
}
long size = redisTemplate.opsForHyperLogLog().size(redis);
System.out.println(size);
}
// Merge the three groups of data, and then count the independent total number of duplicate data after merging
// It's quite the same as you know the daily traffic data. You want to know the independent uv of these three days
@Test
public void testHyperLogLogUnion(){
String redis2 = "test:hll:02";
for (int i = 1; i <= 10000 ; i++) {
redisTemplate.opsForHyperLogLog().add(redis2,i);
}
String redis3 = "test:hll:03";
for (int i = 5001; i <= 15000 ; i++) {
redisTemplate.opsForHyperLogLog().add(redis3,i);
}
String redis4 = "test:hll:04";
for (int i = 10001; i <= 20000 ; i++) {
redisTemplate.opsForHyperLogLog().add(redis4,i);
}
String unionKey = "test:hll:union";
redisTemplate.opsForHyperLogLog().union(unionKey,redis2,redis3,redis4);
long size = redisTemplate.opsForHyperLogLog().size(unionKey);
System.out.println(size);
}
/*Bitmap*/
// Counts the Boolean value of a set of data. In a year, it will be set as 1 after check-in, otherwise it will be 0 by default, and then the number of arrivals in a year will be counted.
// A Boolean value that counts a set of data
@Test
public void testBitMap(){
String redisKey = "test:bm:01";
// Record / / if not set, the default value is false
redisTemplate.opsForValue().setBit(redisKey,1,true);
redisTemplate.opsForValue().setBit(redisKey,4,true);
redisTemplate.opsForValue().setBit(redisKey,7,true);
redisTemplate.opsForValue().setBit(redisKey,9,true);
// query
System.out.println(redisTemplate.opsForValue().getBit(redisKey,0));
System.out.println(redisTemplate.opsForValue().getBit(redisKey,4));
System.out.println(redisTemplate.opsForValue().getBit(redisKey,7));
System.out.println(redisTemplate.opsForValue().getBit(redisKey,9));
// Statistics
Object obj = redisTemplate.execute(new RedisCallback() {
@Override
public Object doInRedis(RedisConnection redisConnection) throws DataAccessException {
return redisConnection.bitCount(redisKey.getBytes());
}
});
System.out.println(obj);
}
// Count the Boolean values of the three groups of data, and perform OR operation on the three groups of data
@Test
public void testBitMapOperation(){
String redisKey2 = "test:bm:02";
// record
redisTemplate.opsForValue().setBit(redisKey2,0,true);
redisTemplate.opsForValue().setBit(redisKey2,1,true);
redisTemplate.opsForValue().setBit(redisKey2,2,true);
String redisKey3 = "test:bm:03";
// record
redisTemplate.opsForValue().setBit(redisKey3,2,true);
redisTemplate.opsForValue().setBit(redisKey3,3,true);
redisTemplate.opsForValue().setBit(redisKey3,4,true);
String redisKey4 = "test:bm:04";
// record
redisTemplate.opsForValue().setBit(redisKey4,4,true);
redisTemplate.opsForValue().setBit(redisKey4,5,true);
redisTemplate.opsForValue().setBit(redisKey4,6,true);
String redisKey = "test:bm:or";
Object obj = redisTemplate.execute(new RedisCallback() {
@Override
public Object doInRedis(RedisConnection redisConnection) throws DataAccessException {
redisConnection.bitOp(RedisStringCommands.BitOperation.OR,
redisKey.getBytes(), redisKey2.getBytes(), redisKey3.getBytes(), redisKey4.getBytes());
return redisConnection.bitCount(redisKey.getBytes());
}
});
System.out.println(obj);
//The following seven true values are output
System.out.println(redisTemplate.opsForValue().getBit(redisKey, 0));
System.out.println(redisTemplate.opsForValue().getBit(redisKey, 1));
System.out.println(redisTemplate.opsForValue().getBit(redisKey, 2));
System.out.println(redisTemplate.opsForValue().getBit(redisKey, 3));
System.out.println(redisTemplate.opsForValue().getBit(redisKey, 4));
System.out.println(redisTemplate.opsForValue().getBit(redisKey, 5));
System.out.println(redisTemplate.opsForValue().getBit(redisKey, 6));
}
5. Website data statistics
- UV(Unique Vistor)
- For independent visitors, you need to remove the weight statistics through the user's IP
- Statistics are required for each visit
- Hyperlog has good performance and small access space
- DAU(Daily Active User)
- Daily active users, weight removal statistics by user ID
- Once visited, it is considered active
- Bitmap has good performance and can count accurate results
- Website data statistics is based on Redis. In RedisUtil class, create a new key
public class RedisKeyUtil {
private static final String SPLIT = ":";
private static final String PREFIX_UV = "uv";
private static final String PREFIX_DAU = "dau";
// Single day uv
public static String getUVKey(String date) {
return PREFIX_UV + SPLIT + date;
}
// Interval UV
public static String getUVKey(String startDate, String endDate) {
return PREFIX_UV + SPLIT + startDate + SPLIT + endDate;
}
// Single day DAU
public static String getDAUKey(String date) {
return PREFIX_DAU + SPLIT + date;
}
// Interval DAU
public static String getDAUKey(String startDate, String endDate) {
return PREFIX_DAU + SPLIT + startDate + SPLIT + endDate;
}
}
- Under the service package, create a new class DataService
package com.ateam.community.service;
import com.ateam.community.util.RedisKeyUtil;
import org.apache.kafka.common.protocol.types.Field;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.dao.DataAccessException;
import org.springframework.data.redis.connection.RedisConnection;
import org.springframework.data.redis.connection.RedisStringCommands;
import org.springframework.data.redis.core.RedisCallback;
import org.springframework.data.redis.core.RedisTemplate;
import org.springframework.stereotype.Service;
import java.text.SimpleDateFormat;
import java.util.ArrayList;
import java.util.Calendar;
import java.util.Date;
@Service
public class DataService {
@Autowired
private RedisTemplate redisTemplate;
private SimpleDateFormat sdf = new SimpleDateFormat("yyyyMMdd");
// Counts the specified IP into the UV
public void recordUV(String ip) {
String redisKey = RedisKeyUtil.getUVKey(sdf.format(new Date()));
redisTemplate.opsForHyperLogLog().add(redisKey,ip);
}
// Counts UV s within the specified date range
public long calculateUV(Date start, Date end) {
if (start == null || end == null) {
throw new IllegalArgumentException("Parameter cannot be empty");
}
// Organize key s within the date range
ArrayList<String> keyList = new ArrayList<>();
Calendar calendar = Calendar.getInstance();
calendar.setTime(start);
while (!calendar.getTime().after(end)) {
String key = RedisKeyUtil.getUVKey(sdf.format(calendar.getTime()));
keyList.add(key);
calendar.add(Calendar.DATE,1);
}
// Merge these data
String redisKey = RedisKeyUtil.getUVKey(sdf.format(start), sdf.format(end));
redisTemplate.opsForHyperLogLog().union(redisKey,keyList.toArray());
// Return statistics
return redisTemplate.opsForHyperLogLog().size(redisKey);
}
// Count the specified user into the DAU
public void recordDAU(int userId){
String redisKey = RedisKeyUtil.getDAUKey(sdf.format(new Date()));
redisTemplate.opsForValue().setBit(redisKey,userId,true);
}
// Counts daus within the specified date range
public long calculateDAU(Date start, Date end) {
if (start == null || end == null) {
throw new IllegalArgumentException("Parameter cannot be empty");
}
// Organize key s within the date range
ArrayList<byte[]> keyList = new ArrayList<>();
Calendar calendar = Calendar.getInstance();
calendar.setTime(start);
while (!calendar.getTime().after(end)) {
String key = RedisKeyUtil.getDAUKey(sdf.format(calendar.getTime()));
keyList.add(key.getBytes());
calendar.add(Calendar.DATE,1);
}
// Perform OR operation
return (long) redisTemplate.execute(new RedisCallback() {
@Override
public Object doInRedis(RedisConnection redisConnection) throws DataAccessException {
String redisKey = RedisKeyUtil.getDAUKey(sdf.format(start), sdf.format(end));
redisConnection.bitOp(RedisStringCommands.BitOperation.OR,
redisKey.getBytes(),keyList.toArray(new byte[0][0])); //Convert to byte[0][0] format
return redisConnection.bitCount(redisKey.getBytes());
}
});
}
}
- Write interceptor DataInterceptor
package com.ateam.community.controller.interceptor;
import com.ateam.community.entity.User;
import com.ateam.community.service.DataService;
import com.ateam.community.util.HostHolder;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.stereotype.Component;
import org.springframework.web.servlet.HandlerInterceptor;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;
@Component
public class DataInterceptor implements HandlerInterceptor {
@Autowired
private DataService dataService;
@Autowired
private HostHolder hostHolder;
@Override
public boolean preHandle(HttpServletRequest request, HttpServletResponse response, Object handler) throws Exception {
// Statistical UV
String ip = request.getRemoteHost();
dataService.recordUV(ip);
// Statistical DAU
User user = hostHolder.getUser();
if (user != null) {
dataService.recordDAU(user.getId());
}
return true;
}
}
In WebMvcConfig, configure the interceptor
package com.ateam.community.config;
import com.ateam.community.controller.interceptor.*;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.servlet.config.annotation.InterceptorRegistry;
import org.springframework.web.servlet.config.annotation.WebMvcConfigurer;
@Configuration
public class WebMvcConfig implements WebMvcConfigurer {
@Autowired
private AlphaInterceptor alphaInterceptor;
@Autowired
private LoginTicketInterceptor loginTicketInterceptor;
// @Autowired
// private LoginRequiredInterceptor loginRequiredInterceptor;
@Autowired
private MessageInterceptor messageInterceptor;
@Autowired
private DataInterceptor dataInterceptor;
@Override
public void addInterceptors(InterceptorRegistry registry) {
registry.addInterceptor(alphaInterceptor)
.excludePathPatterns("/**/*.css","/**/*.js","/**/*.png","/**/*.jpg","/**/*.jpeg")//wildcard
.addPathPatterns("/register","/login");
registry.addInterceptor(loginTicketInterceptor)
.excludePathPatterns("/**/*.css","/**/*.js","/**/*.png","/**/*.jpg","/**/*.jpeg");//wildcard
// This is the login authentication written by yourself. It is now managed by spring security. This is obsolete
// registry.addInterceptor(loginRequiredInterceptor)
// .excludePathPatterns("/**/*.css","/**/*.js","/**/*.png","/**/*.jpg","/**/*.jpeg");// wildcard
registry.addInterceptor(messageInterceptor)
.excludePathPatterns("/**/*.css","/**/*.js","/**/*.png","/**/*.jpg","/**/*.jpeg");//wildcard
registry.addInterceptor(dataInterceptor)
.excludePathPatterns("/**/*.css","/**/*.js","/**/*.png","/**/*.jpg","/**/*.jpeg");//wildcard
}
}
- Under the controller package, create a new DataController class to process the request
package com.ateam.community.controller;
import com.ateam.community.service.DataService;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.format.annotation.DateTimeFormat;
import org.springframework.stereotype.Controller;
import org.springframework.ui.Model;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import java.util.Date;
@Controller
public class DataController {
@Autowired
private DataService dataService;
// Statistics page
@RequestMapping(value = "/data", method = {RequestMethod.GET,RequestMethod.POST})
public String getDataPage() {
return "/site/admin/data";
}
// Statistics website UV
@RequestMapping(value = "/data/uv", method = RequestMethod.POST)
//The client passes a Date string. Spring accepts this string to be converted to Date, but you need to tell it the format of the Date string
public String getUV(@DateTimeFormat(pattern = "yyyy-MM-dd") Date start,
@DateTimeFormat(pattern = "yyyy-MM-dd") Date end,
Model model) {
long uv = dataService.calculateUV(start, end);
model.addAttribute("uvResult",uv);
model.addAttribute("uvStartDate",start);
model.addAttribute("uvEndDate",end);
return "forward:/data";
}
// Statistics website UV
@RequestMapping(value = "/data/dau", method = RequestMethod.POST)
public String getDAU(@DateTimeFormat(pattern = "yyyy-MM-dd") Date start,
@DateTimeFormat(pattern = "yyyy-MM-dd") Date end,
Model model) {
long dau = dataService.calculateDAU(start, end);
model.addAttribute("dauResult",dau);
model.addAttribute("dauStartDate",start);
model.addAttribute("dauEndDate",end);
return "forward:/data";
}
}
- Process data under admin package HTML page
<!-- content -->
<div class="main">
<!-- website UV -->
<div class="container pl-5 pr-5 pt-3 pb-3 mt-3">
<h6 class="mt-3"><b class="square"></b> website UV</h6>
<form class="form-inline mt-3" method="post" th:action="@{/data/uv}">
<input type="date" class="form-control" required name="start" th:value="${#dates.format(uvStartDate,'yyyy-MM-dd')}"/>
<input type="date" class="form-control ml-3" required name="end" th:value="${#dates.format(uvEndDate,'yyyy-MM-dd')}"/>
<button type="submit" class="btn btn-primary ml-3">Start statistics</button>
</form>
<ul class="list-group mt-3 mb-3">
<li class="list-group-item d-flex justify-content-between align-items-center">
Statistical results
<span class="badge badge-primary badge-danger font-size-14" th:text="${uvResult}">0</span>
</li>
</ul>
</div>
<!-- Active user -->
<div class="container pl-5 pr-5 pt-3 pb-3 mt-4">
<h6 class="mt-3"><b class="square"></b> Active user</h6>
<form class="form-inline mt-3" th:action="@{/data/dau}" method="post">
<input type="date" class="form-control" required name="start" th:value="${#dates.format(dauStartDate,'yyyy-MM-dd')}"/>
<input type="date" class="form-control ml-3" required name="end" th:value="${#dates.format(dauEndDate,'yyyy-MM-dd')}"/>
<button type="submit" class="btn btn-primary ml-3">Start statistics</button>
</form>
<ul class="list-group mt-3 mb-3">
<li class="list-group-item d-flex justify-content-between align-items-center">
Statistical results
<span class="badge badge-primary badge-danger font-size-14" th:text="${dauResult}">0</span>
</li>
</ul>
</div>
</div>
- Configure permissions
Our data view page also needs certain permissions to open, so we need to manage permissions. If permissions are not in place and cannot be accessed, the default is that the administrator has permissions
.antMatchers(
"/discuss/delete",
"/data/**" // Website data statistics
)
.hasAnyAuthority(
AUTHORITY_ADMIN
)
6. Task execution and scheduling
- JDK thread pool
- ExecutorService
- ScheduledExecutorService
- Spring thread pool
- ThreadPoolTaskExecutor
- ThreadPoolTaskScheduler
- Distributed timed task
- Spring Quartz
- Spring Quartz
6.1 introduction to several types of thread pools
The following four thread pools will have problems in a distributed environment. Because both servers execute Scheduler scheduled tasks every x minutes, conflicts are easy to occur. Even if there is no conflict, it should not be executed twice, but only once. The data related to the scheduled task is stored in the memory of the server, and multiple servers store multiple copies of data.
- JDK thread pool
ExcecutorService
ScheduledExecutorService - Spring thread pool
ThreadPoolTaskExecutor
ThreadPoolTaskScheduler
Using Quartz to implement Scheduler timing tasks under distributed conditions, there is no problem. Because the related data of scheduled tasks are stored in the same database, there is only one data of scheduled tasks. If scheduled tasks are executed at the same time, the database will be locked to allow multiple servers to queue for access without conflict. After a server accesses the data, it can change the data to "completed". Then the incoming Quartz will not complete the task again when it sees that the task has been completed.
Distributed timed task
- Spring Quartz: website
6.2 testing of JDK thread pool and Spring thread pool
The following configurations are related to Spring thread pool
In application Properties file
# TaskExecutionProperties # For browser access, with this ThreadPoolTaskExecutor, how many browser accesses cannot be predicted. spring.task.execution.pool.core-size=5 spring.task.execution.pool.max-size=15 spring.task.execution.pool.queue-capacity=100 # TaskSchedulingProperties # The access of the server. With this ThreadPoolTaskScheduler, how often the server executes tasks, what tasks to execute, and how many threads to use can be predicted. There is no need to configure core size, max size and so on. Just write directly with a few threads. spring.task.scheduling.pool.size=5
A ThreadPoolConfig is created in the config package
package com.ateam.community.config;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.annotation.EnableScheduling;
@Configuration
@EnableScheduling
@EnableAsync//This parameter is for alphaservice The @ Async annotation on the execute1() method in Java takes effect
public class ThreadPoolConfig {//Without this configuration class (the name of the configuration class doesn't matter) and the @ enableshcheduling annotation, the ThreadPoolTaskScheduler can't inject (@ Autowired), that is, it can't initialize and can't get an object of ThreadPoolTaskScheduler
}
In alphaservice Java, create a new method
@Service
public class AlphaService {
// Let the method be called asynchronously in a multithreaded environment. That is, the method and the main thread execute concurrently
@Async
public void execute1() {
logger.debug("execute1");
}
/*@Scheduled(initialDelay = 10000, fixedRate = 1000)*/ //The default units of these two parameters are milliseconds
public void execute2() {
logger.debug("execute2");
}
}
Under the test package, create a new test class ThreadPoolTests
package com.ateam.community;
import com.ateam.community.service.AlphaService;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.boot.test.context.SpringBootTest;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import org.springframework.scheduling.concurrent.ThreadPoolTaskScheduler;
import org.springframework.test.context.ContextConfiguration;
import org.springframework.test.context.junit4.SpringRunner;
import org.springframework.web.bind.annotation.ResponseStatus;
import java.util.Date;
import java.util.concurrent.*;
@RunWith(SpringRunner.class)
@SpringBootTest
@ContextConfiguration(classes = CommunityApplication.class)
public class ThreadPoolTests {
private static final Logger logger = LoggerFactory.getLogger(ThreadPoolTests.class);
// JDK common thread pool
private ExecutorService executorService = Executors.newFixedThreadPool(5);
// JDK thread pool that can execute scheduled tasks
private ScheduledExecutorService scheduledExecutorService = Executors.newScheduledThreadPool(5);
// Thread pool for spirng
// ordinary
@Autowired
private ThreadPoolTaskExecutor taskExecutor;
// timing
@Autowired
private ThreadPoolTaskScheduler taskScheduler;
@Autowired
private AlphaService alphaService;
// Regular sleep
private void sleep(long m) {
try {
Thread.sleep(m);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
// 1.JDK common thread pool
@Test
public void testExecutorsService(){
Runnable task = new Runnable() {
@Override
public void run() {
logger.debug("Hello,ExecutorService");
}
};
for (int i = 0; i < 10 ; i++) {
executorService.submit(task);
}
sleep(10000);
}
// 2.JDK timed task thread pool
@Test
public void testScheduledExecutorService(){
Runnable task = new Runnable() {
@Override
public void run() {
logger.debug("hello,ScheduledExecutorService");
}
};
scheduledExecutorService.scheduleAtFixedRate(task,10000,1000, TimeUnit.MILLISECONDS);
sleep(300000);
}
// spring common thread pool
//In application Configuring TaskExecutionProperties in properties will make Spring common thread pool more flexible than JDK common thread pool
@Test
public void testThreadPoolTaskExecutor(){
Runnable task = new Runnable() {
@Override
public void run() {
logger.debug("hello,ScheduledExecutorService");
}
};
for (int i = 0; i < 10; i++) {
taskExecutor.submit(task);
}
sleep(10000);
}
// Thread pool for spring scheduled tasks
//In application Configure TaskSchedulingProperties in properties
@Test
public void testThreadPoolTaskScheduler(){
Runnable task = new Runnable() {
@Override
public void run() {
logger.debug("hello,ScheduledExecutorService");
}
};
Date startTime = new Date(System.currentTimeMillis() + 10000);
taskScheduler.scheduleAtFixedRate(task,startTime,1000);
sleep(300000);
}
//Spring common thread pool (Simplified)
// @Test
// public void testThreadPoolTaskExecutorSimple(){
//
//
// for (int i = 0; i < 20; i++) {
// alphaService.execute1();
// }
// sleep(10000);
// }
// Spring timed task thread pool (Simplified)
//Once executed, the alphaservice will be automatically dropped execute2()
@Test
public void testThreadPoolTaskSchedulerSimple(){
sleep(30000);
}
}
6.3 testing of distributed timed tasks
Distributed scheduled tasks - Spring Quartz
Import Quartz tables into the community database_ mysql_ innodb. sql.
Several interfaces of Spring Quartz:
Scheduler interface: Quartz core scheduling tool. All tasks scheduled by Quartz are called through this interface, and we do not need to read or write.
Job interface: defines a task. The execute() method specifies what to do.
JobDetai interface: configure Job, name, group, description and other configuration information.
Trigger interface: configure when a Job runs and how often it runs repeatedly.
Summary: the Job interface defines a task and configures the Job through JobDetail and Trigger interfaces. After configuration, when the program starts, Quartz will read the configuration information and immediately save the information it reads to the database and those tables. Later, the task is executed by reading the table. Once the data is initialized to the database, the configuration of JobDetail and Trigger will no longer be used. That is, the configuration of JobDetail and Trigger is only used for the first time.
Important tables:
- Introduce dependency
<!-- quartz-->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-quartz</artifactId>
</dependency>
- Create a new package quartz
Under the quartz package, create a new class AlphaJob
package com.ateam.community.quartz;
import org.quartz.Job;
import org.quartz.JobExecutionContext;
import org.quartz.JobExecutionException;
public class AlphaJob implements Job {
@Override
public void execute(JobExecutionContext context) throws JobExecutionException {
System.out.println(Thread.currentThread().getName() + ": execute a quartz job.");
}
}
- Configure Quartz
Under the config package, create a new QuartzConfig class
package com.ateam.community.config;
import com.ateam.community.quartz.AlphaJob;
import com.ateam.community.quartz.PostScoreRefreshJob;
import org.quartz.JobDataMap;
import org.quartz.JobDetail;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.quartz.JobDetailFactoryBean;
import org.springframework.scheduling.quartz.SimpleTriggerFactoryBean;
// Configuration - > Database - > call
//This configuration is only read for the first time, and the information is initialized to the database. In the future, Quartz will access the database to get this information, and will no longer access this configuration class. The premise is to configure application QuartzProperties of properties. If there is no configuration, these configurations are stored in memory, not in the database.
@Configuration
public class QuartzConfig {
//Factorybeans simplify the instantiation of beans:
//1. Encapsulate the Bean instantiation process through FactoryBean
//2. Assemble FactoryBean into Spring container
//3. Inject FactoryBean into other beans
//4. The Bean gets the object instance managed by the FactoryBean
// Configure JobDetail
// @Bean
public JobDetailFactoryBean alphaJobDetail() {
JobDetailFactoryBean factoryBean = new JobDetailFactoryBean();
factoryBean.setJobClass(AlphaJob.class);
factoryBean.setName("alphaJob");//Don't duplicate the names of other tasks.
factoryBean.setGroup("alphaJobGroup");//Multiple tasks can belong to the same group
factoryBean.setDurability(true);//Is the task persistent? true means that even if the task is not needed, its trigger is gone, and the task does not need to be deleted. It should be saved all the time
factoryBean.setRequestsRecovery(true);//Is the task recoverable.
return factoryBean;
}
// Configure Trigger(SimpleTriggerFactoryBean,CronTriggerFactoryBean)
//SimpleTriggerFactoryBean can handle the simple scenario of triggering every 10 minutes. CronTriggerFactoryBean can trigger every Friday at 10 p.m. in this complex scenario, cron expression.
// @Bean
public SimpleTriggerFactoryBean alphaTrigger(JobDetail alphaJobDetail) {//The variable name of parameter JobDetail alphaJobDetail must be consistent with the function name of JobDetailFactoryBean
SimpleTriggerFactoryBean factoryBean = new SimpleTriggerFactoryBean();
factoryBean.setJobDetail(alphaJobDetail);
factoryBean.setName("alphaTrigger");er Take a name
factoryBean.setGroup("alphaTriggerGroup");
factoryBean.setRepeatInterval(3000);//3000 Ms = 3 seconds
factoryBean.setJobDataMap(new JobDataMap());//The bottom layer of Trigger needs to store some Job states. You need to specify which object you use to store them. The default type "new JobDataMap()" is specified here
return factoryBean;
}
}
In application Properties to configure Quartz
# If this is not configured, Quartz will also work, because Spring has made a default configuration for it. Make it read the JobDetail and Trigger configured in QuartzConfig. # However, if these are not configured, Quartz will read data from memory (we configured) instead of from the database, which will cause problems in the distributed runtime # QuartzProperties spring.quartz.job-store-type=jdbc spring.quartz.scheduler-name=communityScheduler spring.quartz.properties.org.quartz.scheduler.instanceId=AUTO spring.quartz.properties.org.quartz.jobStore.class=org.quartz.impl.jdbcjobstore.JobStoreTX spring.quartz.properties.org.quartz.jobStore.driverDelegateClass=org.quartz.impl.jdbcjobstore.StdJDBCDelegate spring.quartz.properties.org.quartz.jobStore.isClustered=true spring.quartz.properties.org.quartz.threadPool.class=org.quartz.simpl.SimpleThreadPool spring.quartz.properties.org.quartz.threadPool.threadCount=5
- Under the test package, create a new test class QuartzTests
@RunWith(SpringRunner.class)
@SpringBootTest
@ContextConfiguration(classes = CommunityApplication.class)//Configuration class
public class QuartzTests {
@Autowired
private Scheduler scheduler;
@Test
public void testDeleteJob(){
try {
boolean b = scheduler.deleteJob(new JobKey("alphaJob", "alphaJobGroup"));//Delete a Job, that is, delete the Job related data in the database. new JobKey(Job name, Job group name). These two parameters uniquely determine a Job
System.out.println(b);
} catch (SchedulerException e) {
e.printStackTrace();
}
}
}
7. Hot post ranking
How to calculate the score of a post:
For general posts, the longer the time, the lower the quantitative score, while the more praise points and replies, the higher the quantitative score. Generally, a log is made for the increased score of the like reply to increase the impact of the like reply just released. With the passage of time, the negative effect of time is reflected and the score decreases, which is similar to the actual situation.
There are 2 things to do in this section:
- When posting, praising, commenting and refining, put the post id into the Set set of Redis, then Set a scheduled task, take out these post IDS one by one every 5 minutes, recalculate their score, and then update the score value of discusspost in the database and elasticsearch.
- selectDiscussPosts() and its corresponding line (Mapper, Dao, Service, Controller, thymeleaf pages) are modified, and the parameter int orderMode is added. Then, you can sort according to the popularity of posts. You can toggle sorting with "latest / heat"
7.1 data layer changes
In RedisUtil class, add the method to obtain the key
public class RedisKeyUtil {
private static final String SPLIT = ":";
private static final String PREFIX_POST = "post";
// Post score
public static String getPostScoreKey() {
return PREFIX_POST + SPLIT + "score";
}
}
In the DiscussPostMapper class under the dao package, modify the selectDiscussPosts method to increase the method of updating scores
//orderMode is 0 by default, which means that it is arranged according to the time sequence, and the latest one is in the front. When orderMode is 1, it means to arrange according to the heat, that is, according to the score of the post.
List<DiscussPost> selectDiscussPosts(int userId, int offset, int limit, int orderMode);
int updateScore(int id, double score);
Under the mapper package, discuss map XML, modify the corresponding SQL
<select id="selectDiscussPosts" resultType="com.ateam.community.entity.DiscussPost">
select <include refid="selectFields"></include>
from discuss_post
where status != 2
<if test="userId != 0">
and user_id = #{userId}
</if>
<if test="orderMode==0">
order by discuss_type desc, create_time desc
</if>
<if test="orderMode==1">
order by discuss_type desc, score desc, create_time desc
</if>
limit #{offset}, #{limit}
</select>
<update id="updateScore">
update discuss_post
set score = #{score}
where id = #{id}
</update>
7.2 regularly calculate post scores
- Posts (posts should have an initial score, and the newer the post, the higher the score), likes, comments and refinements should be recalculated.
In discusspostcontroller Java, refinement, post, method, modify the code
// Refining
@RequestMapping(value = "/wonderful", method = RequestMethod.POST)
@ResponseBody
public String setWonderful(int id) {
discussPostService.updateStatus(id,1);
// Trigger posting event
Event event = new Event()
.setTopic(TOPIC_PUBLISH)
.setUserId(hostHolder.getUser().getId())
.setEntityType(ENTITY_TYPE_POST)
.setEntityId(id);
eventProducer.fireEvent(event);
// Calculate post score
String redisKey = RedisKeyUtil.getPostScoreKey();
redisTemplate.opsForSet().add(redisKey,id);
return CommunityUtil.getJSONString(0);
}
@RequestMapping(value = "/add", method = RequestMethod.POST)
@ResponseBody
public String addDiscussPost(String title, String content) {
User user = hostHolder.getUser();
if (user == null) {
return CommunityUtil.getJSONString(403,"You haven't logged in yet!");
}
DiscussPost post = new DiscussPost();
post.setUserId(user.getId());
post.setTitle(title);
post.setContent(content);
post.setCreateTime(new Date());
discussPostService.addDiscussPost(post);
// Trigger posting event
Event event = new Event()
.setTopic(TOPIC_PUBLISH)
.setUserId(user.getId())
.setEntityType(ENTITY_TYPE_POST)
.setEntityId(post.getId());
eventProducer.fireEvent(event);
// Calculate post score
String redisKey = RedisKeyUtil.getPostScoreKey();
redisTemplate.opsForSet().add(redisKey,post.getId());
// In case of wrong amount, it will be processed in the future
return CommunityUtil.getJSONString(0,"Published successfully");
}
In commentcontrol and LikeController, judgmental comment posts and praise posts can only be modified. Here, take praise as an example
2. Under the quartz package, create a new class PostScoreRefreshJob to refresh the post scores regularly
package com.ateam.community.quartz;
import com.ateam.community.entity.DiscussPost;
import com.ateam.community.service.DiscussPostService;
import com.ateam.community.service.ElasticsearchService;
import com.ateam.community.service.LikeService;
import com.ateam.community.util.CommunityConstant;
import com.ateam.community.util.RedisKeyUtil;
import org.quartz.Job;
import org.quartz.JobExecutionContext;
import org.quartz.JobExecutionException;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.data.redis.core.BoundSetOperations;
import org.springframework.data.redis.core.RedisTemplate;
import java.text.ParseException;
import java.text.SimpleDateFormat;
import java.util.Date;
public class PostScoreRefreshJob implements Job, CommunityConstant {
private static final Logger logger = LoggerFactory.getLogger(PostScoreRefreshJob.class);
@Autowired
private RedisTemplate redisTemplate;
@Autowired
private DiscussPostService discussPostService;
@Autowired
private LikeService likeService;
@Autowired
private ElasticsearchService elasticsearchService;
// ATeam community Era
private static final Date epoch;
static {
try {
epoch = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss").parse("2014-09-01 00:00:00");
} catch (ParseException e) {
throw new RuntimeException("Preliminary trial ATeam Era failed!" + e.getMessage());
}
}
@Override
public void execute(JobExecutionContext context) throws JobExecutionException {
String redisKey = RedisKeyUtil.getPostScoreKey();
BoundSetOperations operations = redisTemplate.boundSetOps(redisKey);
if (operations.size() == 0) {
logger.info("[Task cancellation] There are no posts to brush!");
return;
}
logger.info("[Task start] Brushing post score:" + operations.size());
while (operations.size() > 0) {
this.refresh((Integer) operations.pop());
}
logger.info("[End of task] The post score has been brushed");
}
private void refresh(int postId) {
DiscussPost post = discussPostService.findDiscussPostById(postId);
if (post == null) {
logger.error("The post does not exist: id = " + postId);
return;
}
// Is it the essence?
boolean wonderful = post.getStatus() == 1;
// Number of comments
int commentCount = post.getCommentCount();
// Number of likes
long likeCount = likeService.findEntityLikeCount(ENTITY_TYPE_POST,postId);
// Calculate weight
double w = (wonderful ? 75 : 0) + commentCount * 10 + likeCount * 2;
// Score = post weight + distance days
double score = Math.log10(Math.max(w,1))
+ (post.getCreateTime().getTime() - epoch.getTime()) / (1000 * 3600 * 24);
//Update post score
discussPostService.updateScore(postId,score);
//Synchronize search data
post.setScore(score);
elasticsearchService.saveDiscussPost(post);
}
}
- In the QuartzConfig class under the config package, configure the scheduled task just now
package com.ateam.community.config;
import com.ateam.community.quartz.AlphaJob;
import com.ateam.community.quartz.PostScoreRefreshJob;
import org.quartz.JobDataMap;
import org.quartz.JobDetail;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.quartz.JobDetailFactoryBean;
import org.springframework.scheduling.quartz.SimpleTriggerFactoryBean;
// Configuration - > Database - > call
@Configuration
public class QuartzConfig {
//FactoryBean can be simplified to the instantiation process of Bean:
//1. Encapsulate the Bean instantiation process through FactoryBean
//2. Assemble FactoryBean into Spring container
//3. Inject FactoryBean into other beans
//4. The Bean gets the object instance managed by the FactoryBean
// Task of refreshing post scores
// Configure JobDetail
@Bean
public JobDetailFactoryBean postScoreRefreshJobDetail() {
JobDetailFactoryBean factoryBean = new JobDetailFactoryBean();
factoryBean.setJobClass(PostScoreRefreshJob.class);
factoryBean.setName("postScoreRefreshJob");
factoryBean.setGroup("communityJobGroup");
factoryBean.setDurability(true);
factoryBean.setRequestsRecovery(true);
return factoryBean;
}
// Configure Trigger(SimpleTriggerFactoryBean,CronTriggerFactoryBean)
@Bean
public SimpleTriggerFactoryBean postScoreRefreshTrigger(JobDetail postScoreRefreshJobDetail) {
SimpleTriggerFactoryBean factoryBean = new SimpleTriggerFactoryBean();
factoryBean.setJobDetail(postScoreRefreshJobDetail);
factoryBean.setName("postScoreRefreshTrigger");
factoryBean.setGroup("communityTriggerGroup");
factoryBean.setRepeatInterval(1000 * 60 * 60); // Refresh once every 1h
factoryBean.setJobDataMap(new JobDataMap());
return factoryBean;
}
}
- Modify the code that uses the selectDiscussPosts method
Find the places where the selectDiscussPosts method has been used, and then modify them one by one.
- Latest / hottest
Go to HomeController first and then index html. When refreshing the page, there is no orderMode parameter, so you need to set the orderMode to 0 by default in the Controller, and then pass it to the thymeleaf page to display "latest".
@RequestMapping(value = "/index", method = RequestMethod.GET)
public String getIndexPage(Model model, Page page,
@RequestParam(name = "orderMode", defaultValue = "0") int orderMode){
// Before method invocation, spring MVC will automatically instantiate the Model and Page and inject the Page into the Model
// Therefore, the data in the Page object can be accessed directly in thymeleaf
page.setRows(discussPostService.findDiscussPostRows(0));
page.setPath("/index?orderMode=" + orderMode);
List<DiscussPost> list = discussPostService.findDiscussPosts(0, page.getOffset(), page.getLimit(),orderMode);
ArrayList<Map<String, Object>> discussPosts = new ArrayList<>();
if (list != null){
for (DiscussPost post : list){
HashMap<String, Object> map = new HashMap<>();
map.put("post",post);
User user = userService.findUserById(post.getUserId());
map.put("user",user);
// Get the number of likes per post
long likeCount = likeService.findEntityLikeCount(ENTITY_TYPE_POST,post.getId());
map.put("likeCount", likeCount);
discussPosts.add(map);
}
}
model.addAttribute("discussPosts",discussPosts);
model.addAttribute("orderMode",orderMode);
return "/index";
}
index.html
8. Student growth chart
- wkhtmltopdf
- wkhtmltopdf url file
- wkhtmltoimage url file
- java
- Runtime.getRuntime().exec()
- Runtime.getRuntime().exec()
8.1 wkhtmltopdf
wkhtmltopdf directory structure:( wkhtmltopdf website)
Configure environment variables
- Command line use wk
# Actually, we should execute the command of wkhtmltopdf in the path "D:\wkhtmltopdf\bin", because we have configured the path in the environment variable, so we can enter the command anywhere. # Turn the web page into pdf and save it in the folder. The folder will not be generated automatically. You need to create it manually. The generated pdf file needs to be named by yourself. C:\Users\dell>wkhtmltopdf https://www.nowcoder.com E:/data/wk/wk-pdfs/1.pdf # Turn web pages into pictures C:\Users\dell>wkhtmltoimage https://www.nowcoder.com E: \ coding related software \ wkhtmltopdf \ my data \ wk images \ e: / data / wk / wk images / 1 png C:\Users\dell>wkhtmltoimage --quality 75 https://www.nowcoder.com E:/data/wk/wk-images/2.png # -- quality 75 means that the picture is compressed to 75% of the original quality. This is to reduce the space occupied by the picture (MB)
- Using java to call wk
Runtime.getRuntime.exec()
Under the test package, create a new class WKTests
public class WKTests {
public static void main(String[] args){
String cmd = "D:\\wkhtmltopdf\\bin\\wkhtmltoimage --quality 75 http://localhost:8080/community/index E:\\data\\wk\\wk-images\\4.png";
try {
//When the Runtime executes a command, it just submits the command to the local operating system, and the rest is executed by the operating system. Java will not wait for the operating system. Java will directly execute the next line. So it will output ok first and then generate pictures.
//That is, the main function and the generated picture are asynchronous and concurrent
Runtime.getRuntime().exec(cmd);
System.out.println("ok.");
} catch (IOException e) {
e.printStackTrace();
}
}
}
8.2 project realization growth chart
Functions realized:
- Save the growth chart generated by the web page (according to the url) locally: http://localhost:8080/community/share?htmlUrl=https://www.nowcoder.com
- Display the local image on the web page through a url: http://localhost:8080/community/share/image/ Picture name without suffix (i.e. UUID)
wk related configurations:
application.properties
# wk # Web page transfer pdf/picture #These two are our customized configurations. Because these two paths will be different before and after going online, they should be made into configurable paths #After going online, the installation path of wkhtmltoimage command of wkhtmltopdf software wk.image.command=D:/wkhtmltopdf/bin/wkhtmltoimage #After going online, the storage location of the pictures generated by wkhtmltopdf software wk.image.storage=E:/data/wk/wk-images
Under the config package, create a new class WKConfig
//The @ Configuration here is not for Configuration, but for initializing the class as Bean when the program starts to execute, and then for executing the init() method at the beginning of the program
@Configuration
public class WKConfig {
private static final Logger logger = LoggerFactory.getLogger(WKConfig.class);
@Value("${wk.image.storage}")
private String wkImageStorage;
//Spring's @ PostConstruct annotation is on the method, which means that this method is executed immediately after spring instantiates the Bean, and then other beans will be instantiated. There can be multiple @ PostConstruct annotation methods in a Bean.
@PostConstruct
public void init() {
// Create WK picture directory
File file = new File(wkImageStorage);
if (!file.exists()) {
file.mkdir();
logger.info("establish WK Picture Directory:" + wkImageStorage);
}
}
}
Under the controller package, create a new ShareController class
@Controller
public class ShareController implements CommunityConstant {
private static final Logger logger = LoggerFactory.getLogger(ShareController.class);
@Autowired
private EventProducer eventProducer;
@Value("${community.path.domain}")
private String domain;
@Value("${server.servlet.context-path}")
private String contextPath;
@Value("${wk.image.storage}")
private String wkImageStorage;
@RequestMapping(value = "share", method = RequestMethod.GET)
@ResponseBody
public String share(String htmlUrl) {
// file name
String fileName = CommunityUtil.generateUUID();
// event
// Asynchronous growth chart
Event event = new Event()
.setTopic(TOPIC_SHARE)
.setData("htmlUrl",htmlUrl)
.setData("fileName",fileName)
.setData("suffix",".png");
eventProducer.fireEvent(event);
Map<String, Object> map = new HashMap<>();
map.put("shareUrl", domain + contextPath + "/share/image/" + fileName);
return CommunityUtil.getJSONString(0,null, map);
}
// Get long graph
@RequestMapping(value = "/share/image/{fileName}", method = RequestMethod.GET)
public void getShareImage(@PathVariable("fileName") String fileName, HttpServletResponse response) {
if (StringUtils.isBlank(fileName)) {
throw new IllegalArgumentException("File name cannot be empty!");
}
response.setContentType("image/png");
File file = new File(wkImageStorage + "/" + fileName + ".png");
try (ServletOutputStream os = response.getOutputStream();
FileInputStream fis = new FileInputStream(file)
) {
byte[] buffer = new byte[1024];
int len = 0;
while ((len = fis.read(buffer)) != -1) {
os.write(buffer,0,len);
}
} catch (IOException e) {
logger.error("Failed to get picture:" + e.getMessage());
}
}
}
In EventConsumer class, add a consumption sharing event
@Component
public class EventConsumer implements CommunityConstant {
private static final Logger logger = LoggerFactory.getLogger(Event.class);
@Autowired
private MessageService messageService;
@Autowired
private DiscussPostService discussPostService;
@Autowired
private ElasticsearchService elasticsearchService;
@Value("${wk.image.storage}")
private String wkImageStorage;
@Value("${wk.image.command}")
private String wkImageCommand;
// Consumption sharing event
@KafkaListener(topics = {TOPIC_SHARE})
public void handleShareMessage(ConsumerRecord record) {
if (record == null || record.value() == null) {
logger.error("The content of the message is empty!");
return;
}
// Using fastjson to convert json string into Event object
Event event = JSONObject.parseObject(record.value().toString(), Event.class);
if (event == null) {
logger.error("Message format error!");
return;
}
String htmlUrl = (String) event.getData().get("htmlUrl");
String fileName = (String) event.getData().get("fileName");
String suffix = (String) event.getData().get("suffix");
String cmd = wkImageCommand + " --quality 75 "
+ htmlUrl + " " + wkImageStorage + "/" + fileName + suffix;
try {
Runtime.getRuntime().exec(cmd);
logger.info("Student growth chart success:" + cmd);
} catch (IOException e) {
logger.error("Failed to generate growth chart:" + e.getMessage());
}
}
}
9. Optimize website performance
Only local cache and DB (database). If it is the data of popular posts, both servers take out popular posts from the database and update them to the local cache. One copy of the same data is saved on both servers, which is no problem. If it is a user related problem, such as user login credentials, the user is in login status on server 1, and there is no such credential in the local cache on server 2, the user will not be in login status, which is not enough.
In this case, Redis can be used to solve the problem. Both servers obtain the login status of users from Redis.
The local cache space is small, and the Redis cache space is large. Most requests will be intercepted by these two levels.
If there is no requested data in the local cache and Redis, the data will be obtained from the database and sent to the app (service component), and then the data obtained from db will be updated from the app to the local cache and Redis.
The data elimination mechanism of cache is based on time and usage frequency.
We only use caching for data that changes less frequently. If it is the latest post, it cannot be cached, and the update is too fast, so this class will cache the hottest ranking of posts, because this ranking updates the score only once in a period of time, so the popular ranking of Posts remains unchanged between the two updated scores.
- Import dependency
<!-- caffeine-->
<dependency>
<groupId>com.github.ben-manes.caffeine</groupId>
<artifactId>caffeine</artifactId>
<version>2.7.0</version>
</dependency>
- caffeine configuration
# caffeine #Are custom configurations # Posts means posts. If you want to cache comments, you can caffeine comments # The first one means to cache 15 pages of posts, and the second one means that the data stored in the cache will be cleaned up automatically after 3 minutes, which is called automatic elimination. There is also an active elimination. If the post is updated, the post in the cache will be eliminated. # There is only automatic elimination, not active elimination, because we cache page by page. If a post is updated, it is inappropriate to brush out all posts on this page # In other words, the number of posts, comments and likes on this page will be delayed in these three minutes, which is not in line with the real number, but it will not affect the use. caffeine.posts.max-size=15 caffeine.posts.expire-seconds=180
- Test caffeine
Under the test package, create a new test class CaffeineTests
@RunWith(SpringRunner.class)
@SpringBootTest
@ContextConfiguration(classes = CommunityApplication.class)//Configuration class
public class CaffeineTests {
@Autowired
private DiscussPostService discussPostService;
@Test
public void initDataForTest() {
for (int i = 0; i < 300000; i++) {
DiscussPost post = new DiscussPost();
post.setUserId(111);
post.setTitle("Internet job search warm spring program");
post.setContent("The employment situation this year is indeed not optimistic. After a year, it was like diving, and the whole discussion area was full of sadness! Is it true that no one wants it in the 19th session?! Is there really no way out for the 18th session to be optimized?! Everybody's“wail mournfully”And“Tragic experience”It affected the hearts of the young brothers and sisters of Niuke who lurked in the discussion area every day, so Niuke decided: it's time to do something for everyone! To help everyone through“severe winter”，Niuke special union 60+Enterprises, open the internet job search warm spring program, facing the 18th&19 Session, save 0 offer!");
post.setCreateTime(new Date());
post.setScore(Math.random() * 2000);
discussPostService.addDiscussPost(post);
}
}
@Test
public void testCache(){
System.out.println(discussPostService.findDiscussPosts(0,0,10,1));
System.out.println(discussPostService.findDiscussPosts(0,0,10,1));
System.out.println(discussPostService.findDiscussPosts(0,0,10,1));
System.out.println(discussPostService.findDiscussPosts(0,0,10,0));
}
}
- Using caffeine in the project
Under the service package, modify the relevant code in the DiscussPostService class
@Service
public class DiscussPostService {
private static final Logger logger = LoggerFactory.getLogger(DiscussPostService.class);
@Resource
private DiscussPostMapper discussPostMapper;
@Autowired
private SensitiveFilter sensitiveFilter;
@Value("${caffeine.posts.max-size}")
private int maxSize;
@Value("${caffeine.posts.expire-seconds}")
private int expireSeconds;
// Caffeine core interfaces: Cache, LoadingCache, AsyncLoadingCache
// LoadingCache: synchronize the cache. If there is no cache, the threads to be read queue up. Caffeine fetches the data into the cache, and then reads the data in the cache one by one. We use this.
// AsyncLoadingCache: asynchronous cache, which supports simultaneous reading of the same data by multiple threads
// value is cached according to key
// Post list cache
private LoadingCache<String, List<DiscussPost>> postListCache;
// Total number of Posts cache
private LoadingCache<Integer, Integer> postRowsCache;
@PostConstruct
private void init() {
// Preliminary post list caching
postListCache = Caffeine.newBuilder()
.maximumSize(maxSize)
.expireAfterAccess(expireSeconds, TimeUnit.SECONDS)
.build(new CacheLoader<String, List<DiscussPost>>() {
//When trying to fetch data from the cache, caffeine will check whether there is data in the cache, return if there is, and call the following load() method to fetch the data from the database
//So the load() method tells caffeine how to get the data from the database
@Override
public @Nullable List<DiscussPost> load(@NonNull String key) throws Exception {
if (key.length() == 0) {
throw new IllegalArgumentException("Parameter error");
}
String[] params = key.split(":");
if (params == null || params.length != 2) {
throw new IllegalArgumentException("Parameter error");
}
int offset = Integer.parseInt(params[0]);
int limit = Integer.parseInt(params[1]);
// L2 cache: redis - > MySQL
logger.debug("load post list from DB");
return discussPostMapper.selectDiscussPosts(0,offset,limit,1);
}
});
// Total number of initial Posts cache
postRowsCache = Caffeine.newBuilder()
//This should have been separately in application Configured in properties,
//Such as caffeine posts-count. Max size and caffeine posts-count. expire-seconds. But the teacher is lazy to match it alone, and there is no error in reusing cached posts, so this is the case.
.maximumSize(maxSize)
.expireAfterAccess(expireSeconds, TimeUnit.SECONDS)
.build(new CacheLoader<Integer, Integer>() {
@Override
public @Nullable Integer load(@NonNull Integer key) throws Exception {
logger.debug("load post rows from DB");
return discussPostMapper.selectDiscussPostRows(key);
}
});
}
public List<DiscussPost> findDiscussPosts(int userId, int offset, int limit, int orderMode) {
if (userId == 0 && orderMode == 1) {
//If the userId and orderMode are certain, then the remaining two changed quantities can be combined into a key, which can be separated by anything, such as a colon
return postListCache.get(offset + ":" + limit);
}
logger.debug("load post list from DB");
return discussPostMapper.selectDiscussPosts(userId, offset, limit,orderMode);
}
public int findDiscussPostRows(int userId) {
if (userId == 0) {
//In fact, there is no need to use userId as the key, because the userId here is always 0, but there must be a key, so you can only use 0 as the key all the time.
return postRowsCache.get(userId);
}
logger.debug("load post rows from DB");
return discussPostMapper.selectDiscussPostRows(userId);
}
}
- performance testing
You can use the stress test tool to simulate multiple users accessing the server at the same time for testing.
Jmeter is recommended: Apache JMeter - Download Apache JMeter website