1, Foreword
Two days ago, we did an import function. The import started very slowly. It took more than one minute to import 2w pieces of data. Later, we optimized it bit by bit, from directly linking the list into Mysql, assigning the list into Mysql, and multi threading the list into Mysql. Time is getting less and less. It was very cool, and finally it became within 10s. Let's show the process.
2, Connect the list directly to Mysql
Batch import using mybatis:
@Transactional(rollbackFor = Exception.class)
public int addFreshStudentsNew2(List<FreshStudentAndStudentModel> list, String schoolNo) {
if (list == null || list.isEmpty()) {
return 0;
}
List<StudentEntity> studentEntityList = new LinkedList<>();
List<EnrollStudentEntity> enrollStudentEntityList = new LinkedList<>();
List<AllusersEntity> allusersEntityList = new LinkedList<>();
for (FreshStudentAndStudentModel freshStudentAndStudentModel : list) {
EnrollStudentEntity enrollStudentEntity = new EnrollStudentEntity();
StudentEntity studentEntity = new StudentEntity();
BeanUtils.copyProperties(freshStudentAndStudentModel, studentEntity);
BeanUtils.copyProperties(freshStudentAndStudentModel, enrollStudentEntity);
String operator = TenancyContext.UserID.get();
String studentId = BaseUuidUtils.base58Uuid();
enrollStudentEntity.setId(BaseUuidUtils.base58Uuid());
enrollStudentEntity.setStudentId(studentId);
enrollStudentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());
enrollStudentEntity.setOperator(operator);
studentEntity.setId(studentId);
studentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());
studentEntity.setOperator(operator);
studentEntityList.add(studentEntity);
enrollStudentEntityList.add(enrollStudentEntity);
AllusersEntity allusersEntity = new AllusersEntity();
allusersEntity.setId(enrollStudentEntity.getId());
allusersEntity.setUserCode(enrollStudentEntity.getNemtCode());
allusersEntity.setUserName(enrollStudentEntity.getName());
allusersEntity.setSchoolNo(schoolNo);
allusersEntity.setTelNum(enrollStudentEntity.getTelNum());
allusersEntity.setPassword(enrollStudentEntity.getNemtCode()); //The password is set to the candidate number
allusersEntityList.add(allusersEntity);
}
enResult = enrollStudentDao.insertAll(enrollStudentEntityList);
stuResult = studentDao.insertAll(studentEntityList);
allResult = allusersFacade.insertUserList(allusersEntityList);
if (enResult > 0 && stuResult > 0 && allResult) {
return 10;
}
return -10;
}
Mapper.xml
<insert id="insertAll" parameterType="com.dmsdbj.itoo.basicInfo.entity.EnrollStudentEntity">
insert into tb_enroll_student
<trim prefix="(" suffix=")" suffixOverrides=",">
id,
remark,
nEMT_aspiration,
nEMT_code,
nEMT_score,
student_id,
identity_card_id,
level,
major,
name,
nation,
secondary_college,
operator,
sex,
is_delete,
account_address,
native_place,
original_place,
used_name,
pictrue,
join_party_date,
political_status,
tel_num,
is_registry,
graduate_school,
create_time,
update_time </trim>
values
<foreach collection="list" item="item" index="index" separator=",">
(
#{item.id,jdbcType=VARCHAR},
#{item.remark,jdbcType=VARCHAR},
#{item.nemtAspiration,jdbcType=VARCHAR},
#{item.nemtCode,jdbcType=VARCHAR},
#{item.nemtScore,jdbcType=VARCHAR},
#{item.studentId,jdbcType=VARCHAR},
#{item.identityCardId,jdbcType=VARCHAR},
#{item.level,jdbcType=VARCHAR},
#{item.major,jdbcType=VARCHAR},
#{item.name,jdbcType=VARCHAR},
#{item.nation,jdbcType=VARCHAR},
#{item.secondaryCollege,jdbcType=VARCHAR},
#{item.operator,jdbcType=VARCHAR},
#{item.sex,jdbcType=VARCHAR},
0,
#{item.accountAddress,jdbcType=VARCHAR},
#{item.nativePlace,jdbcType=VARCHAR},
#{item.originalPlace,jdbcType=VARCHAR},
#{item.usedName,jdbcType=VARCHAR},
#{item.pictrue,jdbcType=VARCHAR},
#{item.joinPartyDate,jdbcType=VARCHAR},
#{item.politicalStatus,jdbcType=VARCHAR},
#{item.telNum,jdbcType=VARCHAR},
#{item.isRegistry,jdbcType=TINYINT},
#{item.graduateSchool,jdbcType=VARCHAR},
now(),
now()
)
</foreach>
</insert>
Code Description:
The underlying mapper is generated through reverse engineering. The batch insertion is as follows: insert into tb_enroll_student()values ()，()…….();
This disadvantage is that the database generally has a default setting, that is, the data of each sql operation cannot exceed 4M. You can change this value on the server by setting the max_ allowed_ Packet 'variable., although we can
Similarly, modify my.ini plus max_allowed_packet =6710886467108864=64M, the default size is 4194304, that is 4m
After the modification, restart the mysql service. If you modify through the command line, you don't need to restart the mysql service.
This operation is completed, but we can't guarantee the maximum size of the project at a time. This has disadvantages. Therefore, group import can be considered.
3, Group and import the list into Mysql
The same applies to mybatis batch insertion. The difference is that each import is calculated in groups, and then imported multiple times:
@Transactional(rollbackFor = Exception.class)
public int addFreshStudentsNew2(List<FreshStudentAndStudentModel> list, String schoolNo) {
if (list == null || list.isEmpty()) {
return 0;
}
List<StudentEntity> studentEntityList = new LinkedList<>();
List<EnrollStudentEntity> enrollStudentEntityList = new LinkedList<>();
List<AllusersEntity> allusersEntityList = new LinkedList<>();
for (FreshStudentAndStudentModel freshStudentAndStudentModel : list) {
EnrollStudentEntity enrollStudentEntity = new EnrollStudentEntity();
StudentEntity studentEntity = new StudentEntity();
BeanUtils.copyProperties(freshStudentAndStudentModel, studentEntity);
BeanUtils.copyProperties(freshStudentAndStudentModel, enrollStudentEntity);
String operator = TenancyContext.UserID.get();
String studentId = BaseUuidUtils.base58Uuid();
enrollStudentEntity.setId(BaseUuidUtils.base58Uuid());
enrollStudentEntity.setStudentId(studentId);
enrollStudentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());
enrollStudentEntity.setOperator(operator);
studentEntity.setId(studentId);
studentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());
studentEntity.setOperator(operator);
studentEntityList.add(studentEntity);
enrollStudentEntityList.add(enrollStudentEntity);
AllusersEntity allusersEntity = new AllusersEntity();
allusersEntity.setId(enrollStudentEntity.getId());
allusersEntity.setUserCode(enrollStudentEntity.getNemtCode());
allusersEntity.setUserName(enrollStudentEntity.getName());
allusersEntity.setSchoolNo(schoolNo);
allusersEntity.setTelNum(enrollStudentEntity.getTelNum());
allusersEntity.setPassword(enrollStudentEntity.getNemtCode()); //The password is set to the candidate number
allusersEntityList.add(allusersEntity);
}
int c = 100;
int b = enrollStudentEntityList.size() / c;
int d = enrollStudentEntityList.size() % c;
int enResult = 0;
int stuResult = 0;
boolean allResult = false;
for (int e = c; e <= c * b; e = e + c) {
enResult = enrollStudentDao.insertAll(enrollStudentEntityList.subList(e - c, e));
stuResult = studentDao.insertAll(studentEntityList.subList(e - c, e));
allResult = allusersFacade.insertUserList(allusersEntityList.subList(e - c, e));
}
if (d != 0) {
enResult = enrollStudentDao.insertAll(enrollStudentEntityList.subList(c * b, enrollStudentEntityList.size()));
stuResult = studentDao.insertAll(studentEntityList.subList(c * b, studentEntityList.size()));
allResult = allusersFacade.insertUserList(allusersEntityList.subList(c * b, allusersEntityList.size()));
}
if (enResult > 0 && stuResult > 0 && allResult) {
return 10;
}
return -10;
}
Code Description:
This operation can avoid the above errors, but inserting multiple times virtually increases the operation practice and is easy to timeout. So this method is not worth advocating.
Improve again, using multithreaded batch import.
4, Multi thread batch import Mysql
The batch import of mybatis is still used. The difference is that it is grouped according to the number of threads, and then a multi-threaded pool is established for import.
@Transactional(rollbackFor = Exception.class)
public int addFreshStudentsNew(List<FreshStudentAndStudentModel> list, String schoolNo) {
if (list == null || list.isEmpty()) {
return 0;
}
List<StudentEntity> studentEntityList = new LinkedList<>();
List<EnrollStudentEntity> enrollStudentEntityList = new LinkedList<>();
List<AllusersEntity> allusersEntityList = new LinkedList<>();
list.forEach(freshStudentAndStudentModel -> {
EnrollStudentEntity enrollStudentEntity = new EnrollStudentEntity();
StudentEntity studentEntity = new StudentEntity();
BeanUtils.copyProperties(freshStudentAndStudentModel, studentEntity);
BeanUtils.copyProperties(freshStudentAndStudentModel, enrollStudentEntity);
String operator = TenancyContext.UserID.get();
String studentId = BaseUuidUtils.base58Uuid();
enrollStudentEntity.setId(BaseUuidUtils.base58Uuid());
enrollStudentEntity.setStudentId(studentId);
enrollStudentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());
enrollStudentEntity.setOperator(operator);
studentEntity.setId(studentId);
studentEntity.setIdentityCardId(freshStudentAndStudentModel.getIdCard());
studentEntity.setOperator(operator);
studentEntityList.add(studentEntity);
enrollStudentEntityList.add(enrollStudentEntity);
AllusersEntity allusersEntity = new AllusersEntity();
allusersEntity.setId(enrollStudentEntity.getId());
allusersEntity.setUserCode(enrollStudentEntity.getNemtCode());
allusersEntity.setUserName(enrollStudentEntity.getName());
allusersEntity.setSchoolNo(schoolNo);
allusersEntity.setTelNum(enrollStudentEntity.getTelNum());
allusersEntity.setPassword(enrollStudentEntity.getNemtCode()); //The password is set to the candidate number
allusersEntityList.add(allusersEntity);
});
int nThreads = 50;
int size = enrollStudentEntityList.size();
ExecutorService executorService = Executors.newFixedThreadPool(nThreads);
List<Future<Integer>> futures = new ArrayList<Future<Integer>>(nThreads);
for (int i = 0; i < nThreads; i++) {
final List<EnrollStudentEntity> EnrollStudentEntityImputList = enrollStudentEntityList.subList(size / nThreads * i, size / nThreads * (i + 1));
final List<StudentEntity> studentEntityImportList = studentEntityList.subList(size / nThreads * i, size / nThreads * (i + 1));
final List<AllusersEntity> allusersEntityImportList = allusersEntityList.subList(size / nThreads * i, size / nThreads * (i + 1));
Callable<Integer> task1 = () -> {
studentSave.saveStudent(EnrollStudentEntityImputList,studentEntityImportList,allusersEntityImportList);
return 1;
};
futures.add(executorService.submit(task1));
}
executorService.shutdown();
if (!futures.isEmpty() && futures != null) {
return 10;
}
return -10;
}
Code Description:
The above is to establish a fixed number of threads by applying ExecutorService, then group them according to the number of threads and import them in batch. On the one hand, it can relieve the pressure on the database. On the other hand, the number of threads on the other side is large, which will improve the running time of the program to a certain extent. The disadvantage is that it depends on the server configuration. If the configuration is good, you can turn on multipoint threads. If the configuration is poor, you can turn it down.
5, Summary
Through the use of this operation, it has been continuously improved, and the project use skills are also good. Come on ~ ~ multithreading~~