System centos7 5 message nighttime alarm prompt: task kworker / 0:1:475 blocked for more than 120 seconds

The details are as follows:

kernel: INFO: task kworker/0:1:47475 blocked for more than 120 seconds.

Jan 9 23:26:02 hblf-ct-dsm009 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 9 23:26:02 hblf-ct-dsm009 kernel: kworker/0:1 D ffff9b57af61acc0 0 47475 2 0x00000080
Jan 9 23:26:02 hblf-ct-dsm009 kernel: Call Trace:
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] ? scsih_qcmd+0x3ab/0x520 [mpt3sas]
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] schedule+0x29/0x70
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] schedule_timeout+0x221/0x2d0
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] ? select_task_rq_fair+0x5a6/0x760
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] ? scsi_request_fn+0x48/0x680
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] wait_for_completion+0xfd/0x140
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] ? wake_up_state+0x20/0x20
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] kthread_create_on_node+0xaa/0x140
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] create_worker+0xeb/0x200
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] manage_workers.isra.26+0xf6/0x2a0
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] worker_thread+0x383/0x3c0
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] ? manage_workers.isra.26+0x2a0/0x2a0
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] kthread+0xd1/0xe0
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] ? insert_kthread_work+0x40/0x40
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] ret_from_fork_nospec_begin+0x21/0x21
Jan 9 23:26:02 hblf-ct-dsm009 kernel: [] ? insert_kthread_work+0x40/0x40
Jan 9 23:26:02 hblf-ct-dsm009 kernel: INFO: task kworker/15:1:50115 blocked for more than 120 seconds.
Jan 9 23:26:02 hblf-ct-dsm009 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Jan 9 23:26:02 hblf-ct-dsm009 kernel: kworker/15:1 D ffff9b5baee5acc0 0 50115 2 0x00000080
Jan 9 23:26:02 hblf-ct-dsm009 kernel: Call Trace:

Cause of problem:

By default, Linux uses up to 40% of the available memory as the file system cache. When this threshold is exceeded, the file system will write all the memory in the cache to the disk, resulting in subsequent IO requests being synchronized.

When the cache is written to disk, there is a default timeout of 120 seconds. The reason for the above problem is that IO cannot write all the data in the cache to disk within 120 seconds. The IO system responds slowly, causing more and more requests to accumulate, and eventually all the system memory is occupied, resulting in the system losing response.  

 "Echo 0 > / proc / sys / kernel / hung_task_timeout_secs" means to turn off the alarm prompt.
terms of settlement:
Adjust the proportion of memory occupied by dirty data in the file system cache, adjust parameters, reduce the percentage of dirty data to start processing, and improve the maximum capacity of dirty data. So that the file system cache will not be written to the hard disk so fast, and the system has time to process dirty data. It mainly includes the following two parameters:

vm.dirty_background_ratio specifies that when the number of dirty pages cached by the file system reaches the percentage of the system memory (e.g. 5%), background writeback processes such as pdflush/flush/kdmflush will be triggered to run to asynchronously brush a certain cached dirty pages into external memory.
vm.dirty_ratio specifies that when the number of dirty pages cached by the file system reaches 10% of the system memory (for example), the system has to start processing the cached dirty pages (because there are a large number of dirty pages at this time, it is necessary to brush some dirty pages into external memory to avoid data loss). In this process, many application processes may be blocked because the system processes file IO instead.

sysctl -a | grep dirty # check the current parameter. I will double the trigger value and adjust the tolerance to 60%

#Write kernel

vi /etc/sysctl.conf 

vm.dirty_background_ratio = 5 # originally 10

vm.dirty_ratio = 60 # originally 20

sysctl -p # enables modification of kernel parameters