problem
The scheduled task couldn't get up last night. It uses dubbo architecture and relies on service
20:02:02.406 [main] INFO com.alibaba.spring.beans.factory.annotation.ConfigurationBeanBindingRegistrar - The configuration bean definition [name : org.apache.dubbo.config.ApplicationConfig#0, content : Root bean: class [org.apache.dubbo.config.ApplicationConfig]; scope=; abstract=false; lazyInit=null; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] has been registered. 20:02:02.407 [main] INFO com.alibaba.spring.util.BeanRegistrar - The Infrastructure bean definition [Root bean: class [com.alibaba.spring.beans.factory.annotation.ConfigurationBeanBindingPostProcessor]; scope=; abstract=false; lazyInit=null; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=nullwith name [configurationBeanBindingPostProcessor] has been registered. 20:02:02.407 [main] INFO com.alibaba.spring.beans.factory.annotation.ConfigurationBeanBindingRegistrar - The configuration bean definition [name : org.apache.dubbo.config.RegistryConfig#0, content : Root bean: class [org.apache.dubbo.config.RegistryConfig]; scope=; abstract=false; lazyInit=null; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] has been registered. 20:02:02.407 [main] INFO com.alibaba.spring.beans.factory.annotation.ConfigurationBeanBindingRegistrar - The configuration bean definition [name : org.apache.dubbo.config.ConsumerConfig#0, content : Root bean: class [org.apache.dubbo.config.ConsumerConfig]; scope=; abstract=false; lazyInit=null; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] has been registered. 20:02:03.945 [main] INFO org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor - class org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor was destroying!
This is the last log output from the task task
I didn't know what the situation was, so I tried to restart and found that I couldn't get up
20:39:13.826 [main] INFO com.alibaba.spring.util.BeanRegistrar - The Infrastructure bean definition [Root bean: class [com.alibaba.spring.beans.factory.annotation.ConfigurationBeanBindingPostProcessor]; scope=; abstract=false; lazyInit=null; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=nullwith name [configurationBeanBindingPostProcessor] has been registered. 20:39:13.826 [main] INFO com.alibaba.spring.beans.factory.annotation.ConfigurationBeanBindingRegistrar - The configuration bean definition [name : org.apache.dubbo.config.RegistryConfig#0, content : Root bean: class [org.apache.dubbo.config.RegistryConfig]; scope=; abstract=false; lazyInit=null; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] has been registered. 20:39:13.826 [main] INFO com.alibaba.spring.beans.factory.annotation.ConfigurationBeanBindingRegistrar - The configuration bean definition [name : org.apache.dubbo.config.ConsumerConfig#0, content : Root bean: class [org.apache.dubbo.config.ConsumerConfig]; scope=; abstract=false; lazyInit=null; autowireMode=0; dependencyCheck=0; autowireCandidate=true; primary=false; factoryBeanName=null; factoryMethodName=null; initMethodName=null; destroyMethodName=null] has been registered. 20:39:15.627 [main] INFO org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor - class org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor was destroying!
It's a similar error. I think there may be a problem with the service, and then I tested it with postman. It's very smooth.
Then I restarted the zk and service, and found that the task still couldn't get up... Outrageous
Dubbo admin found the problem
The process is very tortuous, so I won't mention it.
Finally, the reason was found in Dubbo admin
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-10-21 20:52:30,540 [DubboSaveRegistryCache-thread-1] WARN com.alibaba.dubbo.registry.support.AbstractRegistry (AbstractRegistry.java:189) - [DUBBO] Failed to save registry store file, cause: No space left on device, dubbo version: 2.6.2, current host: 172.17.0.1 java.io.IOException: No space left on device at java.io.FileOutputStream.writeBytes(Native Method) at java.io.FileOutputStream.write(FileOutputStream.java:326) at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221) at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282) at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125) at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207) at java.io.BufferedWriter.flushBuffer(BufferedWriter.java:129) at java.io.BufferedWriter.write(BufferedWriter.java:230) at java.io.Writer.write(Writer.java:157) at java.util.Properties.store0(Properties.java:840) at java.util.Properties.store(Properties.java:818) at com.alibaba.dubbo.registry.support.AbstractRegistry.doSaveProperties(AbstractRegistry.java:170) at com.alibaba.dubbo.registry.support.AbstractRegistry$SaveProperties.run(AbstractRegistry.java:500) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at
Don't pick too many
The key points are: * * No space left on device**
Solution startup
It's easy to find the reason.
#View disk space usage df -h
Then switch to the root directory
cd /
See which take up more space
find . -type f -size +800M
rm delete it. After deleting it, check whether it is really cleaned up (when the file is deleted and the space is not released)
Because there is a handle
stay Linux perhaps Unix In the system, through rm Or file manager deleting a file will unlink it from the folder structure of the file system(unlink).However, it is assumed that the file is Open (a process is in use), then the process will still be able to read the file and the disk space will be occupied all the time. And what I deleted was oracle Alarm of log file The file should be in use at the time of deletion
First, get a list of files that have been deleted but are still occupied by the application, such as the following:
lsof |grep deleted
How to release the process?
The method is to kill the corresponding process, or stop the application using this file, and let the os reclaim disk space on its own initiative
By truncating files in the proc file system, the kill process can force the system to recycle the files allocated to the file in use.
Check the usage of disk space and find that the space has been recycled
Start the scheduled task again
I mainly cleaned up the var/log/messages file
Attach cleaning method
Introduction to linux files
1. / var/log/secure records the files that log in to the system to access data (for example, pop3, ssh, telnet, ftp, etc. will be recorded here);
2. / ar/log/btmp records the login information, which has been encoded, so it must be parsed in lastb;
lastb | awk '{ print $3}' | sort | uniq -c | sort -nr | more
3. / var/log/message almost all errors occurred in the boot system will be recorded here;
4. / var/log/boot.log records the startup or shutdown information displayed by some services started or shut down;
5. / var/log/maillog records the access and exchange of mail;
6. / var/log/cron is used to record the contents of crontab (scheduled task) service;
7. / var/log/lastlog records the last login information of each user;
8. / var/log/btmp records incorrect login attempts;
9. / var/log/dmesg kernel log;
10. / var/log/yum.log package information installed using yum
The file system of Linux is better than that of Windows and will not produce fragments. It is particularly important for servers running for a long time, and the Linux system itself will not produce a large number of garbage files like Windows. I don't know if this statement has credibility! At least we can make sure that the file system of Linux system is excellent!
Error:
rm -f logfile
reason:
The application has opened the file handle. Deleting it directly will cause:
1. The application cannot release the log file and write correctly
2. Display that disk space is not released
Correct way:
cat /dev/null > logfile
#Example: cat /dev/null > /var/log/messages
Save the following code as a script file with. sh suffix, and then cooperate with crontab to clean the log regularly;
#!/bin/sh cat /dev/null > /var/log/secure cat /dev/null > /var/log/btmp cat /dev/null > /var/log/message cat /dev/null > /var/log/boot.log cat /dev/null > /var/log/maillog cat /dev/null > /var/log/cron cat /dev/null > /var/log/lastlog cat /dev/null > /var/log/btmp cat /dev/null > /var/log/dmesg cat /dev/null > /var/log/yum.log