Recently, a machine in one of our company’s factories has been frequently alarming due to low disk space. Upon investigation, it was found that this was a test machine for the MES service provided by a vendor, which had a complete set of testing services built on it. The vendor’s Spring Boot application outputs logs to a single file without splitting them by date or file size, resulting in extremely large log files, sometimes reaching dozens of gigabytes, filling up the disk space.Since this is a service provided by the vendor, we cannot modify their code or configuration files, so we can only perform some operations at the operating system level.1. Manually clearing the log file once, such as executing > xxx.log, will directly clear the log file while retaining the file handle, without needing to restart the application. However, when the log traffic is very high, a small amount of logs may be lost.2. Using logrotate to periodically split the logs. Logrotate is a built-in log management tool in Linux that can archive logs based on date, size, etc. The configuration files are located in the /etc/logrotate directory, and you can add a configuration file like the following:
/var/log/XXX.log { missingok daily copytruncate rotate 7 notifempty}As shown above, during log rotation, a copy of xxx.log is made, and then the truncate operation is performed on xxx.log, which is the same as the operation in step 1. However, using the copytruncate operation can lead to two issues when the traffic is high: 1) a small amount of logs may be lost; 2) since writing logs to disk is an asynchronous operation, it is easy to encounter a situation where the first part of the log is lost while the latter part is written, resulting in garbled text in the first few lines of the log file, which the operating system may recognize as binary data. The root cause of these two issues is that the JVM process still holds the log file handle and continues to write to it, which cannot be resolved at the operating system level.A better approach1. If you can modify the service’s code and configuration, it is best to use Java logging components like logback or log4j to implement log rotation. These components are mature and can achieve log rolling with minimal configuration.2. If you cannot modify the code, you can use programs like rotatelogs or cronlog in Linux to implement log rotation. Both work similarly by using the Linux operating system’s pipe operations to redirect the business program’s logs to a buffer, allowing the rotatelogs or cronlog program itself to decide which log file to write to, without the business program directly manipulating the log file. However, this solution requires specifying the use of rotatelogs or cronlog when starting the business program, so the business program must be restarted once.
java --add-opens java.base/java.util=ALL-UNNAMED --add-opens java.base/sun.reflect.annotation=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/sun.security.action=ALL-UNNAMED -Xms1024m -Xmx2048m -Xmn256m -Xss256k -server -XX:+HeapDumpOnOutOfMemoryError -jar ${APP_NAME} 2>&1 | rotatelogs -L server.log server.log.%Y-%m-%d 86400As shown in the command above, the log file is output to rotatelogs, which implements log rotation, generating a new file each day.By implementing log rolling and combining it with crontab scheduled tasks, you can retain logs for only X days, deleting the rest.