Summary of Linux Log Management Experience (crontab + logrotate)

Click the "Linux Tech Enthusiast" above and select "Set as Star" to receive high-quality articles promptly.

☞【Insight】ChatGPT 4.0 is unlocked, no limit on questions!!!
☞【Insight】Tsinghua University senior's self-study Linux notes, top-level quality!
☞【Insight】Comprehensive guide to commonly used Linux commands, all in one article.
☞【Insight】Collection! Linux basic to advanced learning roadmap.

Link: https://www.cnblogs.com/xiaoyaozhe/p/17671275.html

Log Management Objectives

Log management generally includes two main parts:

1. Log content: Reasonable log content (log anchors, content format, etc.) can provide the most effective assistance for application service execution records and problem troubleshooting.

2. Log archiving rules: This includes log splitting methods (by date, by file size, etc.) and the number of archived logs, such as only keeping the last month.

For self-developed application services, log management can be customized by developers through logging components like logback, log4j, etc. However, for third-party components installed and deployed, such as MySQL, nginx, redis, etc., and third-party components referenced by developers like nacos, sentinel, etc., unless the component provides rich logging configuration parameters, it will not be possible to manage log files according to the developers’ requirements.

Special Log Scenarios

Some special application services or components, if not configured specifically, will cause log files to be uncontrolled by default, leading to difficulties in later cleanup. Common examples include:

• Starting application services with nohup, if log redirection is not configured, or redirected to a single file, the system will continuously output logs to the nohup.out file or the redirected single file.

• The MySQL database supports configuring the log file path but cannot automatically clean up log content.

• Nginx supports configuring log content templates and log file paths (default access.log, error.log) but cannot automatically clean up log content.

In the above situations, logs will continuously output in a single file, and after a certain period, the log file will occupy an unlimited amount of disk storage, causing operational failures in the entire system. Search for the public account: Architect Guide, reply: Architect to receive materials.

Special Tool – Scheduled Cleanup

You can use the built-in Linux scheduling tool crontab + log cleanup script to achieve scheduled cleanup, as shown below:

crontab -e# Scheduled log cleanup, keep the last 7 days1 0 * * * find /logs.dir/ -mtime +7 | xargs rm -rf

Special Attention

In Linux systems, some application service components like nginx and MySQL, during operation, will track log storage files using file handles, which can lead to the following issues:

1. Renaming the log file while creating a new file with the same name will still cause the component to output logs to the original file.

2. Deleting the log file (rm -f) also requires restarting the application service component process; otherwise, the deleted file will remain occupied and unable to release disk resources.

3. Deleted but occupied files cannot be viewed with the ls -l command, nor can they be counted for disk usage with the du -sh command, but the df -h command will show the actual disk usage. Only the lsof command can be used to view open file descriptors, causing significant trouble for operations and maintenance.

In response to this situation:

1. If it is just a one-time cleanup of files, you can use the method of emptying the file, such as echo > log.log, or other methods of clearing, but be careful not to directly rm -f delete.

2. If rm -f delete has already been executed, you can use the lsof | grep -i deleted command to view the deleted but unrecoverable files, and then restart the corresponding process to recover.

3. If you want to retain log content while controlling storage capacity, you need to use logrotate’s copy + truncate method, which means only copying the log file content for archiving and then truncating the current log file (instead of deleting it).

Special Tool – logrotate

For application service components that are inconvenient to manage logs customarily, you can maintain custom scripts or develop application software for maintenance, but it is strongly recommended to use the log management tool logrotate integrated into the Linux system. This tool is scheduled by the Linux system’s crontab and supports custom storage rules for related log files (or other files), but log content can only be output according to the implementation of the application service component.

logrotate provides many functional parameters, with the following being the most commonly used:

• Log splitting cycle

• Log file extension

• Log file splitting methods, including new + delete, copy + truncate, etc., suitable for different application service scenarios

• Log content compression

• Number of archived log files

logrotate command format: logrotate [OPTION...] <configfile>-d, --debug: debug mode, test if the configuration file has errors.-f, --force: force dump files.-m, --mail=command: send logs to the specified email after compressing.-s, --state=statefile: use the specified state file.-v, --verbose: display the dumping process.
vi /etc/logrotate.d/nginx     # Create a new nginx file in the /etc/logrotate.d/ directory, with the following content:/usr/share/nginx/log/*.log{dailymissingokrotate 7compressdelaycompressnotifemptycreate 644 root rootsharedscriptspostrotate[ ! -f /var/run/nginx.pid ] || kill -USR1 `cat /var/run/nginx.pid`endscript}

# Calling methodlogrotate -d /etc/logrotate.d/nginx# Combine with crontab for scheduled execution echo "0 0 * * * /usr/sbin/logrotate -vf /etc/logrotate.d/nginx > /dev/null 2>&1" >> /var/spool/cron/root


# Parameter description:compress                                  Compress the dumped logs using gzipnocompress                                Do not perform gzip compressioncopytruncate                              For logs that are still open, back up the current log and truncate it; this is a copy and then clear method, and there may be a time difference between copying and clearing, which may lose some log data.nocopytruncate                            Back up the log file but do not truncatecreate mode owner group                   Specify the attributes for creating new files during rotation, such as create 0777 nobody nobodynocreate                                  Do not create new log filesdelaycompress                             When used with compress, the dumped log files will be compressed during the next dumpnodelaycompress                           Override the delaycompress option, compressing during the dump.missingok                                 If the log is missing, do not report an error and continue to the next log.errors address                            Send error messages during dumping to the specified email addressifempty                                   Rotate even if the log file is empty; this is the default option of logrotate.notifempty                                Do not rotate when the log file is emptymail address                              Send the dumped log file to the specified email addressnomail                                    Do not send log files during dumpingolddir directory                          Place the dumped log files in the specified directory, which must be on the same file system as the current log file.noolddir                                  Place the dumped log files in the same directory as the current log file.sharedscripts                             Run the postrotate script, which is executed once after all logs are rotated. If this is not configured, the script will be executed once for each log rotation.prerotate                                 Instructions to be executed before logrotate dumps, such as modifying file attributes; must be on a separate linepostrotate                                Instructions to be executed after logrotate dumps, such as restarting (kill -HUP) a service; must be on a separate line.daily                                     Specify the dump cycle as dailyweekly                                    Specify the dump cycle as weeklymonthly                                   Specify the dump cycle as monthlyrotate count                              Specify the number of times to dump the log file before deletion, 0 means no backup, 5 means keep 5 backups.dateext                                   Use the current date as the naming formatdateformat .%s                            Used with dateext, immediately follows the next line, defines the filename after splitting, must be used with dateext, only supports %Y %m %d %s four parameters.size(或minsize) log-size                  Only dump when the log file reaches the specified size, log-size can specify bytes (default) and KB (sizek) or MB (sizem).
# When the log file >= log-size, it will be dumped. The following are valid formats: (other formats of unit size have not been tested)size = 5 or size 5 (dump when >= 5 bytes)size = 100k or size 100ksize = 100M or size 100M

Appendix: Simple Configuration of logrotate

MySQL/data/mysql/log/mysqld.log{    daily    dateext    dateyesterday    copytruncate    notifempty    missingok    olddir backup    rotate 60        compress}
nginx/usr/local/nginx/logs/access.log/usr/local/nginx/logs/error.log{    daily    dateext    dateyesterday    copytruncate    notifempty    missingok    olddir backup    rotate 30        compress}

Appendix: Log Management of Common Components During Runtime

1. Nginx does not support automatic cleanup, default single file continues to write, and will not automatically roll.

2. MySQL does not support automatic cleanup, default single file continues to write, and will not automatically roll.

3. Zookeeper supports automatic cleanup by default (limits file size and number), maintained by log4j configuration files.

4. Redis does not support automatic cleanup, only records a small amount of core logs, single file continues to write, but by default only records a small amount of core logs, can be ignored.

5. Kafka’s data recording logs (topic, offset, etc.) support automatic cleanup, maintained by configuration files.

6. Kafka operation logs are by default in the installation directory logs, support automatic rolling, but will not automatically clean up, maintained by log4j configuration files.

-End-

If you have read this far, it means you like the articles from this public account. Welcome to pin (star) this public account Linux Tech Enthusiast, so you can get push notifications first!

In this public account Linux Tech Enthusiast, reply: Linux to receive 2T of learning materials!
Recommended Reading
1. ChatGPT Chinese version 4.0, everyone can use it, fast and stable!
2. Common Linux commands, a full 20,000 words, the most comprehensive summary on the internet.
3. Linux Learning Guide (Collection Edition)
4. Access official ChatGPT and Claude without translation, stable with after-sales service.