Lost in Linux event logs? This guide helps you decode, filter, and troubleshoot like a pro—no more despairing at endless logs!
Hello Gophers, I am gogogo, welcome to follow me.
If your Linux system is acting up and you don’t know why, the logs can tell you where the problem lies. This guide covers the key logs to check and how to use them to troubleshoot—whether you’re running a personal setup or managing servers.
“Talk is cheap. Show the code,” let’s get started!
1 What are Linux Event Logs?
Linux event logs are records of system activities, errors, warnings, and informational messages generated by the Linux kernel, applications, and services. You can think of them as a diary of your system—they document everything that happens behind the scenes.
These logs provide a comprehensive view of what is happening on your system, making them invaluable for:
- Tracking system issues
- Monitoring security issues
- Understanding application behavior
- Identifying performance bottlenecks
Each log entry typically contains:
- Timestamp: The time the event occurred
- Hostname: Which system generated the event
- Application/Service Name: What generated the event
- Process ID (PID): Which process wrote the log
- Message: The actual event details
2 Linux Event Logs
The most comprehensive list of events in Linux can usually be found in /var/log/syslog (on Debian-based systems like Ubuntu) or /var/log/messages (on Red Hat-based systems like CentOS).
These files contain almost everything that happens on your system.
However, Linux does not put all its eggs in one basket. Here are the locations for different types of event logs:
| Log File | Contents | Usage |
|---|---|---|
| /var/log/syslog or/var/log/messages | General system events | Suitable for most troubleshooting |
| /var/log/auth.log or/var/log/secure | Authentication attempts | Great for security monitoring |
| /var/log/kern.log | Kernel messages | Very useful for hardware and driver issues |
| /var/log/dmesg | Boot messages | Helps troubleshoot boot issues |
| /var/log/apache2/ or/var/log/httpd/ | Web server activity | Web service troubleshooting |
| /var/log/mysql/ | Database server logs | MySQL/MariaDB issues |
| /var/log/apt/ | Package management | Installation and update tracking |
| /var/log/cron | Scheduled task logs | Cron job debugging |
| /var/log/boot.log | System boot logs | Boot sequence issues |
| /var/log/faillog | Failed login attempts | Security auditing |
Understanding the Log Directory Structure
The /var/log directory follows a logical organizational structure:
- System-wide logs are located in the main directory
- Application-specific logs are located in subdirectories
- Rotated logs have extensions (.1, .2.gz, etc.) Running ls -la /var/log will show all available logs, including hidden ones.
3 Linux Logging Architecture
Linux uses a layered approach to logging:
① Applications generate log messages② Logging libraries (like libsyslog) format these messages③ Logging daemons (like rsyslog or syslog-ng) route these messages④ Storage backends save them to files, databases, or remote servers
Modern Linux distributions use one of the two main logging systems: traditional Syslog and Systemd Journal:
| Logging System | Syslog | Systemd Journal |
|---|---|---|
| Log Format | Text-based log files | Binary structured logs |
| Daemon Management | Managed by rsyslog or syslog-ng daemons | Managed by systemd-journald |
| Configuration Method | Configured via /etc/rsyslog.conf or/etc/syslog-ng/syslog-ng.conf | Configured via /etc/systemd/journald.conf |
| Storage Location | Stored as plain text files in /var/log/ | Stored in binary format in /var/log/journal/ |
| Viewing Method | Text tools (grep/tail/jq, etc.) | Accessed via the journalctl command |
Many systems use both systems simultaneously, with journald feeding events to rsyslog.
4 Essential Commands for Viewing Linux Events
You don’t need fancy tools to get started. These commands will help you navigate logs like a pro:
Ⅰ journalctl Command
If your system uses systemd (most modern distributions do), then journalctl is your Swiss Army knife:
# All logsjournalctl# View today's logsjournalctl --since today# View real-time logs (like tail -f)journalctl -f# View logs for a specific servicejournalctl -u apache2# View logs for a specific time rangejournalctl --since "2023-10-15 10:00:00" --until "2023-10-15 11:00:00"# View logs for a specific executablejournalctl /usr/bin/sshd# View logs for a specific PIDjournalctl _PID=1234# View logs for a specific userjournalctl _UID=1000# View kernel messagesjournalctl -kⅡ Classic Log Commands
Applicable for traditional system logs or specific log files:
# View the latest 50 lines of logs.tail -n 50 /var/log/syslog# View real-time logs.tail -f /var/log/syslog# Search for error messages.grep "error" /var/log/syslog# Search all logs case-insensitively.grep -i "failed" /var/log/auth.log# View logs with context (3 lines before and after).grep -A 3 -B 3 "critical" /var/log/syslog# Count occurrences of a specific event.grep -c "authentication failure" /var/log/auth.log# Use less to easily browse large log files.less /var/log/syslogⅢ Advanced Log Analysis with awk and sed
For more complex log analysis:
# Extract IP information from auth.log file.awk '/Failed password/ {print $11}' /var/log/auth.log | sort | uniq -c | sort -nr# Filter access.log logs by HTTP status code.awk '$9 == 404 {print $7}' /var/log/apache2/access.log | sort | uniq -c | sort -nr# Extract only timestamp and error information.sed -n 's/.*\(\d\{2\}:\d\{2\}:\d\{2\}\).*error: \(.*\)/\1 \2/p' /var/log/syslog5 Understanding Log Priorities
Linux logs are not just random records; they are organized by severity. Understanding these priorities helps you filter the signal from the noise:
| Priority | Name | Meaning | Example |
|---|---|---|---|
| 0 | emerg | System unusable – panic mode | Kernel crash, hardware failure |
| 1 | alert | Immediate action required | System database corruption |
| 2 | crit | Critical condition | Disk error |
| 3 | err | Error condition | Application crash |
| 4 | warning | Warning condition | Configuration issue |
| 5 | notice | Normal but significant condition | Service start/stop |
| 6 | info | Informational message | Routine operational events |
| 7 | debug | Debug-level messages | Detailed development information |
Use these when filtering by severity with journalctl:
# Show only errors and above priority logs.journalctl -p err# Show warning and above priority logs.journalctl -p warning# Count events at each level.for i in emerg alert crit err warning notice info debug; do echo -n "$i: " journalctl -p $i --since today | grep -v "-- Journal" | wc -l; done6 Advanced Event Monitoring Techniques
| Method | Solution | Basic Setup |
|---|---|---|
| Elastic StackRequires a powerful environment for search and visualization | Elasticsearch stores and indexes logsLogstash processes and normalizes logsKibana provides visual dashboardsBeats (Filebeat, Metricbeat) collect and forward logs | Install Elasticsearch and start the serviceInstall Kibana and connect it to ElasticsearchInstall Filebeat on log sourcesConfigure Filebeat to collect specific logs(Optional) Set up Logstash for advanced processing |
| Prometheus+GrafanaFor real-time metrics and event-driven alerts | Prometheus collects and stores time-series dataNode_exporter exposes system metricsGrafana visualizes data and creates alerts | Install Prometheus for metrics collectionSet up node_exporter to expose system metricsConfigure Prometheus to scrape these metricsInstall Grafana for visualizationCreate dashboards and alert rules |
| Fail2ban Automated Alerts | Fail2ban monitors logs for suspicious activity and takes action | Install fail2ban (sudo apt install fail2ban)Configure logs to monitor |
| Custom | Write custom log analyzers | For specialized needs, simple scripts can effectively analyze logs |
Example: Ⅰ fail2ban
# Install fail2bansudo apt install fail2ban# Check its statussudo systemctl status fail2banConfiguration:
[sshd]enabled = trueport = sshfilter = sshdlogpath = /var/log/auth.logmaxretry = 5bantime = 3600Configuration Explanation:
- Monitors SSH login attempts in auth.log.
- Bans IP after 5 failed attempts.
- Keeps the ban for 1 hour.
Ⅱ Custom
#!/usr/bin/env python3import reimport syserror_pattern = re.compile(r'(\w{3} \d{2} \d{2}:\d{2}:\d{2}).*ERROR: (.*)')with open('/var/log/application.log', 'r') as f: for line in f: match = error_pattern.search(line) if match: timestamp, message = match.groups() print(f"{timestamp}: {message}")This script extracts error messages with timestamps from application logs, simplifying troubleshooting.
7 Using Events to Troubleshoot Common Linux Issues
Now it’s time for practical application—let’s see how logs can help solve real problems.
① Finding Failed Login Attempts
Security issues? Check the authentication logs:
grep "Failed password" /var/log/auth.logLook for:
- Repeated attempts from the same IP
- Attempts to access non-existent users
- Attempts outside of normal working hours
For summary reports:
grep "Failed password" /var/log/auth.log | \
awk '{print $11}' | sort | uniq -c | sort -nr | head -10The above shows the top 10 IPs with failed login attempts.
② Debugging Application Crashes
When an application keeps crashing:
journalctl -u nginx --since "10 minutes ago"Look for:
- Segmentation faults
- Memory overflow errors
- Permission issues
- Missing dependencies
For failed systemd service startups:
systemctl status application.servicejournalctl -u application.service -n 50Note the exit codes—they often point to specific issues:
- Exit code 1: General error
- Exit code 126: Command not executable
- Exit code 127: Command not found
- Exit code 137: Process killed (usually by OOM killer)
③ Identifying Disk Space Issues
Out of space?
grep "No space left on device" /var/log/syslogFor a comprehensive disk space analysis:
# Find largest log filesfind /var/log -type f -exec du -h {} \; | sort -hr | head -20# Check if any logs are growing unusually fastwatch -n 1 'ls -la /var/log/*.log'# See which processes are writing to logslsof | grep "/var/log"④ Resolving Network Connection Issues
Network issues:
# Check for dropped connectionsgrep "Connection reset by peer" /var/log/syslog# Look for firewall blocksgrep "UFW BLOCK" /var/log/kern.log# Check for DNS resolution problemsgrep "resolv" /var/log/syslogFor deeper network debugging:
- /var/log/networkmanager for NetworkManager logs
- DHCP client logs are in syslog
- VPN connection logs (OpenVPN, WireGuard)
⑤ Investigating Boot Issues
When your system fails to boot:
# Check early boot logsjournalctl -b -1 -p err# Look for filesystem check errorsgrep "fsck" /var/log/boot.log# Check for hardware detection issuesdmesg | grep -i errorKey boot log locations:
- journalctl -b for systemd systems
- /var/log/boot.log for traditional init systems
- /var/log/dmesg for kernel boot messages
8 When to Pay Attention to Logs Beyond syslog
While syslog contains a comprehensive list of events, some applications maintain their own logs:
| Service Type | Log Location | Description |
|---|---|---|
| MySQL/MariaDB | Error log: /var/log/mysql/error.log Slow query log: /var/log/mysql/mysql-slow.log General query log: /var/log/mysql/mysql.log |
Table corruption Connection issues Query timeouts |
| PostgreSQL | Main log: /var/log/postgresql/postgresql-X.Y-main.log Various levels can be configured in postgresql.conf |
|
| Apache | Access log: /var/log/apache2/access.log Error log: /var/log/apache2/error.log Custom vhost logs, depending on configuration |
|
| Nginx | Access log: /var/log/nginx/access.log Error log: /var/log/nginx/error.log |
|
| Docker | Container logs: docker logs container_name Docker daemon: journalctl -u docker.service |
|
| Kubernetes | Pod logs: kubectl logs pod_name Node logs: usually in journald Control plane logs: in /var/log/kube-* or as containers |
|
| Mail Server Postfix | Email log: /var/log/mail.log Email errors: /var/log/mail.err |
SMTP authentication failures Delivery failures Spam filtering issues |
9 Advanced Log Parsing Tools
① GoAccess
A real-time web log analyzer that provides insights through a terminal-based dashboard and HTML reports.
# Terminal dashboard for Apache logsgoaccess /var/log/apache2/access.log -c# Generate HTML reportgoaccess /var/log/apache2/access.log --log-format=COMBINED -o report.html② Lnav (Log File Navigator)
Lnav provides an interactive way to analyze multiple log files in a unified view.
# Open multiple logs at once.lnav /var/log/syslog /var/log/auth.log /var/log/apache2/error.log# Press '?' for help in the interactive interface③ Logwatch
An automated log analysis and reporting tool that summarizes logs and provides daily reports.
# Install logwatchsudo apt install logwatch# Generate a report for yesterday'sudologwatch --detail high --range yesterday --output stdout10 Best Practices for Linux Event Management
Follow these tips to keep your logging system effective:
| Strategy | Description |
|---|---|
| Set up log rotation,to prevent disk space issues | Configure based on size and time Consider compressing archives Set appropriate retention periods |
| Use consistent timestamp formats | Preferably in ISO 8601 format (YYYY-MM-DD HH:MM:SS) Include timezone information Consider using UTC for multi-region setups |
| In multi-server environments,include hostname in logs | Crucial for centralized logging Helps trace issues across services Consider adding application name and version |
| Configure appropriate log levels,too much or too little is equally bad | Use DEBUG only temporarily Keep INFO for normal operations Ensure ERROR and CRITICAL are always logged |
| Regularly back up important logs | Include logs in backup strategy Consider longer retention for security logs Retain audit logs for compliance requirements |
| Implement log security measures | Set appropriate permissions (usually 640 or 600) Use dedicated log user accounts Consider using log signing to prevent tampering |
| Establish log monitoring and alerts | Alert on critical errors Set up trend monitoring Create dashboards for visibility |
| Document your logging architecture | Map all log sources Document retention policies Create troubleshooting guides |
11 Conclusion
The logging system in Linux provides you with a powerful window into what is happening on your system. The comprehensive list of events in these logs, especially in syslog or journald, gives you the insights you need to troubleshoot quickly.
#LinuxEvents#events #logs #logAnalysis #metrics #strategies #bestPractices #monitor #Metrics #Logs #Tracing
Please open in the WeChat client