How to Investigate a Linux System Breach?

When a company experiences a hacker intrusion; when the system crashes; when a security incident affects normal business operations, can you respond immediately and provide a solution? The following article will teach you the Linux emergency response process!

Intrusion Investigation Approach

01

Apache Log Path

Access Log:

/var/log/apache2/access.log (Debian/Ubuntu) or /var/log/httpd/access_log (CentOS/RHEL)

Nginx Log Path

Access Log:

/var/log/nginx/access.log

Apache Root Directory

Debian/Ubuntu (including Kali Linux):

/var/www/html/

CentOS/RHEL:

/var/www/html/

In Nginx the default root directory is usually:

Debian/Ubuntu systems: /var/www/html CentOS/RHEL systems: /usr/share/nginx/html

02

Detection & Analysis

System Status Check

Process Analysis

top -c                     # View processes with abnormal CPU/memory usage ps aux --sort=-%cpu       # Sort processes by CPU usage pstree -p                 # View process tree to find abnormal parent-child relationships

Network Connection Analysis

netstat -antp | grep ESTABLISHED  # View active connections lsof -i :suspicious_port  # Locate the process listening on the port ss -tulwn                       # Replace netstat, show more detailed network information

User and Login Check

last -ai                   # View login records (focus on root user) grep "Accepted" /var/log/auth.log  # Check SSH successful login logs cat /etc/passwd | grep -E "/bin/bash|/bin/sh"  # Check loggable users

Username: Password: User ID: Group ID: User Description: Home Directory: Login Shell

1. Query privileged users (uid is 0)

[root@localhost ~]# awk -F: '$3==0{print $1}' /etc/passwd;

2. Query accounts that can log in remotely

[root@localhost ~]# awk '/\$1|\$6/{print $1}' /etc/shadow

3. Check if other accounts besides root have sudo privileges. If not needed for management, ordinary accounts should have sudo privileges removed.

[root@localhost ~]# more /etc/sudoers | grep -v "^#\|^$" | grep "ALL=(ALL)"

4. Disable or delete unnecessary and suspicious accounts

  usermod -L user    Disable account, account cannot log in, /etc/shadow second column starts with !  userdel user       Delete user

  userdel -r user    Delete user and also remove the user directory under /home

File System Check

Sensitive File Scan

find / -name "*.php" -mtime -3   # Find PHP files modified in the last 3 days (WebShell) find / -perm -4000 -ls           # Find SUID privilege escalation files grep -r "eval(base64_decode" /var/www/html  # Scan for encrypted code in web directory

Hidden File Detection

ls -la /tmp              # View hidden files in /tmp directory lsattr /suspicious_file         # Check file hidden attributes (e.g., undeletable)

Log Analysis

System Logs

journalctl -u sshd   # View SSH service logs tail -f /var/log/secure      # Real-time monitor authentication logs

Terminate Malicious Processes

ps -aux                          # View processes kill -9 malicious_process_PID               # Force terminate process systemctl stop malicious_service_name         # Stop malicious service

Check Scheduled Tasks

1. Create scheduled tasks using crontab

crontab -l List detailed contents of a user's cron service

Check System Logs

Default log storage location: /var/log/ Check log configuration: more /etc/rsyslog.conf