Linux Emergency Response – Common Service Logs

System Logs

btmp

/var/log/btmp records all login attempts that failed, displaying the last ten entries.

root@mon0dy-ubuntu:~# lastb --time-format iso -10
root     ssh:notty    58.56.52.226     2023-03-11T14:30:23+0800 - 2023-03-11T14:30:23+0800  (00:00)
root     ssh:notty    58.56.52.226     2023-03-11T14:30:20+0800 - 2023-03-11T14:30:20+0800  (00:00)
root     ssh:notty    58.56.52.226     2023-03-11T14:30:16+0800 - 2023-03-11T14:30:16+0800  (00:00)
root     ssh:notty    58.56.52.226     2023-03-11T14:30:05+0800 - 2023-03-11T14:30:05+0800  (00:00)
root     ssh:notty    58.56.52.226     2023-03-11T14:30:02+0800 - 2023-03-11T14:30:02+0800  (00:00)
root     ssh:notty    58.56.52.226     2023-03-11T14:29:55+0800 - 2023-03-11T14:29:55+0800  (00:00)
         ssh:notty    64.62.197.191    2023-03-11T09:26:44+0800 - 2023-03-11T09:26:44+0800  (00:00)
         ssh:notty    64.62.197.187    2023-03-10T20:29:56+0800 - 2023-03-10T20:29:56+0800  (00:00)
admin    ssh:notty    43.156.108.211   2023-03-10T07:54:41+0800 - 2023-03-10T07:54:41+0800  (00:00)
admin    ssh:notty    43.156.108.211   2023-03-10T07:54:39+0800 - 2023-03-10T07:54:39+0800  (00:00)

btmp begins 2023-03-01T07:46:00+0800
root@mon0dy-ubuntu:~# 
lastb | awk '{print $3}' | sort | uniq -c | sort -n
awk '{print $3}'  :Extracts the third column from the output data.
sort  :Classifies the data.
uniq -c :Removes duplicates from the classified data and counts them.
sort -n :Classifies the deduplicated and counted data and sorts it in ascending order.

Why do we see data like Thu? If we look at lastb again, we will find that some usernames are empty, so when using awk ‘{print $3}’, it selects the column after Sun, which needs to be handled carefully.

wtmp

/var/log/wtmp records all users who have successfully logged into the system.

Date formatting: <span>last --time-format iso</span>, which looks more pleasant.

SSH Logs

Command parameters to view network connections.

Proto: Protocol name.

Recv-Q: Network receive queue.

This indicates that the received data is in the local receive buffer, but how much has not been retrieved by the process, recv. If the receive queue Recv-Q remains blocked, it may be under a denial-of-service attack.

send-Q: Network send queue.

This refers to data that has not been acknowledged by the other party or is still in the local buffer. If the send queue Send-Q cannot be cleared quickly, it may be due to an application sending packets too quickly or the other party not receiving packets quickly enough.

Both recv-Q and send-Q should typically be 0; if not, there may be an issue. Packets should not be piling up in either queue. A brief non-zero situation is acceptable.

  1. Local Address: Local address.
  1. 0.0.0.0:2000: Listens on port 2000 for all IP addresses on the server (0.0.0.0 represents all local IPs).
  2. *:80: Listens on port 80 for any IPv4 and IPv6 IP.
  3. :::2000: Also listens on port 2000 for all local IPs. The difference from 0.0.0.0:2000 is that this represents an IPv6 address, while 0.0.0.0 represents all local IPv4 addresses.
  4. “:::”: The first two “::” are shorthand for “0:0:0:0:0:0:0:0”, equivalent to IPv6’s “0.0.0.0”. It represents all local IPv6 addresses, and the third “:” is the separator between IP and port.
  5. 127.0.0.1:8080: Listens on port 8080 for the loopback address. If a service only listens on the loopback address, it can only be accessed locally and cannot be accessed remotely via TCP/IP.
  6. ::1:9000: Listens on port 9000 for the IPv6 loopback address, where ::1 represents the IPv6 loopback address.
  7. 192.168.1.1:80: Listens on port 80 for the IP address 192.168.1.1.
  1. Foreign Address: External address communicating with the local port. The display rules are the same as for Local Address.
  2. State: Status, link state, with a total of 11 types. The state column has 12 possible states, the first 11 describe the TCP connection establishment’s three-way handshake and the TCP connection disconnection’s four-way handshake process.

Two important state parameters are:

  1. <span>LISTEN</span>: The server must first open a socket to listen, with the status being LISTEN. This is a connection request from a remote TCP port.
  2. <span>ESTABLISHED</span>: Represents an open connection where both parties can exchange or are already exchanging data. It indicates an open connection where data can be sent to the user.

To find special permissions, the default is root, -F indicates the delimiter.

If the third part is 0, print the first part, which is root.

awk -F: '{if($3==0) print $1}' /etc/passwd

To find users who can log in.

s=$( sudo cat /etc/shadow | grep '^[^:]*:[^
*!]' | awk -F: '{print $1}');for i in $s;do cat /etc/passwd | grep -v "/bin/false\|/nologin"| grep $i;done | sort | uniq | awk -F: '{print $1}'

To view currently connected SSH sessions, there are many methods, as follows:

root@mon0dy-ubuntu:/opt/collie# who -a
           system boot  2022-02-19 01:02
LOGIN      tty1         2022-02-18 17:02               821 id=tty1
LOGIN      ttyS0        2022-02-18 17:02               810 id=tyS0
root     - pts/0        2023-03-11 11:38   .          1300 (58.56.52.226)
root     - pts/1        2023-03-11 11:38 02:39        1319 (58.56.52.226)
           run-level 5  2022-02-18 17:03
           pts/2        2023-03-05 15:02             20164 id=ts/2  term=0 exit=0
           pts/3        2023-03-01 10:06             16760 id=ts/3  term=0 exit=0
           pts/4        2022-12-10 21:39              7303 id=ts/4  term=0 exit=0
           pts/5        2022-12-10 21:39              7338 id=ts/5  term=0 exit=0
root@mon0dy-ubuntu:/opt/collie# w
 14:18:45 up 385 days, 21:16,  4 users,  load average: 0.13, 0.16, 0.17
USER     TTY      FROM             LOGIN@   IDLE   JCPU   PCPU WHAT
root     pts/0    58.56.52.226     11:38    5.00s  0.19s  0.00s w
root     pts/1    58.56.52.226     11:38    2:39m 19.58s 19.55s top
root@mon0dy-ubuntu:/opt/collie# last -p now
root     pts/1        58.56.52.226     Sat Mar 11 11:38   still logged in
root     pts/0        58.56.52.226     Sat Mar 11 11:38   still logged in

wtmp begins Wed Mar  1 09:40:18 2023
root@mon0dy-ubuntu:/opt/collie# netstat -tnpa | grep 'ESTABLISHED.*sshd'
tcp        0      0 172.24.17.27:22         58.56.52.226:61764      ESTABLISHED 1318/sshd: root@not 
tcp        0     52 172.24.17.27:22         58.56.52.226:61763      ESTABLISHED 1263/sshd: root@pts 
root@mon0dy-ubuntu:/opt/collie# pgrep -af sshd
1165 /usr/sbin/sshd -D
1263 sshd: root@pts/0,pts/1
1318 sshd: root@notty    
root@mon0dy-ubuntu:/opt/collie# echo $SSH_CONNECTION
58.56.52.226 61763 172.24.17.27 22
root@mon0dy-ubuntu:/opt/collie# ss | grep ssh
tcp               ESTAB               0                    0                                                                                       172.24.17.27:ssh                                        58.56.52.226:61764                   
tcp               ESTAB               0                    0                                                                                       172.24.17.27:ssh                                        58.56.52.226:61763                   
root@mon0dy-ubuntu:/opt/collie# 

Logs

Ubuntu:/var/log/auth.log
Centos:/var/log/secure

Note that some logs may be packaged; auth.log is the secure log.

Successful Login

root@mon0dy-ubuntu:/opt/collie# cat /var/log/auth.log | grep "Accept"
Mar  5 13:41:06 mon0dy-ubuntu sshd[16791]: Accepted password for root from 58.56.52.226 port 22646 ssh2
Mar  5 13:41:07 mon0dy-ubuntu sshd[16843]: Accepted password for root from 58.56.52.226 port 22648 ssh2
Mar  5 13:41:26 mon0dy-ubuntu sshd[17180]: Accepted password for root from 58.56.52.226 port 22650 ssh2
Mar  5 14:00:31 mon0dy-ubuntu sshd[32618]: Accepted password for root from 58.56.52.226 port 6205 ssh2
Mar  5 14:00:31 mon0dy-ubuntu sshd[32641]: Accepted password for root from 58.56.52.226 port 6206 ssh2

Count Successful Logins

root@mon0dy-ubuntu:/var/log# cat /var/log/auth.log | grep "Accept" | perl -e 'while($_=&lt;&gt;){ /for(.*?)from/; print "$1\n";}'|sort|uniq -c|sort -nr
     26  root 

Normal Logout

<span>pam_unix(sshd:session): session closed</span> indicates a normal session closure, so just look for this feature in auth.log.

root@mon0dy-ubuntu:/var/log# cat /var/log/auth.log | grep "pam_unix(sshd:session): session closed"
Mar  5 14:01:11 mon0dy-ubuntu sshd[1010]: pam_unix(sshd:session): session closed for user root
Mar  5 14:01:54 mon0dy-ubuntu sshd[1918]: pam_unix(sshd:session): session closed for user root
Mar  5 14:02:25 mon0dy-ubuntu sshd[2606]: pam_unix(sshd:session): session closed for user root
Mar  5 14:03:49 mon0dy-ubuntu sshd[4296]: pam_unix(sshd:session): session closed for user root
Mar  5 14:06:06 mon0dy-ubuntu sshd[6988]: pam_unix(sshd:session): session closed for user root
Mar  5 14:06:38 mon0dy-ubuntu sshd[7633]: pam_unix(sshd:session): session closed for user root
Mar  5 14:06:40 mon0dy-ubuntu sshd[7712]: pam_unix(sshd:session): session closed for user root
Mar  5 14:06:48 mon0dy-ubuntu sshd[7908]: pam_unix(sshd:session): session closed for user root
Mar  5 14:06:57 mon0dy-ubuntu sshd[8132]: pam_unix(sshd:session): session closed for user root
Mar  5 14:07:05 mon0dy-ubuntu sshd[8328]: pam_unix(sshd:session): session closed for user root
Mar  5 14:07:13 mon0dy-ubuntu sshd[8519]: pam_unix(sshd:session): session closed for user root

Incorrect Login Password

When entering the wrong password multiple times, you may see messages like “message repeated 2 times” and “PAM 2 more authentication failures”, indicating consecutive incorrect password entries.

Mar 11 14:29:53 mon0dy-ubuntu sshd[10106]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=58.56.52.226  user=root
Mar 11 14:29:55 mon0dy-ubuntu sshd[10106]: Failed password for root from 58.56.52.226 port 23238 ssh2
Mar 11 14:30:05 mon0dy-ubuntu sshd[10106]: message repeated 2 times: [ Failed password for root from 58.56.52.226 port 23238 ssh2]
Mar 11 14:30:05 mon0dy-ubuntu sshd[10106]: Connection closed by authenticating user root 58.56.52.226 port 23238 [preauth]
Mar 11 14:30:05 mon0dy-ubuntu sshd[10106]: PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=58.56.52.226  user=root

If there are a large number of “Failed password” entries in a short time, it indicates a brute-force attack.

<span>cat /var/log/auth.log | grep "Failed password for root"</span>

Count Failed Login Usernames and Attempts

“invalid user” indicates that the user does not exist. The perl -e command is used to execute code, and similar functionality can be achieved with while read line; do; done. Here, it matches the values between “for” and “from”, which is root.

root@mon0dy-ubuntu:/var/log# cat /var/log/auth.log | grep "Failed password" | perl -e 'while($_=&lt;&gt;){ /for(.*?)from/; print "$1\n";}'|sort|uniq -c|sort -nr
     41  root 
      1  invalid user yogesh 
      1  invalid user wojcikowski 
      1  invalid user vinicius 
      1  invalid user ubnt 
      1  invalid user tarun 
      1  invalid user svcpunejenkins 
      1  invalid user sharan 
      1  invalid user sardari 
      1  invalid user sanchit 
      1  invalid user sadegh 
      1  invalid user ravinder 
      1  invalid user nishant 
      1  invalid user nisha 
      1  invalid user myproxyoauth 
      1  invalid user monitoring 
      1  invalid user michele 
      1  invalid user manmohan 
      1  invalid user majid 
      1  invalid user karthik 
      1  invalid user jhms 
      1  invalid user jeffery 
      1  invalid user jaya 
      1  invalid user ian 
      1  invalid user helen 
      1  invalid user harsh 
      1  invalid user esmat 
      1  invalid user cloud 
      1  invalid user amit 
      1  invalid user akshat 
      1  invalid user afshin 
      1  invalid user admin 
      1  invalid user abrar 
      1  invalid user a 
root@mon0dy-ubuntu:/var/log# 

Count Attackers’ IPs and Attempts

root@mon0dy-ubuntu:/var/log# cat /var/log/auth.log | grep "Failed password for" | grep "root" | grep -Po '(1\d{2}|2[0-4]\d|25[0-5]|[1-9]\d|[1-9])(\\.(1\d{2}|2[0-4]\d|25[0-5]|[1-9]\d|\d)){3}' |sort|uniq -c|sort -nr     25 213.87.10.3
      6 110.40.210.69
      4 58.56.52.226
      3 101.34.44.134
      2 190.14.158.76
      1 47.252.18.38
root@mon0dy-ubuntu:/var/log# 

Count multiple accounts’ IPs and attempts. Here are the root user and yogesh user; to add more, just append <span>|username</span>. Of course, we can also use awk. The grep -Po command matches the content between the specified two strings, and the regex is a standard way to match IPv4 addresses.

root@mon0dy-ubuntu:/var/log# cat /var/log/auth.log | grep "Failed password for" | grep "root\|yogesh" | grep -Po '(1\d{2}|2[0-4]\d|25[0-5]|[1-9]\d|[1-9])(\\.(1\d{2}|2[0-4]\d|25[0-5]|[1-9]\d|\d)){3}' |sort|uniq -c|sort -nr
     25 213.87.10.3
      6 110.40.210.69
      4 58.56.52.226
      3 101.34.44.134
      2 190.14.158.76
      1 47.252.18.38
      1 112.28.234.131

Change Password

We can see that the password for the git user has been changed.

Mar 11 17:18:42 mon0dy-ubuntu passwd[12484]: pam_unix(passwd:chauthtok): authentication failure; logname=root uid=1003 euid=0 tty= ruser= rhost=  user=git
Mar 11 17:18:50 mon0dy-ubuntu passwd[12660]: pam_unix(passwd:chauthtok): authentication failure; logname=root uid=1003 euid=0 tty= ruser= rhost=  user=git
Mar 11 17:19:13 mon0dy-ubuntu su[12417]: pam_unix(su:session): session closed for user git
Mar 11 17:19:22 mon0dy-ubuntu passwd[13410]: pam_unix(passwd:chauthtok): password changed for git

Switch User

We can see that the user switched from root to git.

Mar 11 17:15:38 mon0dy-ubuntu su[7951]: Successful su for git by root
Mar 11 17:15:38 mon0dy-ubuntu su[7951]: + /dev/pts/2 root:git
Mar 11 17:15:38 mon0dy-ubuntu su[7951]: pam_unix(su:session): session opened for user git by root(uid=0)
Mar 11 17:15:38 mon0dy-ubuntu su[7951]: pam_systemd(su:session): Cannot create session: Already running in a session
Mar 11 17:15:42 mon0dy-ubuntu su[7951]: pam_unix(su:session): session closed for user git

MySQL Logs

Normally, MySQL logs are located at /var/log/mysql/error.log, but the MySQL installed by Baota has a different log path. First, let’s find a segment.

Then search for: <span>grep -r "Skipping generation of RSA key pair as key files are present in data directory" /www/server</span>

The error log is found at: <span>/www/server/data/mon0dy-ubuntu.err</span>, and the slow query log is at: <span>/www/server/data/mysql-slow.log</span> (if slow query injection is utilized, the slow query log needs to be checked).

This time, the first input was correct, and the second and third were incorrect.

Check the logs, and it is correctly recorded.

Failed Login Usernames and Attempts

The four attempts here include two that I tested locally and two that failed remotely.

root@mon0dy-ubuntu:/www/server# cat /www/server/data/mon0dy-ubuntu.err | grep "Access denied for user" | grep "using password: YES" | awk -F "'" '{print $2}' | sort | uniq -c | sort -nr
4 wan
root@mon0dy-ubuntu:/www/server# 

View Failed Login IPs and Attempts

root@mon0dy-ubuntu:/www/server# cat /www/server/data/mon0dy-ubuntu.err | grep "Access denied for user" | grep "using password: YES" | awk -F "'" '{print $2}' | sort| uniq | while read line;do echo $line;cat /www/server/data/mon0dy-ubuntu.err | grep "Access denied for user" | grep "using password" | awk -F "'" '{print $4}' | sort | uniq -c | sort -nr; done
wan
3 localhost
258.56.52.226
root@mon0dy-ubuntu:/www/server# 

FTP Logs

Create a new FTP with Baota.

Log in, try a few incorrect passwords, then log in with the correct password.

root@mon0dy-ubuntu:~# netstat -pantu | grep ftp
tcp        0      0 172.24.17.27:39091      0.0.0.0:*               LISTEN      9975/pure-ftpd (IDL 
tcp        0      0 0.0.0.0:21              0.0.0.0:*               LISTEN      1091/pure-ftpd (SER 
tcp        0      0 172.24.17.27:21         58.56.52.226:57508      ESTABLISHED 10359/pure-ftpd (ID 
tcp        7      0 172.24.17.27:21         58.56.52.226:57497      ESTABLISHED 9975/pure-ftpd (IDL 
tcp6       0      0 :::21                   :::*                    LISTEN      1091/pure-ftpd (SER 

However, the so-called pureftpd.log was not found. After researching, it was found that pureftpd logs are located in /var/log/syslog, where the downloaded flag can be seen.

The initial few login failures.

Count Failed Login User Attempts

root@mon0dy-ubuntu:~# cat /var/log/syslog | grep 'Authentication failed for user' | cut -d "[" -f 3 | cut -d "]" -f 1 | sort | uniq -c | sort -nr
      5 mon
root@mon0dy-ubuntu:~# 

cat means to slice. The cut -d ‘delimiter’ -f fields (used with specific delimiters), -d: followed by the delimiter; -f: used with -d to split information into segments, and -f is used to extract the specified segment.

If not sliced, the first -f 3 takes the third segment, which is mon], and then cuts ] to take the first segment, which is to the left of ]: mon.

Count Failed Login User IP Attempts

First, slice to obtain the username, which is mon, and then slice to obtain the IP, as the format is ([email protected]), so we need to cut @ and ).

root@mon0dy-ubuntu:~# cat /var/log/syslog | grep 'Authentication failed for user' | cut -d "[" -f 3 | cut -d "]" -f 1 | sort | uniq | while read line;do echo $line;cat /var/log/syslog | grep $line | grep "Authentication failed for user" |cut -d "@" -f 2 | cut -d ')' -f 1 | sort | uniq -c | sort -nr; done
mon
      5 58.56.52.226
root@mon0dy-ubuntu:~# 

This aligns correctly.

Redis Logs

The configuration file is located at /www/server/redis/redis.conf, and the default logs are in /var/log/redis, but the Redis installed by Baota has logs at /www/server/redis/redis.log.

It can be seen that there is no password by default, as it is commented out.

The log save path is also written in the configuration file, and the default log level is notice, with debug, verbose, and warning as other levels.

The logs are essentially the command line output logs.

Change the log level to verbose, set the IP to 0.0.0.0, and change protected-mod to no, then restart.

Connect and execute some commands.

Looking back at the logs, it is found that it only records the IP, not the specific commands executed.

MongoDB Logs

By checking the status, you can quickly determine the location of the config.

Then you can obtain the logpath.

When installed with Baota, it is generally at /www/server/mongodb/log/config.log.

Then perform some operations locally.

After that, check the logs, focusing on the useful parts.

Connections before authentication:

{"t":{"$date":"2023-03-11T19:40:36.272+08:00"},"s":"I",  "c":"NETWORK",  "id":22943,   "ctx":"listener","msg":"Connection accepted","attr":{"remote":"58.56.52.226:8198","connectionId":3,"connectionCount":1}}

Authentication failure log: Authentication failed.

Incorrect password:

{"t":{"$date":"2023-03-11T19:34:47.264+08:00"},"s":"I",  "c":"ACCESS",   "id":20249,   "ctx":"conn2","msg":"Authentication failed","attr":{"mechanism":"SCRAM-SHA-1","speculative":false,"principalName":"admin","authenticationDatabase":"admin","remote":"58.56.52.226:19368","extraInfo":{},"error":"AuthenticationFailed: SCRAM authentication failed, storedKey mismatch"}}

Account error:

{"t":{"$date":"2023-03-11T19:40:49.427+08:00"},"s":"I",  "c":"ACCESS",   "id":20249,   "ctx":"conn3","msg":"Authentication failed","attr":{"mechanism":"SCRAM-SHA-1","speculative":false,"principalName":"root","authenticationDatabase":"admin","remote":"58.56.52.226:8198","extraInfo":{},"error":"UserNotFound: Could not find user \"root\" for db \"admin\""}}

Authentication succeeded:

{"t":{"$date":"2023-03-11T19:35:02.646+08:00"},"s":"I",  "c":"ACCESS",   "id":20250,   "ctx":"conn2","msg":"Authentication succeeded","attr":{"mechanism":"SCRAM-SHA-1","speculative":false,"principalName":"admin","authenticationDatabase":"admin","remote":"58.56.52.226:19368","extraInfo":{}}}

Some information about the connector: the connector’s machine version is Ubuntu 18, and the MongoDB version is 3.6.3.

{"t":{"$date":"2023-03-11T19:40:36.272+08:00"},"s":"I",  "c":"NETWORK",  "id":51800,   "ctx":"conn3","msg":"client metadata","attr":{"remote":"58.56.52.226:8198","client":"conn3","doc":{"application":{"name":"MongoDB Shell"},"driver":{"name":"MongoDB Internal Client","version":"3.6.3"},"os":{"type":"Linux","name":"Ubuntu","architecture":"x86_64","version":"18.04"}}}}

Check the number of times logged in as root.

root@mon0dy-ubuntu:/etc# cat /www/server/mongodb/log/config.log | grep "Could not find user" | awk -F '"' '{print $36}' | sort|uniq -c|sort -nr
      1 root
root@mon0dy-ubuntu:/etc# 

apt-get Logs

/var/log/apt/history.log records the history of apt-get commands, including what was installed, updated, and the specific software package versions.

/var/log/apt/term.log records the installation process.

Alternatives Logs

/var/log/alternatives.log

Software updates are used to manage different software with the same functionality or different versions of the same software, usually left during upgrades, recording the update time and specific replacement process.

dpkg Logs

Installation package manager logs record all installations, including those compiled and installed, not just those installed via apt-get, such as mysql57, which was compiled and installed via Baota.

Leave a Comment