Understand logs and manage logs in Linux

linux without background image

How to manage logs in Linux?

JOURNALD

 Journal is the system that collects and manages logs – syslog messages, kernel log messages, initial ram disk and early boot messages, messages from running services. In RHEL 7 journald implements the journal and replaces rsyslog as the default log management service.

The journald daemon collects data from all available sources and stores them in a binary format for easy and dynamic manipulation.

To manage journald we have journalctl command. Let us get the hang of journald.

    • List all logs from the oldest entry.
# journalctl
    • List all logs from the oldest entry in UTC.
# journalctl --utc
    • List all logs from current boot or older boot.
# journalctl -b
# journalctl -b -2
    • List total boots of the system that journald is aware of. Takes time if the persistent logging is enabled for a long time.
# journalctl --list-boots
    • List all logs for given time.
# journalctl --since yesterday
# journalctl --since 09:00 --until "1 hour ago"
# journalctl --since "2016-12-30" --until "2017-01-01 03:00"
# journalctl --since "2017-01-01 17:15:00"
    • List all logs from a unit.
# journalctl -u sshd
# journalctl -u sshd --since today
# journalctl -u sshd -u httpd --since today
    • List all logs by UID, GID, PID. For this we will use journal fields. To get the available values of a journal fileds use -F with journalctl.
# journalctl -F _UID
# journalctl -F _GID
# journalctl -F _PID
# journalctl _UID=1000
# journalctl _GID=100
# journalctl _PID=3224
    • List all logs of an executable file or device.
# journalctl /bin/bash
    • List kernel logs.
# journalctl -k
# journalctl -k -b -3
    • Journal Disk Usage.
# journalctl --disk-usage
    • Actively follow logs.
# journalctl -f
    • Display recent logs.
# journalctl -n
# journalctl -n 20
    • Display logs in JSON format.
# journalctl -u sshd -o json
# journalctl -u sshd -o json-pretty

Journald logs are not persistently stored by default. They are stored in /run/log/journal/ directory and are cleared on system reboot. To make the journald logs persistent, create /var/log/journal/ directory so that journald logs will be stored in it and will not get cleared on system reboot.

The main configuration file for journald is /etc/systemd/journald.conf.

# mkdir /var/log/journal
# chown root.systemd-journal /var/log/journal
# chmod 2755 /var/log/journal
# killall -USR1 systemd-journald

RSYSLOG

Rsyslog is Rocket-fast SYStem for LOG processing.

Journald forwards all logs to rsyslog which stores them in plain text files under /var/log/ directory.

Important log files:

  • /var/log/messages – Most syslog messages are logged here
  • /var/log/secure – Security and Authentication related logs
  • /var/log/maillog – Mail server related logs
  • /var/log/cron – Crontab related logs
  • /var/log/boot.log Booting related logs

The main configuration file for rsyslog is /etc/rsyslog.conf. In configuration file, rule lines are written to store log messages in various files. Each rule line consists of two parts – “selector field” and “action field”. Selector field is divided into two – “facility” and “priority”. Action field specifies what action must be taken for the matched rule – generally a file name in which to store the log for the matched rule.

There are many facilities –
auth, authpriv, cron, daemon, kern, lpr, mail, mark, news, security, syslog, user, uucp, local0 … local7.

There are several priorities –
emerg, alert, crit, error, warn, notice, info, debug.

To store logs related to mail facility in user defined /var/log/mymails.log file the rule should be written to main configuration file of rsyslog as follows.

# vim /etc/rsyslog.conf
mail.*	/var/log/mymails.log
ESC:wq
# systemctl reload rsyslog
# logger -p mail.info "Test mail log"
# tail /var/log/mymails.log