0012_Linux Log Files Overview

Log files on Ubuntu Server are crucial for monitoring system activities, troubleshooting issues, and ensuring security. They store information about system events, user activities, and application behavior.

Common Log Locations and Descriptions

  1. /var/log/syslog

    • Description: The main system log file records general system events, including boot messages, service start-ups, and shutdowns.

  2. /var/log/auth.log

    • Description: Logs all authentication-related events, such as login attempts, sudo usage, and SSH connections.

  3. /var/log/kern.log

    • Description: Contains messages from the kernel, including hardware errors, device detection, and kernel panics.

  4. /var/log/dmesg

    • Description: Captures kernel ring buffer messages, which include boot-time hardware detection and initialization logs.

  5. /var/log/apt/history.log

    • Description: Records a history of package installations, upgrades, and removals via the APT package manager.

  6. /var/log/apt/term.log

    • Description: Contains detailed terminal output from package installations, including errors and warnings during APT operations.

  7. /var/log/faillog

    • Description: Stores failed login attempts data, tracking how many failed attempts occurred for each user.

  8. /var/log/boot.log

    • Description: Logs the system boot process, including messages from various services and daemons that start during boot.

  9. /var/log/daemon.log

    • Description: Logs messages from system daemons (background services) that run in the background, such as cron, ntpd, etc.

  10. /var/log/messages (if available)

    • Description: Another general system log file, similar to /var/log/syslog, that, records various system events.

  11. /var/log/mail.log

    • Description: Log messages related to the mail system, including emails sent and received by the server.

  12. /var/log/ufw.log

    • Description: Logs firewall (Uncomplicated Firewall – UFW) events, including allowed and blocked connections.

  13. /var/log/apache2/ (for Apache web server)

    • Description: Contains logs for the Apache web server. Key files include:
      • access.log: Records all requests made to the server.
      • error.log: Logs server errors, warnings, and other diagnostic information.

  14. /var/log/mysql/ (for MySQL/MariaDB database server)

    • Description: Contains logs for the MySQL or MariaDB database server. Key files include:
      • error.log: Logs errors encountered by the database server.
      • slow-query.log: Records queries that take longer than a specified time to execute.

  15. /var/log/nginx/ (for Nginx web server)

    • Description: Contains logs for the Nginx web server. Key files include:
      • access.log: Records all requests made to the server.
      • error.log: Logs server errors, warnings, and other diagnostic information.

  16. /var/log/btmp

    • Description: Logs failed login attempts, similar to /var/log/faillog, but in a binary format.
    • Что можно посмотреть: Неудачные попытки входа в систему в бинарном формате.

  17. /var/log/wtmp

    • Description: Logs successful login attempts, user logouts, and system boot/reboots in a binary format.

  18. /var/log/cron.log

    • Description: Logs messages related to cron jobs, including execution times and any errors.

  19. /var/log/journal/

    • Description: Contains logs from systemd-journald, which includes detailed system logs in a binary format.

1. cat

  • Description: A simple utility to display the contents of files on the screen.

    Note : cat Outputs the file content without any filtering or formatting.

Example Usage:

cat /var/log/syslog

2. less

  • Description: An interactive file viewer that allows scrolling and searching within the file.

Useful Keys:

  • /string — Search for a string in the file.
  • n — Go to the next search match.
  • q — Quit the viewer.

Example Usage:

less /var/log/syslog

3. tail

  • Description: Displays the last lines of a file. Useful for viewing logs as they are updated.

Keys and Examples:

  • -f — Follow the file as it is updated
  • -n N — Show the last N lines 
tail -f /var/log/syslog
tail -n 50 /var/log/syslog

4. head

  • Description: Displays the first lines of a file.

Example Usage:

-n N — Show the first N lines:

 
head /var/log/syslog
head -n 50 /var/log/syslog

5. grep

  • Description: Searches for lines that match a specified pattern.

Keys and Examples:

  • -i — Ignore case;
     
  • -r — Recursive search through all files in a directory;
     
  • -A N — Show N lines after the match;
     
  • -B N — Show N lines before the match:
     
grep -i error /var/log/syslog
grep -r "connection" /var/log/
grep -A 5 "error" /var/log/syslog
grep -B 5 "error" /var/log/syslog

6. journalctl

  • Description: A tool to view logs managed by the systemd system. It provides convenient filtering and searching capabilities.

Keys and Examples / Ключи и примеры использования:

  • -b — Show logs from the current boot;
  • -u <service> — Show logs for a specific service;
  • -n – Show recent events in the system (-n 20, 20 last lines);
  • -f –  Show real-time events;
  • -p <priority> — Filter logs by priority level:

Priority Levels / Уровни приоритета:

  • emerg (0) — Emergency messages / the system is inoperable;
  • alert (1) — Alerts / Immediate intervention is required;
  • crit (2) — Critical errors / critical condition;
  • err (3) — Errors;
  • warning (4) — Warnings;
  • notice (5) — Notices / everything is fine, but you should pay attention;
  • info (6) — Informational messages;
  • debug (7) — Debug messages. 
  • -f — Follow the log in real-time;
  • --since and --until — Show logs from a specific period 
journalctl -b
journalctl -u ssh.service
journalctl -p 4 err
journalctl -f
journalctl --since "2024-09-01 00:00:00" --until "2024-09-02 00:00:00"
journalctl ---since yesterday
journalctl --since 09:00 --until now
journalctl --since 10:00 --until "1 hour ago"

Viewing logs of previous sessions

Using journalctl, you can view information about previous sessions in the system — in some cases, this can be useful.
However, remember that saving information about previous sessions is not supported by default in all Linux distributions. Sometimes, it needs to be activated.

To do this, open the journal.conf configuration file, find the [Journal] section in it, and change the value of the storage parameter to persistent:

sudo nano /etc/systemd/journald.conf
...
[Journal]
Storage=persistent

You can view the list of previous downloads using the command:

journalctl --list-boots
#Example output
 0 177999c1b6e746fca06a7ca53dd02155 Mon 2024-09-02 15:34:20 PDT—Tue 2024-09-03 22:04:07 PDT

Its output consists of four columns. The first indicates the serial number of the download, the second — its ID, and the third — the date and time. To view the log for a specific download, you can use the IDs from both the first and second columns:

journalctl -b 0
#or
journalctl -b 177999c1b6e746fca06a7ca53dd02155

You can view logs for any process by specifying its identification number (PID) in the journalctl command, for example:

journalctl _PID=789

The _UID and _GID filters are used to view logs of processes running on behalf of a specific user or group, respectively. Suppose we have a web server running on behalf of the www-data user. Let’s first determine the ID of this user and then use it:

id -u www-data
#Example output
35
journalctl _UID=35

To display a list of users and groups on the console about which there are entries in the logs, you can do this:

journalctl -F _UID
journalctl -F _GUID

Other filters can be used with the journalctl command. You can view the list of all available filters by running the command.

man systemd.journal-fields

You can also view logs for a process by specifying the path to it. Sometimes, you can get more detailed information this way (for example, view records for all child processes:

journalctl /usr/bin/docker

The –k or —dmesg option is used to view kernel messages:

journalctl -k
#or
journalctl --dmesg
# Kernel boot 2 messages
journalctl -k -b -2
 
Logging logs to standard output

By default, journalctl uses the external less utility to output log messages. In this case, applying standard utilities for processing text data (for example, grep) to them is impossible. This problem is easily solved: use the –no-pager option and all messages will be written to standard output:

journalctl --no-pager

Afterward, they can be transferred to other utilities for further processing or saved in a text file.

Choosing the output format

Using the -o option, you can convert log data into various formats, which facilitates their parsing and further processing, for example:

journalctl  -u nginx.service -o json

{ "__CURSOR" : "s=13a21661cf4948289c63075db6c25c00;i=116f1;b=81b58db8fd9046ab9f847ddb82a2fa2d;m=19f0daa;t=50e33c33587ae;x=e307daadb4858635", "__REALTIME_TIMESTAMP" : "1422990364739502", "__MONOTONIC_TIMESTAMP" : "27200938", "_BOOT_ID" : "81b58db8fd9046ab9f847ddb82a2fa2d", "PRIORITY" : "6", "_UID" : "0", "_GID" : "0", "_CAP_EFFECTIVE" : "3fffffffff", "_MACHINE_ID" : "752737531a9d1a9c1e3cb52a4ab967ee", "_HOSTNAME" : "desktop", "SYSLOG_FACILITY" : "3", "CODE_FILE" : "src/core/unit.c", "CODE_LINE" : "1402", "CODE_FUNCTION" : "unit_status_log_starting_stopping_reloading", "SYSLOG_IDENTIFIER" : "systemd", "MESSAGE_ID" : "7d4958e842da4a758f6c1cdc7b36dcc5", "_TRANSPORT" : "journal", "_PID" : "1", "_COMM" : "systemd", "_EXE" : "/usr/lib/systemd/systemd", "_CMDLINE" : "/usr/lib/systemd/systemd", "_SYSTEMD_CGROUP" : "/", "UNIT" : "nginx.service", "MESSAGE" : "Starting A high performance web server and a reverse proxy server...", "_SOURCE_REALTIME_TIMESTAMP" : "1422990364737973" }

A JSON object can be represented in a more structured and human-readable form by specifying the json-pretty or json-sse format:

journalctl -u nginx.service -o json-pretty

{
    "__CURSOR" : "s=13a21661cf4948289c63075db6c25c00;i=116f1;b=81b58db8fd9046ab9f847ddb82a2fa2d;m=19f0daa;t=50e33c33587ae;x=e307daadb4858635",
    "__REALTIME_TIMESTAMP" : "1422990364739502",
    "__MONOTONIC_TIMESTAMP" : "27200938",
    "_BOOT_ID" : "81b58db8fd9046ab9f847ddb82a2fa2d",
    "PRIORITY" : "6",
    "_UID" : "0",
    "_GID" : "0",
    "_CAP_EFFECTIVE" : "3fffffffff",
    "_MACHINE_ID" : "752737531a9d1a9c1e3cb52a4ab967ee",
    "_HOSTNAME" : "desktop",
    "SYSLOG_FACILITY" : "3",
    "CODE_FILE" : "src/core/unit.c",
    "CODE_LINE" : "1402",
    "CODE_FUNCTION" : "unit_status_log_starting_stopping_reloading",
    "SYSLOG_IDENTIFIER" : "systemd",
    "MESSAGE_ID" : "7d4958e842da4a758f6c1cdc7b36dcc5",
    "_TRANSPORT" : "journal",
    "_PID" : "1",
    "_COMM" : "systemd",
    "_EXE" : "/usr/lib/systemd/systemd",
    "_CMDLINE" : "/usr/lib/systemd/systemd",
    "_SYSTEMD_CGROUP" : "/",
    "UNIT" : "nginx.service",
    "MESSAGE" : "Starting A high performance web server and a reverse proxy server...",
    "_SOURCE_REALTIME_TIMESTAMP" : "1422990364737973"
}

In addition to JSON, log data can be converted to the following formats:

cat — only messages from logs without service fields;
export is a binary format suitable for exporting or backing up logs;
short — syslog output format;
short-iso — syslog output format with timestamps in ISO 8601 format;
short-monotonic — syslog output format with monotonic timestamp (monotonic timestamp);
short-precise — syslog output format with accurate timestamps (the time of events is specified with microsecond accuracy);
verbose – is the most detailed format for presenting data (including even those fields not displayed in other formats).

 
Logging management

Determining the current volume of logs

Over time, the volume of logs grows, and they take up more and more space on the hard disk. To find out the amount of logs currently available, use the command:

journalctl --disk-usage

Log Rotation

Log rotation is configured using the —vacuum–size and —vacuum–time options. The first of them sets the maximum allowable size for logs stored on disk (in our example, 5 GB):

sudo journalctl --vacuum-size=5G

The extra files will be automatically deleted as soon as the volume of logs exceeds the specified figure.
The –vacuum-time option works similarly. It sets the retention period for logs, after which they will be automatically deleted:

sudo journalctl --vacuum-time=1years

Configuring log rotation in the configuration file

Log rotation settings can also be specified in the configuration file /etc/systemd/journald.conf, which includes, among others, the following parameters:

SystemMaxUse= the maximum amount that logs can occupy on disk;
SystemKeepFree= the amount of free space that should remain on the disk after saving logs;
SystemMaxFileSize= the size of the log file, after which it should be deleted from disk;
RuntimeMaxUse= the maximum amount that logs can occupy in the file system /run;
RuntimeKeepFree= the amount of free space that should remain in the /run file system after saving logs;
RuntimeMaxFileSize= the size of the log file, after which it should be deleted from the /run file system.

Centralized storage of logs

One of the most common tasks in the work of a system administrator is to set up the collection of logs from several machines and then place them in a centralized repository.
Systems provides special components for solving this problem: systemd-journal-remote, systemd-journal-upload and systemd-journal-gatewayd.

Using the systemd-journal-remote command, you can receive logs from remote hosts and save them (the systemd-journal-gatewayd daemon must be running on each of these hosts), for example:

systemd-journal-remote −−url https://some.host:19531/

As a result of executing the above command, logs from the host https://some.host / will be saved in the var/log/journal/some.host/remote-some~host directory.journal .

Using the systemd-journal-remote command, you can also put the logs available on the local machine in a separate directory, for example:

journalctl -o export | systemd-journal-remote -o /tmp/dir -

The systemd-journal-upload command is used to upload logs from the local machine to the remote storage:

systemd-journal-upload --url https://some.host:19531/