Linux performance monitoring is very important for Linux administrator as well as for developers. It is one of the most important skills you can have the ability to monitor and optimize the performance of your systems. Whether you’re troubleshooting issues or simply trying to get the most out of your hardware, understanding the various tools and commands available to you is crucial.
In this post, we’ll take a look at some of the most essential command-line tools for monitoring Linux performance. From system uptime to CPU and memory usage, we’ll cover the key metrics you need to know to keep your systems running smoothly. We’ll also explore some advanced commands that can help you dig deeper into performance issues and identify bottlenecks.
Whether you’re new to Linux or a seasoned pro, this guide will give you the knowledge you need to effectively monitor and optimize your systems. So let’s dive in and take a closer look at the tools and commands that can help you keep your Linux systems running at their best.
Uptime
The uptime command is a simple command that shows the current time, the amount of time the system has been running, and the number of users currently logged in. It gives a quick overview of the system’s health and can help identify any potential issues.
The output of the command typically looks like this:
uptime
21:33:44 up 2:34, 3 users, load average: 10.00, 9.01, 1.05
- The first line shows the current time (21:33:44 in this example).
- The second line shows how long the system has been running (in this example, 2 hours and 34 minutes).
- The third line shows the number of users currently logged in e.g. 3 users are currently logged in. If this number is very big then we need to understand that many users are working on it and accordingly do our analysis.
- The last line shows the system load averages for the past 1, 5, and 15 minutes. Load average is a measure of the amount of work that a computer system is doing.
In the example above, the load averages are 10.00, 9.01, and 1.05, which indicates that the system has started doing significant things in last 5 mins onwards..
By running this command regularly and tracking the output, you can identify any trends that may indicate performance issues, such as a consistently high load average or a large number of logged-in users.
dmesg | tail
The dmesg command is used to examine or control the kernel message buffer. The kernel message buffer is a circular buffer in which the kernel stores messages that are generated by the kernel and by device drivers. These messages can include information about system initialization, hardware detection, and other events that take place while the system is running.
By default, dmesg will print all the messages that are currently stored in the kernel message buffer. By piping the output of dmesg to the tail command, we can display only the last 10 lines of the kernel message buffer. This can be useful for displaying recent kernel messages, such as those generated during system initialization or during the detection of new hardware.
The output of the command typically looks like this:
dmesg | tail
[13142.732107] usb 1-1.3: new high-speed USB device number 5 using xhci_hcd
[13142.846475] usb 1-1.3: New USB device found, idVendor=8087, idProduct=07dc, bcdDevice= 0.01
[13142.846478] usb 1-1.3: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[13142.847110] hub 1-1.3:1.0: USB hub found
[13142.847555] hub 1-1.3:1.0: 6 ports detected
[13142.904383] usb 1-1.3.3: new high-speed USB device number 6 using xhci_hcd
The above output shows some recent kernel messages that have been generated by the system. The messages include information about USB device detection and a new USB hub being found.
By using dmesg | tail, you can quickly view last 10 kernel messages and troubleshoot any issues related to hardware or driver problems.
vmstat
The vmstat command is a useful tool for monitoring system performance and identifying bottlenecks. It reports information about processes, memory, paging, block IO, traps, and CPU activity.
The vmstat command takes two optional arguments: the first is the sampling interval (in seconds), and the second is the number of samples.
For example, the command vmstat 1 will display a snapshot of the system’s performance every second, while vmstat 1 5 will display 5 snapshots of the system’s performance, each taken one second apart.
The output of the command typically looks like this:
vmstat 1 5
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 220216 7056048 0 0 3 4 7 1 6 3 3 2 94 0 0
0 0 220216 7056396 0 0 1856 0 29 0 5351 6812 6 5 89 0 0
0 0 220216 7054096 0 0 3072 0 48 0 3526 4215 2 1 97 0 0
0 0 220212 7054064 0 0 0 0 0 0 3413 4357 2 2 96 0 0
0 0 220212 7056180 0 0 3840 0 60 0 3299 4231 2 4 94 0 0
The first line of the output shows the header, which describes the different columns of the output:
procs: The number of processes in different states. ‘r’ is the number of running processes, ‘b’ is the number of blocked processes.memory: The amount of memory used by the system. ‘swpd’ is the amount of virtual memory used, ‘free’ is the amount of idle memory, ‘buff’ is the amount of memory used as buffers, and ‘cache’ is the amount of memory used as cache.swap: The amount of swap space used. ‘si’ is the amount of memory swapped in from disk, ‘so’ is the amount of memory swapped out to disk.io: The amount of input/output operations. ‘bi’ is the number of blocks received from a block device (block input), ‘bo’ is the number of blocks sent to a block device (block output).system: The number of system calls. ‘in’ is the number of interrupts per second, ‘cs’ is the number of context switches per second.cpu: The percentage of CPU utilization. ‘us’ is the percentage of CPU time spent on user processes, ‘sy’ is the percentage of CPU time spent on system processes, ‘id’ is the percentage of time the CPU is idle, ‘wa’ is the percentage of time the CPU is waiting for I/O to complete, ‘st’ is the percentage of time stolen from a virtual machine.
By using vmstat, you can quickly identify bottlenecks and potential issues with the system’s performance. For example, high values for ‘si’ and ‘so’ may indicate that the system is running low on memory and is swapping frequently. High values for ‘us’ or ‘sy’ may indicate that the CPU is overutilized, and high values for ‘wa’ may indicate that there are I/O bottlenecks in the system.
Additionally, vmstat also shows the number of running and blocked processes, which can help identify if there are any process related issues such as a high number of blocked processes indicating a deadlock.
mpstat -P ALL 1
The mpstat command is used to monitor the performance of individual CPUs or cores in a multi-processor system. It provides detailed information about the usage of each CPU or core, including the percentage of time spent in user mode, system mode, and idle mode.
There are several other options that can be used with mpstat, some of which are:
-A: Displays statistics for all processors or cores, including individual and cumulative statistics-u: Shows the statistics of CPU time spent in user mode, system mode, nice mode, etc-I SUM: Provide a summary of all interrupt types-I 5,10: Provide statistics of specific interrupt types, in this case 5 and 10-P 0-3: Provide statistics of specific processors or cores, in this case 0 to 3.-r: Displays statistics for disk I/O-t: Displays timestamp with each report
The mpstat -P ALL 1 command will display the usage statistics for all CPUs or cores every second.
The output of the command typically looks like this:
mpstat -P ALL 1
Linux 4.19.0-14-amd64 (server) 01/26/2023 _x86_64_ (2 CPU)
18:02:35 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle
18:02:37 all 1.15 0.00 2.54 0.00 0.00 0.00 0.00 0.00 96.31
18:02:37 0 0.00 0.00 7.80 0.00 0.00 0.00 0.00 0.00 92.20
18:02:37 1 1.50 0.00 3.20 0.00 0.00 0.00 0.00 0.00 95.30
18:02:37 2 1.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.50
18:02:37 3 0.00 0.00 3.10 0.00 0.00 0.00 0.00 0.00 96.90
18:02:37 4 0.00 0.00 1.60 0.00 0.00 0.00 0.00 0.00 98.40
18:02:37 5 1.60 0.00 0.00 0.00 0.00 0.00 0.00 0.00 98.40
18:02:37 6 3.10 0.00 1.50 0.00 0.00 0.00 0.00 0.00 95.40
18:02:37 7 1.50 0.00 3.10 0.00 0.00 0.00 0.00 0.00 95.40
The first line of the output shows the header, which describes the different columns of the output:
CPU: The CPU number%usr: The percentage of CPU time spent in user mode%nice: The percentage of CPU time spent on low-priority processes%sys: The percentage of CPU time spent in system mode%iowait: The percentage of CPU time spent waiting for I/O operations to complete%irq: The percentage of CPU time spent handling interrupts%soft: The percentage of CPU time spent handling softirqs%steal: The percentage of CPU time spent in involuntary wait by the virtual CPU or hypervisor waiting for the real CPU%guest: The percentage of CPU time spent running a virtual CPU for guest operating systems under the control of the Linux kernel%gnice: The percentage of CPU time spent by guest virtual CPUs on low-priority processes.%idle: The percentage of CPU time spent idle
By using mpstat, you can quickly identify which CPUs or cores are the most heavily utilized, and identify any potential performance bottlenecks. It can also help you to identify if any specific process is causing high CPU usage, and if there are any issues related to I/O operations, interrupts, or softirqs.
One important thing to notice is that mpstat shows the average CPU usage over a certain period of time, so it might not capture short burst of high CPU usage.
In addition, mpstat also shows the system wide averages, as well as per-cpu usage, which can be useful in identifying any imbalance or issues related to CPU scheduling.
Overall, mpstat is a useful command for monitoring the performance of individual CPUs or cores and is particularly useful in multi-processor systems. It provides detailed information about the usage of each CPU or core which can be used to quickly identify and troubleshoot performance issues related to CPU usage.
pidstat
The pidstat command is used to monitor the performance of individual processes on a Linux system. It provides detailed information about the usage of each process, including the percentage of CPU time spent in user mode, system mode, and the number of major and minor page faults.
The pidstat 1 command will display the usage statistics for all processes every second.
The output of the command typically looks like this:
Linux 4.19.0-14-amd64 (server) 01/26/2023 _x86_64_ (2 CPU)
01:23:24 PM UID PID %usr %system %guest %CPU CPU minflt/s majflt/s VSZ RSS cmd
01:23:25 PM 0 1 0.00 0.00 0.00 0.00 0 0.00 0.00 15292 868 systemd
01:23:25 PM 0 2 0.00 0.00 0.00 0.00 0 0.00 0.00 0 0 kthreadd
01:23:25 PM 0 4 0.00 0.00 0.00 0.00 0 0.00 0.00 0 0 kworker/0:0H
01:23:25 PM 0 6 0.00 0.00 0.00 0.00 0 0.00 0.00 0 0 mm_percpu_wq
01:23:25 PM 0 7 0.00 0.00 0.00 0.00 0 0.00 0.00 0 0 ksoftirqd/0
The first line of output shows the header,, which describes the different columns of the output:
UID: The user ID of the process ownerPID: The process ID of the process%usr: The percentage of CPU time spent in user mode by the process%system: The percentage of CPU time spent in system mode by the process%guest: The percentage of CPU time spent running a virtual CPU for guest operating systems under the control of the Linux kernel%CPU: The percentage of CPU time spent by the processCPU: The CPU or core on which the process is runningminflt/s: The number of minor page faults per secondmajflt/s: The number of major page faults per secondVSZ: The virtual memory size of the process in bytesRSS: The resident set size of the process in bytescmd: The command that started the process
By using pidstat, you can quickly identify which processes are the most heavily utilizing CPU, memory and other resources. It can also help you to identify any page faults that could be causing performance issues.
It is also possible to monitor specific process by specifying the PID with pidstat -p PID 1
pidstat is a useful command for monitoring the performance of individual processes and is particularly useful in identifying performance issues related to specific process. It provides detailed information about the usage of each process which can be used to quickly identify and troubleshoot performance issues related to CPU, memory and other resources usage.
iostat
The iostat command is used to monitor input/output (I/O) statistics on a Linux system. It provides detailed information about the usage of the disk drives, including the number of read and write operations per second, the number of sectors read and written per second, and the average time it takes for each operation.
Some of the other options that can be used with the iostat command are:
-c: Display the utilization statistics for CPU-d: Display the disk statistics-n: Display the statistics for the number of devices specified-p: Display the statistics for the specified device(s)-t: Display the timestamp with the statistics-x : Display extended statistics-y: Display the statistics in a format that is easy to parse by scripts.-z : Display only devices that are in a running state and omits any devices that are in a stopped state.
For example, the command "iostat -x -p sda 1" will display the extended statistics for the device sda every 1 second & the command “iostat -xz 1" will display the I/O statistics every second.
The output of the command typically looks like this:
Linux 4.19.0-14-amd64 (server) 01/26/2023 _x86_64_ (2 CPU)
Device: rrqm/s wrqm/s r/s w/s rkB/s wkB/s avgrq-sz avgqu-sz await r_await w_await svctm %util
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
The first line of the output shows the header, which describes the different columns of the output:
Device: The name of the disk driverrqm/s: The number of read requests merged per secondwrqm/s: The number of write requests merged per secondr/s: The number of read operations per secondw/s: The number of write operations per secondrkB/s: The number of kilobytes read per secondwkB/s: The number of kilobytes written per secondavgrq-sz: The average size of a request in sectorsavgqu-sz: The average queue length of the requestsawait: The average time (in milliseconds) for I/O requests issued to the device to be servedr_await: The average time (in milliseconds) for read requests to be servedw_await: The average time (in milliseconds) for write requests to be servedsvctm: The average service time (in milliseconds) for I/O requests that were issued to the device%util: The percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device)
By using iostat, you can quickly identify which devices are the most heavily utilized, and if there are any issues related to I/O operations. It can also help you to identify if any specific process is causing high I/O usage, and if there are any issues related to disk drives.
iostat is a useful command for monitoring the performance of disk drives and is particularly useful in identifying performance issues related to I/O operations. It provides detailed information about the usage of disk drives which can be used to quickly identify and troubleshoot performance issues related to disk drives and I/O operations.
sar
The sar (System Activity Report) command is a powerful tool for monitoring and analyzing system performance on Linux systems. It is used to collect, report, and save system activity information such as CPU usage, memory usage, I/O activity, network activity, and more.
The sar command is typically used with options that specify the type of system activity to monitor, the interval at which to collect data, and the file or files in which to save the data.
For example, the sar -u 1 command will display the CPU usage statistics every 1 second.
Some of the other options that can be used with the sar command are:
-b: Display the I/O and transfer rate statistics-r: Display the memory and swap usage statistics-W: Display the paging statistics-P: Display the statistics for the specified processor(s) or core(s)-A: Display all the available statistics-u : Display system usages statistics-n : Display network statistics
It is also possible to specify a date and time range for the data that sar should collect and report, by using options like -s and -e. For example, the command sar -u -s 10:00:00 -e 11:00:00 will display the CPU usage statistics for the time period between 10 am and 11 am.
The sar -n DEV 1 command will display the network device statistics every 1 second.
The sar -n TCP,ETCP 1 command will display the TCP and TCP extension statistics every 1 second.
free
The free command provides a quick way to check the system’s memory usage and can be useful for troubleshooting performance issues related to memory usage.
Here are some of the other options that can be used with the free command:
-b: Display the memory information in bytes-g: Display the memory information in gigabytes-h: Display the memory information in human-readable format (e.g., 1K, 234M, 2G)-k: Display the memory information in kilobytes-l: Display the information about the locked memory-m: Display the information about the amount of free and used memory in the system-o: Display the memory information without any buffer/cache information-t: Display the total memory usage, including the memory used by the buffers and cache
For example, the command free -h will display the memory information in human-readable format.
The free -m command is used to display the amount of free and used memory in the system. The -m option is used to display the memory information in megabytes, rather than in bytes.
Here’s an example of the output of the free -m command:
total used free shared buff/cache available
Mem: 7864 6913 731 123 220 829
Swap: 8191 0 8191
The output is divided into three sections:
- The first line shows the total, used, free, shared, buff/cache, and available memory in the system.
- The second line shows the total, used, and free swap space in the system.
The ‘total’ column shows the total amount of memory installed in the system. ‘used’ column shows the amount of memory currently in use by the system, ‘free’ column shows the amount of memory that is currently not being used by the system, ‘shared’ column shows the amount of memory that is being shared by multiple processes, ‘buff/cache’ column shows the amount of memory that is being used as buffers and cache, and ‘available’ column shows the amount of memory that is available for use.
top
The top command is a real-time system monitoring tool that displays a dynamic, scrolling view of the processes running on a Linux system. It provides detailed information about the performance of the system, including CPU usage, memory usage, and running processes.
Here’s an example of the output of the top command:
top - 11:38:54 up 3 days, 7:03, 2 users, load average: 0.00, 0.01, 0.05
Tasks: 244 total, 2 running, 242 sleeping, 0 stopped, 0 zombie
%Cpu(s): 0.2 us, 0.1 sy, 0.0 ni, 99.7 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 8077248 total, 7097368 used, 979880 free, 60788 buffers
KiB Swap: 8191996 total, 0 used, 8191996 free. 161720 cached Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
839 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u2:2
838 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u2:1
837 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u2:0
836 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/u2:3
835 root 20 0 0 0 0 S 0.0 0.0 0:00.00 kworker/
The first line of the top output shows the current time, the system uptime, the number of users logged in, and the system’s load averages for the past 1, 5, and 15 minutes.
The second line shows the total number of processes running on the system, as well as the number of processes in different states (running, sleeping, stopped, and zombie).
The third line shows the CPU usage statistics, including the percentage of CPU time spent in user mode, system mode, and idle mode, as well as the percentage of CPU time spent waiting for I/O operations to complete.
The fourth line shows the memory usage statistics, including the total, used, and free physical memory, as well as the amount of memory being used as buffers and cache.
The fifth line shows the swap memory usage statistics, including the total, used, and free swap space, as well as the amount of memory being used as cache.
The remaining lines show a list of the processes running on the system, including the process ID, user, priority, virtual memory usage, resident memory usage, shared memory usage, and CPU usage. The process that is currently using the most CPU time is listed at the top of the list.
The top command updates the information in real-time, so the output will change as the system’s performance changes. You can press q to exit the top command.
The top command is a powerful tool for monitoring the performance of a Linux system. It can be used to identify which processes are consuming the most resources, and can help to identify performance bottlenecks. The top command can be used with a variety of options such as -d to specify the delay time, -n to specify the number of iterations and -u to specify a particular user’s process.
Another useful option is -p which allows you to monitor the specific process by giving the process id. For example, top -p 1234 will only show the process with ID 1234. The -H option will also show all the threads of the process, in addition to the parent process.
You can also sort the process list by different criteria like memory usage, CPU usage, process name etc by pressing different keys like M for memory, P for CPU, N for process name etc.
In addition to these options, there are many other options available for the top command that can be used to customize the output and display the information that is most relevant for your specific use case.
Conclusion
In conclusion, monitoring the performance of a Linux system is essential for ensuring that the system is running at optimal performance. The commands we discussed in this post – uptime, dmesg, vmstat, mpstat, pidstat, iostat, free, sar, and top – are all powerful tools that provide detailed information about the system’s performance. By understanding and utilizing these commands, you can quickly identify performance bottlenecks and take steps to optimize the system’s performance. Whether you are a system administrator, developer, or power user, these commands are essential for understanding the performance of your Linux systems. Remember that these commands are just the tip of the iceberg in terms of Linux performance monitoring and there are many other tools and techniques available for monitoring and optimizing Linux systems.
Reference