Linux disk performance monitoring

Top 15 tools to monitor disk IO performance with examples

Table of Contents

Related searches: Linux Disk Usage. Top Storage monitoring tools. How to monitor Disk IO performance with examples. How to check disk read write usage on Linux. Check and monitor disk IO statistics and disk stats in Linux using iostat, vmstat and other tools. How to monitor disk IO by process ID for specific process in Linux. Storage monitoring tools. Get disk read write operation details in Linux with examples.

In my earlier article I gave you an overview of different disk types (HDD, SSD, Optical Disks) and disk interface types (SATA, IDE, SAS, SCSI..) in details with pros and cons. Now in this article I will show you various tools along with examples to monitor disk IO performance in Linux environment.

1. iostat — Report Disk IO Statistics

isotat is part of sysstat rpm. You can install sysstat using yum or any other tool depending upon your environment.

iostat summarises per-disk I/O statistics, providing metrics for IOPS, throughput, I/O request times, and utilization. It can be executed by any user, and is typically the first command used to monitor disk io performance and investigate disk IO statistics and issues at the command line.

iostat provides many options for customizing the output. A useful combination is -dxz 1 , to show disk utilization only ( -d ), extended columns ( -x ), skipping devices with zero metrics ( -z ), and per-second output.

These columns summarize the workload applied. You can check iostat man page to understand the meaning of each column.

2. vmstat — Report virtual memory statistics

vmstat is another monitoring tool which is part of procps-ng rpm. It is most likely possible that procps-ng is installed by default on your Linux node or else you can also install it manually using yum

vmstat reports information about processes, memory, paging, block IO, traps, disks and cpu activity. Here we will use vmstat to monitor disk IO performance in Linux using -d for 1 second with 1 second interval.

To get summary disk IO statistics about disk activity

Follow man page of vmstat to get the complete list of supported arguments using which you can monitor your system resource.

3. iotop — Monitor disk IO Speed

iotop specialises in getting disk stats and is part of iotop rpm. You can install iotop using yum or any other tool depending upon your environment.

iotop watches disk I/O usage information output by the Linux kernel (requires 2.6.20 or later) and displays a table of current I/O usage by processes or threads on the system.

With —only iotop will only show processes or threads actually doing I/O, instead of showing all processes or threads so you can check and monitor disk IO performance.

4. nmon — Monitor System Stats

nmon is not available in the default repository of RHEL/CentOS. You can install if from the EPEL repository. To install the entire EPEL repo on RHEL/CentOS 7

and to install EPEL repo on CentOS/RHEL 8

Next you can install nmon using yum command

Alternatively you can also install nmon tool manually

nmon can display the CPU, memory, network, disks (mini graphs or numbers), file systems, NFS, top processes, resources (Linux version & processors) and on Power micro-partition information.

Execute nmon from the terminal

To only display disk usage statistics press D

To display disk usage statistics with graph press d

5. atop — Advanced System & Process Monitor

You can again install atop using yum (assuming you had installed EPEL repo) or alternatively you can manually install atop to check and monitor disk IO performance in Linux.

The program atop is an interactive monitor to view the load on a Linux system. You can use atop to monitor disk IO by process. It shows the occupation of the most critical hardware resources (from a performance point of view) on system level, i.e. cpu, memory, disk and network.

Now press shift + d for disk activity
Then press c for full command name..

6. collectl — Collects data that describes the current system status

collectl is not available in the internal repo of CentOS/RHEL, so either you can install EPEL repo itself and then use yum to install collectl or manually install it.

collectl will collect data that describes the current system status. We can use collectl to check and monitor disk IO performance in Linux. The following command reports CPU and disk IO statistics. Here c and d represent CPU and Disk.

Follow man page of collectl to see all the supported options.

7. sar — Monitor Disk IO Performance

sar is another famous and widely used too and is part of sysstat rpm. You can install sysstat using yum or any other tool depending upon your environment.

sar is a powerful tool which can be used to monitor all the system resources. But to stick to this article’s topic, we will use -d to monitor disk IO performance for 1 second with an interval of 1 second.

You can check this article to understand about all these columns and other options supported with sar and ksar .

8. blktrace — generate traces of the Disk I/O

blktrace is part of blktrace rpm which should be available in your default repository. You can install it using yum command or other tools based on your environment.

blktrace is a specialized utility for tracing block I/O events

Multiple event lines are printed for each I/O. You can also monitor disk IO by process. The columns are:

• Device major, minor number
• CPU ID
• Sequence number
• Action time, in seconds
• Process ID
• Action identifier (see blkparse(1)): Q == queued, G == get request, P == plug, M == merge, D == issued, C == completed, etc.
• RWBS description (see the “rwbs” section earlier in this chapter): W == write, S == synchronous, etc.
• Address + size [device]

At the end of execution you will get a summary of disk statswith throughput details of read and write operations.

Perf tools are performance analysis tools based on Linux perf_events (aka perf) and ftrace .

9. perf-tools: iolatency

iolatency — summarize block device I/O latency as a histogram and is part of perf-tools , you can download iolatency script from below location

Provide executable permission to the downloaded script

Next execute the script with -Q option which enables queued time. The -Q option includes the block I/O queued time, by tracing based on block_rq_insert instead of block_rq_issue :

Here the disk latency is between 16 — 32 milliseconds.

You can get some more examples on iolatency on the official github page

10. perf-tools: iosnoop — monitor disk IO by process

iosnoop is another tool from perf-tools . You can download iosnoop script from the below path

Provide executable permission to the downloaded script

It will trace disk I/O with details including latency. Here using -p I have provided the PID of cp command for which ionoop will show the latency value to monitor disk IO by process ID.

For more examples and list of supported options you can check the official github page

11. BPF Tools

Traditional Performance tools provide some insight for storage I/O, including IOPS rates, average latency and queue lengths, and I/O by process.

BPF tracing tools can provide additional insight for disk stats and can be used for disk IO performance in Linux.

Credits: BPF Performance Tools

11.1 BPF Pre-requisite

You must install below rpms to be able to use BPF

Here bcc can be installed from the system repository. Make sure bcc rpm version matches the loaded kernel version. So you can install both rpms together

Next install bpftools and bpftrace to be able to monitor disk IO performance

This will download the repo file and place at /etc/yum.repos.d/bpftools.repo . Now you can install bpftrace and bpftools using yum

11.2 biolatency

biolatency is a BCC and bpftrace tool to show disk IO statistics with latency as a histogram. The term device latency refers to the time from issuing a request to the device, to when it completes, including time spent queued in the operating system. We can use biolatency to check disk stats and monitor disk IO performance.

The -D option in biolatency shows histograms for disks stats separately, helping you see how each type performs.

This output shows disk IO statistics for /dev/sda , an internal disk, with I/O latency often between 128 and 4095 microseconds

11.3 biosnoop

biosnoop is a BCC and bpftrace tool that prints a one-line summary for each disk IO statistics.
It prints a line of output for each disk IO statistics, with details including latency (time from device issue to completion).

This allows you to examine disk IO performance in more detail:

The biosnoop columns are:

• TIME(s): I/O completion time in seconds
• COMM: Process name, if cached
• PID: Process ID, if cached
• DISK: Storage device name
• T: Type: R == reads, W == writes
• SECTOR: Address on disk in units of 512-byte sectors
• BYTES: Size of the I/O
• LAT(ms): Duration of the I/O from device issue to device completion

11.4 biotop

biotop is a BCC tool that is similar to top but to get disk stats and disk IO statistics in Linux.

• -C: Don’t clear the screen
• -r ROWS: Number of rows to print

Top of the list are kworker threads initiating writes: this is background write flushing, and the real process that dirtied the pages is not known at this point

11.5 bitesize

bitesize is a BCC and bpftrace tool to show the size of disk IO statistics.

This output shows the disk stats that both the kworker thread and jbd2 are calling I/O mostly in the 0 to 1 Kbyte range.

11.6 ext4slower

ext4slower traces the ext4 file system and checks disk IO performance for ext4 file system, and then only prints those disk stats that exceed a threshold.

Similar tools exist in bcc for other file systems: btrfsslower , xfsslower , and zfsslower . There is also fileslower , which works at the VFS layer and traces everything (although at some higher overhead).

Here I have a dd command running on another terminal to simulate artificial disk IO statistics

Conclusion

In this tutorial we learned about various Linux tools which can be used to check disk usage by different processes and monitor disk IO Performance. Disk read write plays very important role in how application data is processed between RAM and Disk so it is very important that you have disk with good I/O speed and RPM. I would recommend reading about different available disk types and disk interface types

This would give you an idea on choosing the type of drive which suits your requirement. In production environment we mostly prefer HDD over SSD due to cost and many other factors but for laptops and desktops mostly SSD are used. But now a days even in production environment is starting to move to SSD to support cloud environment.

Lastly I hope the steps from the article to monitor disk IO performance, disk stats and disk IO statistics in Linux was helpful. So, let me know your suggestions and feedback using the comment section.

References

I have used below external references for this tutorial guide
BPF Performance Tools

Related Posts

Didn’t find what you were looking for? Perform a quick search across GoLinuxCloud

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

For any other feedbacks or questions you can either use the comments section or contact me form.

Thank You for your support!!

Источник

20 Command Line Tools to Monitor Linux Performance

It’s really a very tough job for every System or Network administrator to monitor and debug Linux System Performance problems every day.

After being a Linux Administrator for 10 years in the IT industry, I came to know that how hard is to monitor and keep systems up and running.

For this reason, we’ve compiled the list of Top 20 frequently used command line monitoring tools that might be useful for every Linux/Unix System Administrator.

These commands are available under all flavors of Linux and can be useful to monitor and find the actual causes of performance problems. This list of commands shown here is very enough for you to pick the one that is suitable for your monitoring scenario.

Linux Command Line Monitoring

1. Top – Linux Process Monitoring

Linux Top command is a performance monitoring program that is used frequently by many system administrators to monitor Linux performance and it is available under many Linux/Unix-like operating systems.

The top command is used to display all the running and active real-time processes in an ordered list and updates it regularly. It displays CPU usage, Memory usage, Swap Memory, Cache Size, Buffer Size, Process PID, User, Commands, and much more.

It also shows high memory and cpu utilization of running processes. The top command is much useful for system administrators to monitor and take corrective action when required. Let’s see the top command in action.

Check Linux Running Processes

For more examples of Top command read: 12 TOP Command Examples in Linux

2. VmStat – Virtual Memory Statistics

Linux VmStat command is used to display statistics of virtual memory, kernel threads, disks, system processes, I/O blocks, interrupts, CPU activity, and much more.

Install VmStat in Linux

By default vmstat command is not available under Linux systems you need to install a package called sysstat (a powerful monitoring tool) that includes a vmstat program.

The common usage of vmstat command format is.

Vmstat System Monitoring Tool

3. Lsof – List Open Files

The lsof command is used in many Linux/Unix-like systems to display a list of all the open files and the processes. The open files included are disk files, network sockets, pipes, devices, and processes.

One of the main reasons for using this command is when a disk cannot be unmounted and displays the error that files are being used or opened. With this command, you can easily identify which files are in use.

The most common format for lsof command is.

List Open Files in Linux

For more usage and examples, read: 10 lsof Command Examples in Linux

4. Tcpdump – Network Packet Analyzer

The tcpdump command is one of the most widely used command-line network packet analyzer or packets sniffer programs that is used to capture or filters TCP/IP packets that are received or transferred on a specific interface over a network.

It also provides an option to save captured packages in a file for later analysis. tcpdump is almost available in all major Linux distributions.

Tcpdump – Network Packet Analyzer

5. Netstat – Network Statistics

The netstat is a command-line tool for monitoring incoming and outgoing network packets statistics as well as interface statistics. It is a very useful tool for every system administrator to monitor network performance and troubleshoot network-related problems.

Netstat – Monitor Linux Network Connections

While in present-day netstat has been deprecated in favor of the ss command, you may still discover netstat in your networking toolkit.

6. Htop – Linux Process Monitoring

htop is a much advanced interactive and real-time Linux process monitoring tool, which is much similar to Linux top command but it has some rich features like a user-friendly interface to manage processes, shortcut keys, vertical and horizontal views of the processes, and much more.

Htop – Linux System Process Viewer

htop is a third-party tool, which doesn’t come with Linux systems, you need to install it using your system package manager tool. For more information on htop installation read our article – Install Htop (Linux Process Monitoring) in Linux.

7. Iotop – Monitor Linux Disk I/O

iotop is also much similar to top command and htop program, but it has an accounting function to monitor and display real-time Disk I/O and processes.

iotop tool is much useful for finding the exact process and high used disk read/writes of the processes.

Install Iotop in Linux

By default, the iotop command is not available under Linux and you need to install it as shown.

The common usage of iotop command format is.

iotop – Monitor Linux Disk IO Usage

8. Iostat – Input/Output Statistics

iostat is a simple tool that will collect and show system input and output storage device statistics. This tool is often used to trace storage device performance issues including devices, local disks, remote disks such as NFS.

Install Iostat in Linux

To get the iostat command, you need to install a package called sysstat as shown.

The common usage of iostat command format is.

iostat – Monitor Disk IO Statistics

9. IPTraf – Real-Time IP LAN Monitoring

IPTraf is an open-source console-based real-time network (IP LAN) monitoring utility for Linux. It collects a variety of information such as IP traffic monitor that passes over the network, including TCP flag information, ICMP details, TCP/UDP traffic breakdowns, TCP connection packet, and byte counts.

It also gathers information of general and detailed interface statistics of TCP, UDP, IP, ICMP, non-IP, IP checksum errors, interface activity, etc.

IPTraf IP Network Monitor

10. Psacct or Acct – Monitor User Activity

psacct or acct tools are very useful for monitoring each user’s activity on the system. Both daemons run in the background and keep a close watch on the overall activity of each user on the system and also what resources are being consumed by them.

These tools are very useful for system administrators to track each user’s activity like what they are doing, what commands they issued, how much resources are used by them, how long they are active on the system etc.

psacct – Monitor Linux User Activities

For installation and example usage of commands read the article on Monitor User Activity with psacct or acct

11. Monit – Linux Process and Services Monitoring

Monit is a free open source and web-based process supervision utility that automatically monitors and manages system processes, programs, files, directories, permissions, checksums, and filesystems.

It monitors services like Apache, MySQL, Mail, FTP, ProFTP, Nginx, SSH, and so on. The system status can be viewed from the command line or using its own web interface.

Monit Monitor Linux System

12. NetHogs – Monitor Per Process Network Bandwidth

NetHogs is an open-source nice small program (similar to Linux top command) that keeps a tab on each process network activity on your system. It also keeps a track of real-time network traffic bandwidth used by each program or application.

Nethogs Monitor Network Traffic in Linux

For installation and usage, read our article: Monitor Linux Network Bandwidth Using NetHogs

13. iftop – Network Bandwidth Monitoring

iftop is another terminal-based free open source system monitoring utility that displays a frequently updated list of network bandwidth utilization (source and destination hosts) that passing through the network interface on your system.

iftop is considered for network usage, what ‘top‘ does for CPU usage. iftop is a ‘top‘ family tool that monitors a selected interface and displays a current bandwidth usage between two hosts.

iftop – Network Bandwidth Monitoring

For installation and usage, read our article: iftop – Monitor Network Bandwidth Utilization

14. Monitorix – System and Network Monitoring

Monitorix is a free lightweight utility that is designed to run and monitor system and network resources as many as possible in Linux/Unix servers.

It has a built-in HTTP web server that regularly collects system and network information and displays them in graphs. It Monitors system load average and usage, memory allocation, disk driver health, system services, network ports, mail statistics (Sendmail, Postfix, Dovecot, etc), MySQL statistics, and many more.

It is designed to monitor overall system performance and helps in detecting failures, bottlenecks, abnormal activities, etc.

Monitorix Monitoring

15. Arpwatch – Ethernet Activity Monitor

Arpwatch is a kind of program that is designed to monitor Address Resolution of (MAC and IP address changes) of Ethernet network traffic on a Linux network.

It continuously keeps watch on Ethernet traffic and produces a log of IP and MAC address pair changes along with a timestamp on a network. It also has a feature to send email alerts to administrators, when a pairing is added or changes. It is very useful in detecting ARP spoofing on a network.

Arpwatch – Monitor ARP Traffic

For installation and usage, read our article: Arpwatch to Monitor Ethernet Activity

16. Suricata – Network Security Monitoring

Suricata is a high-performance open-source Network Security and Intrusion Detection and Prevention Monitoring System for Linux, FreeBSD, and Windows.

It was designed and owned by a non-profit foundation OISF (Open Information Security Foundation).

17. VnStat PHP – Monitoring Network Bandwidth

VnStat PHP is a web-based frontend application for the most popular networking tool called “vnstat“. VnStat PHP monitors network traffic usage in nicely graphical mode.

It displays a total IN and OUT network traffic usage in hourly, daily, monthly, and full summary reports.

For installation and usage, read our article: Monitoring Network Bandwidth Usage

18. Nagios – Network/Server Monitoring

Nagios is a leading open source powerful monitoring system that enables network/system administrators to identify and resolve server-related problems before they affect major business processes.

With the Nagios system, administrators can able to monitor remote Linux, Windows, Switches, Routers, and Printers on a single window. It shows critical warnings and indicates if something went wrong in your network/server which indirectly helps you to begin remediation processes before they occur.

19. Nmon: Monitor Linux Performance

Nmon (stands for Nigel’s performance Monitor) tool, which is used to monitor all Linux resources such as CPU, Memory, Disk Usage, Network, Top processes, NFS, Kernel, and much more. This tool comes in two modes: Online Mode and Capture Mode.

The Online Mode is used for real-time monitoring and Capture Mode is used to store the output in CSV format for later processing.

Nmon – Linux Performance Monitoring tool

20. Collectl: All-in-One Performance Monitoring Tool

Collectl is yet another powerful and feature-rich command-line-based utility, that can be used to gather information about Linux system resources such as CPU usage, memory, network, inodes, processes, nfs, TCP, sockets, and much more.

Collectl Monitoring

We would like to know what kind of monitoring programs you use to monitor the performance of your Linux servers? If we’ve missed any important tool that you would like us to include in this list, please inform us via comments, and please don’t forget to share it.

If You Appreciate What We Do Here On TecMint, You Should Consider:

TecMint is the fastest growing and most trusted community site for any kind of Linux Articles, Guides and Books on the web. Millions of people visit TecMint! to search or browse the thousands of published articles available FREELY to all.

If you like what you are reading, please consider buying us a coffee ( or 2 ) as a token of appreciation.

We are thankful for your never ending support.

Источник