Mellanox connectx 3 linux driver

Mellanox EN Driver for Linux

Mellanox offers a robust and full set of protocol software and driver for Linux with the ConnectX® EN family cards. Designed to provide a high performance support for Enhanced Ethernet with fabric consolidation over TCP/IP based LAN applications. The driver and software in conjunction with the Industry’s leading ConnectX family of cards achieve full line rate, full duplex of up to 100Gbps performance per port.

Related Documents

• High performance single/Dual port support with full line rate, full duplex 10Gbps, 40Gbps and 56Gbps per port
• High scale SR-IOV with 126 Virtual Functions (VFs)
• VXLAN hardware offload (ConnectX-3 Pro)
• TCP/IP stateless offload, and Hardware Checksum Offload for Tx and Rx packets
• Net device statistics per port with both ports in active mode
• Extensive VLAN support with VLAN Tx/Rx acceleration (Hardware VLAN stripping/insertion), Hardware VLAN filtering and Hardware multicast filtering
• Adaptive Interrupt moderation on the receive path patterned according to either latency-bound traffic or bandwidth bound traffic.
• Default NAPI support is enabled per-core to run in interrupt mode
• Performance optimization support with «ethtool» command; Allows tuning of various NIC driver options
• Use fewer high-performance systems in a rack based system, reducing cabling, low power consumption
• Supports all major Linux Distributions
• Supports KVM and XenServer

We’re sorry, this content is unavailable. This website requires a browser that supports iframes

We’re sorry, this content is unavailable. This website requires a browser that supports iframes

NVIDIA Mellanox Cookie Policy

This website uses cookies which may help to deliver content tailored to your preferences and interests, provide you with a better browsing experience, and to analyze our traffic. You may delete and/or block out cookies from this site, but it may affect how the site operates. Further information can be found in our Privacy Policy.

Источник

Linux Drivers

Mellanox OpenFabrics Enterprise Distribution for Linux (MLNX_OFED)

Clustering using commodity servers and storage systems is seeing widespread deployments in large and growing markets such as high performance computing, Artificial Intelligence (AI), data warehousing, online transaction processing, financial services and large scale cloud deployments. To enable distributed computing transparently and with maximum efficiency, applications in these markets require the highest I/O bandwidth and lowest possible latency. These requirements are compounded with the need to support a large interoperable ecosystem of networking, virtualization, storage, and other applications and interfaces. The OFED from OpenFabrics Alliance (www.openfabrics.org) has been hardened through collaborative development and testing by major high performance I/O vendors. Mellanox OFED (MLNX_OFED) is a Mellanox tested and packaged version of OFED that supports two interconnect types using the same RDMA (remote DMA) and kernel bypass APIs called OFED verbs – InfiniBand and Ethernet. Up to 200Gb/s InfiniBand and RoCE (based on the RDMA over Converged Ethernet standard) over 10/25/40/50/100GbE are supported with OFED by Mellanox to enable OEMs and System Integrators to meet the needs end users in the said markets.

Linux Inbox Drivers

Mellanox Adapters’ Linux VPI Drivers for Ethernet and InfiniBand are also available Inbox in all the major distributions, RHEL, SLES, Ubuntu and more. Inbox drivers enable Mellanox High performance solutions for Cloud, Artificial Intelligence, HPC, Storage, Financial Services and more with the Out of box experience of Enterprise grade Linux distributions.

View the matrix of MLNX_OFED driver versions vs. supported hardware and firmware for Mellanox products.

Related Documents

• Virtual Protocol Interconnect (VPI) allows Mellanox ConnectX adapter family to run InfiniBand and Ethernet traffic simultaneously on two ports
• Single software stack that operates across all available Mellanox InfiniBand and Ethernet devices and configurations such as mem-free, SDR/DDR/QDR/FDR/EDR/HDR, 10 /25/40/50/100/200 GbE, and PCI Express modes 3.0 and 4.0
• Support for HPC applications for scientific research, AI, oil and gas exploration, car crash tests, bench marking etc. E.g., Fluent, LS-DYNA
• Support for Data Center applications such as Oracle 11g/10g RAC, IBM DB2, Financial services applications such as IBM WebSphere LLM, Red Hat MRG, NYSE Data Fabric
• Support for high-performance block storage applications utilizing RDMA benefits

Note: By downloading and installing MLNX_OFED package for Oracle Linux (OL) OS, you may be violating your operating system’s support matrix. Please consult with your operating system support before installing.

Note: As of MLNX_OFED v5.1, the following are no longer supported and can be utilized through MLNX_OFED LTS version:

• ConnectX-3 Pro
• ConnectX-3
• Connect-IB
• RDMA experimental verbs library (mlnx_lib)

We’re sorry, this content is unavailable. This website requires a browser that supports iframes

We’re sorry, this content is unavailable. This website requires a browser that supports iframes

NVIDIA Mellanox Cookie Policy

This website uses cookies which may help to deliver content tailored to your preferences and interests, provide you with a better browsing experience, and to analyze our traffic. You may delete and/or block out cookies from this site, but it may affect how the site operates. Further information can be found in our Privacy Policy.

Источник

Mellanox OFED for Windows — WinOF / WinOF-2

Windows OS Host controller driver for Cloud, Storage and High-Performance computing applications utilizing Mellanox’ field-proven RDMA and Transport Offloads

The Mellanox Windows distribution includes software for database clustering, Cloud, High Performance Computing, communications, and storage applications for servers and clients running different versions of Windows OS. This collection consists of drivers, protocols, and management in simple ready-to-install MSIs.
More detailed information on each package is provided in the documentation package available in the Related Documents section.

For ConnectX-3 and ConnectX-3 Pro drivers download WinOF.

For ConnectX-4 and onwards adapter cards drivers download WinOF-2.

View the matrix of WinOF/WinOF-2 drivers versions vs. supported hardware and firmware for Mellanox products.

Related Documents

The Mellanox WinOF and WinOF-2 distribution provides the following benefits:

• Virtual Protocol Interconnect (VPI): Running Ethernet and/or InfiniBand on the same Host Controller
• Support Windows Azure Pack
• Support traditional IP and Sockets based applications leveraging the benefits of RDMA
• Support high-performance block storage applications utilizing RDMA benefits
• Cloud and virtualization:
  • NVGRE and VxLAN Hardware offload (ConnectX-3 Pro and ConnectX-4)
  • SR-IOV
  • Function per-port (ConnectX-4)
• NDK with SMB-Direct
• NDv1 and v2 API support in user space
• Support Teaming and High-Availability
• Support a variety of Windows Server and Client OS )

Note: Features are OS dependent.

Note: For the exact schedule of availability of each feature, please contact Mellanox Support.

Note: WinOF v5.50.54000 includes a driver for Windows Server 2019 only. For other OSes, please see WinOF v5.50.53000.

We’re sorry, this content is unavailable. This website requires a browser that supports iframes

We’re sorry, this content is unavailable. This website requires a browser that supports iframes

Источник

Обновление драйвера Mellanox ConnectX

Для примера обновлю драйвер сетевого адаптера Mellanox ConnectX-3 Pro EN / HP 544+QSFP (764284-B21) который я установил в сервер HPE DL380 Gen10 с Ubuntu Server 18.04.

Для обновления драйвера необходимо переключится на root пользователя:

Сначала я посмотрел версию установленного драйвера:

Выполнив эти команды на нескольких серверах Ubuntu Server 18.04 я заметил что по умолчанию используется драйвер версии 4.0-0.

Посмотрим видит ли система сетевой адаптер, как он называется и его Part number:

Кстати я пробовал установить одновременно сетевые адаптеры на разных чипах (Intel XL710-QDA1 и Mellanox ConnectX-3 / HP 544+QSFP) в сервер HPE DL380 Gen10 и заметил что одновременно они не работают и по команде lspci их не видно.

Потом я посмотрел драйвер на официальном сайте
https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed
Скачал и распаковал LTS версию MLNX_OFED_LINUX-4.9-2.2.4.0-ubuntu18.04-x86_64.tgz.
Так как в самой новой версии MLNX_OFED_LINUX-5.1-2.5.8.0-ubuntu18.04-x86_64.tgz на момент написания статьи были драйвера начиная от ConnectX-4, а для ConnectX-3 не было.

Распаковал скачанный архив:

Для запуска процесса обновления необходимо запустить скрипт установки:

В этом случае будет установлен полный пакет утилит и обновлен драйвер. Скрипт обнаружит все сетевые адаптеры Mellanox и предложит выбрать конкретный или все, а также выполнит обновлении прошивки. Убедимся что в файле distro указана верная версия дистрибутива операционной системы.

Но я решил обновить драйвер и установить минимальный набор утилит:

Во время установки будут установлены и удалены некоторые пакеты, в случае ошибки их можно установить или удалить вручную, а потом вновь запустить скрипт, например:

После успешного обновления перезапустим драйвер:

Перезагрузим сервер и убедимся что сеть работает:

Если необходимо только обновить прошивку, то выполним:

Пример создания резервной копии прошивки и конфигурации сетевого адаптера:

При необходимости закроем порты:

Модули которые автоматически загружаются при запуске операционной системы:
ConnectX®-2/ConnectX®-3/ConnectX®-3 Pro:
mlx4_en, mlx4_core, mlx4_ib
ConnectX®-4/ConnectX®-4 Lx/ConnectX®-5:
mlx5_core, mlx5_ib

Если необходимо выгрузить модули драйвера, то модуль mlx*_core необходимо выгружать в последнюю очередь, например:

Пример загрузки модулей:

Приведу список важных пакетов которые устанавливаются скриптом обновления драйвера или через apt install:
rdma-core, libibmad, opensm, opensm-libs, ibutils-libs, ibutils, infiniband-diags, perftest, mstflint

Источник

How to update the Mellanox ConnectX driver

For example, I will update the driver for the Mellanox ConnectX-3 Pro EN / HP 544+ QSFP (764284-B21) network adapter that I installed in the HPE DL380 Gen10 server with Ubuntu Server 18.04.

To update the driver, you need to switch to the root user:

First I looked at the version of the driver installed:

After running these commands on multiple Ubuntu Server 18.04 servers, I noticed that the default driver is version 4.0-0.

Let’s see if the system sees the network adapter, what it is called and its Part number:

By the way, I tried to install at the same time network adapters on different chips (Intel XL710-QDA1 and Mellanox ConnectX-3 / HP 544+ QSFP) in the HPE DL380 Gen10 server and noticed that they do not work at the same time and cannot be seen by the lspci command.

Then I looked at the driver on the official website
https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed
Downloaded LTS version MLNX_OFED_LINUX-4.9-2.2.4.0-ubuntu18.04-x86_64.tgz.
Since in the newest version MLNX_OFED_LINUX-5.1-2.5.8.0-ubuntu18.04-x86_64.tgz at the time of this writing, there were drivers starting from ConnectX-4, but for ConnectX-3 there were none.

I unpacked the downloaded archive:

To start the update process, you need to run the installation script:

This will install the full utilities package and update the driver. The script will detect all Mellanox network adapters and offer to select specific or all, as well as perform a firmware update. Make sure that the correct version of the operating system distribution is specified in the distro file.

But I decided to update the driver and install the minimal set of utilities:

During installation, some packages will be installed and removed, in case of an error, you can install or remove them manually, and then run the script again, for example:

After a successful update, restart the driver:

Restart the server and make sure the network is working:

If you only need to update the firmware, then run:

An example of creating a backup copy of the firmware and configuration of the network adapter:

Close the ports if necessary:

Modules that are automatically loaded when the operating system starts:
ConnectX®-2/ConnectX®-3/ConnectX®-3 Pro:
mlx4_en, mlx4_core, mlx4_ib
ConnectX®-4/ConnectX®-4 Lx/ConnectX®-5:
mlx5_core, mlx5_ib

If you need to unload driver modules, then the mlx*_core module must be unloaded last, for example:

An example of loading modules:

I will give a list of important packages that are installed by the driver update script or via apt install:
rdma-core, libibmad, opensm, opensm-libs, ibutils-libs, ibutils, infiniband-diags, perftest, mstflint

Источник