ixgb.rst (17480B)
1.. SPDX-License-Identifier: GPL-2.0+ 2 3===================================================================== 4Linux Base Driver for 10 Gigabit Intel(R) Ethernet Network Connection 5===================================================================== 6 7October 1, 2018 8 9 10Contents 11======== 12 13- In This Release 14- Identifying Your Adapter 15- Command Line Parameters 16- Improving Performance 17- Additional Configurations 18- Known Issues/Troubleshooting 19- Support 20 21 22 23In This Release 24=============== 25 26This file describes the ixgb Linux Base Driver for the 10 Gigabit Intel(R) 27Network Connection. This driver includes support for Itanium(R)2-based 28systems. 29 30For questions related to hardware requirements, refer to the documentation 31supplied with your 10 Gigabit adapter. All hardware requirements listed apply 32to use with Linux. 33 34The following features are available in this kernel: 35 - Native VLANs 36 - Channel Bonding (teaming) 37 - SNMP 38 39Channel Bonding documentation can be found in the Linux kernel source: 40/Documentation/networking/bonding.rst 41 42The driver information previously displayed in the /proc filesystem is not 43supported in this release. Alternatively, you can use ethtool (version 1.6 44or later), lspci, and iproute2 to obtain the same information. 45 46Instructions on updating ethtool can be found in the section "Additional 47Configurations" later in this document. 48 49 50Identifying Your Adapter 51======================== 52 53The following Intel network adapters are compatible with the drivers in this 54release: 55 56+------------+------------------------------+----------------------------------+ 57| Controller | Adapter Name | Physical Layer | 58+============+==============================+==================================+ 59| 82597EX | Intel(R) PRO/10GbE LR/SR/CX4 | - 10G Base-LR (fiber) | 60| | Server Adapters | - 10G Base-SR (fiber) | 61| | | - 10G Base-CX4 (copper) | 62+------------+------------------------------+----------------------------------+ 63 64For more information on how to identify your adapter, go to the Adapter & 65Driver ID Guide at: 66 67 https://support.intel.com 68 69 70Command Line Parameters 71======================= 72 73If the driver is built as a module, the following optional parameters are 74used by entering them on the command line with the modprobe command using 75this syntax:: 76 77 modprobe ixgb [<option>=<VAL1>,<VAL2>,...] 78 79For example, with two 10GbE PCI adapters, entering:: 80 81 modprobe ixgb TxDescriptors=80,128 82 83loads the ixgb driver with 80 TX resources for the first adapter and 128 TX 84resources for the second adapter. 85 86The default value for each parameter is generally the recommended setting, 87unless otherwise noted. 88 89Copybreak 90--------- 91:Valid Range: 0-XXXX 92:Default Value: 256 93 94 This is the maximum size of packet that is copied to a new buffer on 95 receive. 96 97Debug 98----- 99:Valid Range: 0-16 (0=none,...,16=all) 100:Default Value: 0 101 102 This parameter adjusts the level of debug messages displayed in the 103 system logs. 104 105FlowControl 106----------- 107:Valid Range: 0-3 (0=none, 1=Rx only, 2=Tx only, 3=Rx&Tx) 108:Default Value: 1 if no EEPROM, otherwise read from EEPROM 109 110 This parameter controls the automatic generation(Tx) and response(Rx) to 111 Ethernet PAUSE frames. There are hardware bugs associated with enabling 112 Tx flow control so beware. 113 114RxDescriptors 115------------- 116:Valid Range: 64-4096 117:Default Value: 1024 118 119 This value is the number of receive descriptors allocated by the driver. 120 Increasing this value allows the driver to buffer more incoming packets. 121 Each descriptor is 16 bytes. A receive buffer is also allocated for 122 each descriptor and can be either 2048, 4056, 8192, or 16384 bytes, 123 depending on the MTU setting. When the MTU size is 1500 or less, the 124 receive buffer size is 2048 bytes. When the MTU is greater than 1500 the 125 receive buffer size will be either 4056, 8192, or 16384 bytes. The 126 maximum MTU size is 16114. 127 128TxDescriptors 129------------- 130:Valid Range: 64-4096 131:Default Value: 256 132 133 This value is the number of transmit descriptors allocated by the driver. 134 Increasing this value allows the driver to queue more transmits. Each 135 descriptor is 16 bytes. 136 137RxIntDelay 138---------- 139:Valid Range: 0-65535 (0=off) 140:Default Value: 72 141 142 This value delays the generation of receive interrupts in units of 143 0.8192 microseconds. Receive interrupt reduction can improve CPU 144 efficiency if properly tuned for specific network traffic. Increasing 145 this value adds extra latency to frame reception and can end up 146 decreasing the throughput of TCP traffic. If the system is reporting 147 dropped receives, this value may be set too high, causing the driver to 148 run out of available receive descriptors. 149 150TxIntDelay 151---------- 152:Valid Range: 0-65535 (0=off) 153:Default Value: 32 154 155 This value delays the generation of transmit interrupts in units of 156 0.8192 microseconds. Transmit interrupt reduction can improve CPU 157 efficiency if properly tuned for specific network traffic. Increasing 158 this value adds extra latency to frame transmission and can end up 159 decreasing the throughput of TCP traffic. If this value is set too high, 160 it will cause the driver to run out of available transmit descriptors. 161 162XsumRX 163------ 164:Valid Range: 0-1 165:Default Value: 1 166 167 A value of '1' indicates that the driver should enable IP checksum 168 offload for received packets (both UDP and TCP) to the adapter hardware. 169 170RxFCHighThresh 171-------------- 172:Valid Range: 1,536-262,136 (0x600 - 0x3FFF8, 8 byte granularity) 173:Default Value: 196,608 (0x30000) 174 175 Receive Flow control high threshold (when we send a pause frame) 176 177RxFCLowThresh 178------------- 179:Valid Range: 64-262,136 (0x40 - 0x3FFF8, 8 byte granularity) 180:Default Value: 163,840 (0x28000) 181 182 Receive Flow control low threshold (when we send a resume frame) 183 184FCReqTimeout 185------------ 186:Valid Range: 1-65535 187:Default Value: 65535 188 189 Flow control request timeout (how long to pause the link partner's tx) 190 191IntDelayEnable 192-------------- 193:Value Range: 0,1 194:Default Value: 1 195 196 Interrupt Delay, 0 disables transmit interrupt delay and 1 enables it. 197 198 199Improving Performance 200===================== 201 202With the 10 Gigabit server adapters, the default Linux configuration will 203very likely limit the total available throughput artificially. There is a set 204of configuration changes that, when applied together, will increase the ability 205of Linux to transmit and receive data. The following enhancements were 206originally acquired from settings published at https://www.spec.org/web99/ for 207various submitted results using Linux. 208 209NOTE: 210 These changes are only suggestions, and serve as a starting point for 211 tuning your network performance. 212 213The changes are made in three major ways, listed in order of greatest effect: 214 215- Use ip link to modify the mtu (maximum transmission unit) and the txqueuelen 216 parameter. 217- Use sysctl to modify /proc parameters (essentially kernel tuning) 218- Use setpci to modify the MMRBC field in PCI-X configuration space to increase 219 transmit burst lengths on the bus. 220 221NOTE: 222 setpci modifies the adapter's configuration registers to allow it to read 223 up to 4k bytes at a time (for transmits). However, for some systems the 224 behavior after modifying this register may be undefined (possibly errors of 225 some kind). A power-cycle, hard reset or explicitly setting the e6 register 226 back to 22 (setpci -d 8086:1a48 e6.b=22) may be required to get back to a 227 stable configuration. 228 229- COPY these lines and paste them into ixgb_perf.sh: 230 231:: 232 233 #!/bin/bash 234 echo "configuring network performance , edit this file to change the interface 235 or device ID of 10GbE card" 236 # set mmrbc to 4k reads, modify only Intel 10GbE device IDs 237 # replace 1a48 with appropriate 10GbE device's ID installed on the system, 238 # if needed. 239 setpci -d 8086:1a48 e6.b=2e 240 # set the MTU (max transmission unit) - it requires your switch and clients 241 # to change as well. 242 # set the txqueuelen 243 # your ixgb adapter should be loaded as eth1 for this to work, change if needed 244 ip li set dev eth1 mtu 9000 txqueuelen 1000 up 245 # call the sysctl utility to modify /proc/sys entries 246 sysctl -p ./sysctl_ixgb.conf 247 248- COPY these lines and paste them into sysctl_ixgb.conf: 249 250:: 251 252 # some of the defaults may be different for your kernel 253 # call this file with sysctl -p <this file> 254 # these are just suggested values that worked well to increase throughput in 255 # several network benchmark tests, your mileage may vary 256 257 ### IPV4 specific settings 258 # turn TCP timestamp support off, default 1, reduces CPU use 259 net.ipv4.tcp_timestamps = 0 260 # turn SACK support off, default on 261 # on systems with a VERY fast bus -> memory interface this is the big gainer 262 net.ipv4.tcp_sack = 0 263 # set min/default/max TCP read buffer, default 4096 87380 174760 264 net.ipv4.tcp_rmem = 10000000 10000000 10000000 265 # set min/pressure/max TCP write buffer, default 4096 16384 131072 266 net.ipv4.tcp_wmem = 10000000 10000000 10000000 267 # set min/pressure/max TCP buffer space, default 31744 32256 32768 268 net.ipv4.tcp_mem = 10000000 10000000 10000000 269 270 ### CORE settings (mostly for socket and UDP effect) 271 # set maximum receive socket buffer size, default 131071 272 net.core.rmem_max = 524287 273 # set maximum send socket buffer size, default 131071 274 net.core.wmem_max = 524287 275 # set default receive socket buffer size, default 65535 276 net.core.rmem_default = 524287 277 # set default send socket buffer size, default 65535 278 net.core.wmem_default = 524287 279 # set maximum amount of option memory buffers, default 10240 280 net.core.optmem_max = 524287 281 # set number of unprocessed input packets before kernel starts dropping them; default 300 282 net.core.netdev_max_backlog = 300000 283 284Edit the ixgb_perf.sh script if necessary to change eth1 to whatever interface 285your ixgb driver is using and/or replace '1a48' with appropriate 10GbE device's 286ID installed on the system. 287 288NOTE: 289 Unless these scripts are added to the boot process, these changes will 290 only last only until the next system reboot. 291 292 293Resolving Slow UDP Traffic 294-------------------------- 295If your server does not seem to be able to receive UDP traffic as fast as it 296can receive TCP traffic, it could be because Linux, by default, does not set 297the network stack buffers as large as they need to be to support high UDP 298transfer rates. One way to alleviate this problem is to allow more memory to 299be used by the IP stack to store incoming data. 300 301For instance, use the commands:: 302 303 sysctl -w net.core.rmem_max=262143 304 305and:: 306 307 sysctl -w net.core.rmem_default=262143 308 309to increase the read buffer memory max and default to 262143 (256k - 1) from 310defaults of max=131071 (128k - 1) and default=65535 (64k - 1). These variables 311will increase the amount of memory used by the network stack for receives, and 312can be increased significantly more if necessary for your application. 313 314 315Additional Configurations 316========================= 317 318Configuring the Driver on Different Distributions 319------------------------------------------------- 320Configuring a network driver to load properly when the system is started is 321distribution dependent. Typically, the configuration process involves adding 322an alias line to /etc/modprobe.conf as well as editing other system startup 323scripts and/or configuration files. Many popular Linux distributions ship 324with tools to make these changes for you. To learn the proper way to 325configure a network device for your system, refer to your distribution 326documentation. If during this process you are asked for the driver or module 327name, the name for the Linux Base Driver for the Intel 10GbE Family of 328Adapters is ixgb. 329 330Viewing Link Messages 331--------------------- 332Link messages will not be displayed to the console if the distribution is 333restricting system messages. In order to see network driver link messages on 334your console, set dmesg to eight by entering the following:: 335 336 dmesg -n 8 337 338NOTE: This setting is not saved across reboots. 339 340Jumbo Frames 341------------ 342The driver supports Jumbo Frames for all adapters. Jumbo Frames support is 343enabled by changing the MTU to a value larger than the default of 1500. 344The maximum value for the MTU is 16114. Use the ip command to 345increase the MTU size. For example:: 346 347 ip li set dev ethx mtu 9000 348 349The maximum MTU setting for Jumbo Frames is 16114. This value coincides 350with the maximum Jumbo Frames size of 16128. 351 352Ethtool 353------- 354The driver utilizes the ethtool interface for driver configuration and 355diagnostics, as well as displaying statistical information. The ethtool 356version 1.6 or later is required for this functionality. 357 358The latest release of ethtool can be found from 359https://www.kernel.org/pub/software/network/ethtool/ 360 361NOTE: 362 The ethtool version 1.6 only supports a limited set of ethtool options. 363 Support for a more complete ethtool feature set can be enabled by 364 upgrading to the latest version. 365 366NAPI 367---- 368NAPI (Rx polling mode) is supported in the ixgb driver. 369 370See https://wiki.linuxfoundation.org/networking/napi for more information on 371NAPI. 372 373 374Known Issues/Troubleshooting 375============================ 376 377NOTE: 378 After installing the driver, if your Intel Network Connection is not 379 working, verify in the "In This Release" section of the readme that you have 380 installed the correct driver. 381 382Cable Interoperability Issue with Fujitsu XENPAK Module in SmartBits Chassis 383---------------------------------------------------------------------------- 384Excessive CRC errors may be observed if the Intel(R) PRO/10GbE CX4 385Server adapter is connected to a Fujitsu XENPAK CX4 module in a SmartBits 386chassis using 15 m/24AWG cable assemblies manufactured by Fujitsu or Leoni. 387The CRC errors may be received either by the Intel(R) PRO/10GbE CX4 388Server adapter or the SmartBits. If this situation occurs using a different 389cable assembly may resolve the issue. 390 391Cable Interoperability Issues with HP Procurve 3400cl Switch Port 392----------------------------------------------------------------- 393Excessive CRC errors may be observed if the Intel(R) PRO/10GbE CX4 Server 394adapter is connected to an HP Procurve 3400cl switch port using short cables 395(1 m or shorter). If this situation occurs, using a longer cable may resolve 396the issue. 397 398Excessive CRC errors may be observed using Fujitsu 24AWG cable assemblies that 399Are 10 m or longer or where using a Leoni 15 m/24AWG cable assembly. The CRC 400errors may be received either by the CX4 Server adapter or at the switch. If 401this situation occurs, using a different cable assembly may resolve the issue. 402 403Jumbo Frames System Requirement 404------------------------------- 405Memory allocation failures have been observed on Linux systems with 64 MB 406of RAM or less that are running Jumbo Frames. If you are using Jumbo 407Frames, your system may require more than the advertised minimum 408requirement of 64 MB of system memory. 409 410Performance Degradation with Jumbo Frames 411----------------------------------------- 412Degradation in throughput performance may be observed in some Jumbo frames 413environments. If this is observed, increasing the application's socket buffer 414size and/or increasing the /proc/sys/net/ipv4/tcp_*mem entry values may help. 415See the specific application manual and /usr/src/linux*/Documentation/ 416networking/ip-sysctl.txt for more details. 417 418Allocating Rx Buffers when Using Jumbo Frames 419--------------------------------------------- 420Allocating Rx buffers when using Jumbo Frames on 2.6.x kernels may fail if 421the available memory is heavily fragmented. This issue may be seen with PCI-X 422adapters or with packet split disabled. This can be reduced or eliminated 423by changing the amount of available memory for receive buffer allocation, by 424increasing /proc/sys/vm/min_free_kbytes. 425 426Multiple Interfaces on Same Ethernet Broadcast Network 427------------------------------------------------------ 428Due to the default ARP behavior on Linux, it is not possible to have 429one system on two IP networks in the same Ethernet broadcast domain 430(non-partitioned switch) behave as expected. All Ethernet interfaces 431will respond to IP traffic for any IP address assigned to the system. 432This results in unbalanced receive traffic. 433 434If you have multiple interfaces in a server, do either of the following: 435 436 - Turn on ARP filtering by entering:: 437 438 echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter 439 440 - Install the interfaces in separate broadcast domains - either in 441 different switches or in a switch partitioned to VLANs. 442 443UDP Stress Test Dropped Packet Issue 444-------------------------------------- 445Under small packets UDP stress test with 10GbE driver, the Linux system 446may drop UDP packets due to the fullness of socket buffers. You may want 447to change the driver's Flow Control variables to the minimum value for 448controlling packet reception. 449 450Tx Hangs Possible Under Stress 451------------------------------ 452Under stress conditions, if TX hangs occur, turning off TSO 453"ethtool -K eth0 tso off" may resolve the problem. 454 455 456Support 457======= 458For general information, go to the Intel support website at: 459 460https://www.intel.com/support/ 461 462or the Intel Wired Networking project hosted by Sourceforge at: 463 464https://sourceforge.net/projects/e1000 465 466If an issue is identified with the released source code on a supported kernel 467with a supported adapter, email the specific information related to the issue 468to e1000-devel@lists.sf.net