snmp_counter.rst (66798B)
1============ 2SNMP counter 3============ 4 5This document explains the meaning of SNMP counters. 6 7General IPv4 counters 8===================== 9All layer 4 packets and ICMP packets will change these counters, but 10these counters won't be changed by layer 2 packets (such as STP) or 11ARP packets. 12 13* IpInReceives 14 15Defined in `RFC1213 ipInReceives`_ 16 17.. _RFC1213 ipInReceives: https://tools.ietf.org/html/rfc1213#page-26 18 19The number of packets received by the IP layer. It gets increasing at the 20beginning of ip_rcv function, always be updated together with 21IpExtInOctets. It will be increased even if the packet is dropped 22later (e.g. due to the IP header is invalid or the checksum is wrong 23and so on). It indicates the number of aggregated segments after 24GRO/LRO. 25 26* IpInDelivers 27 28Defined in `RFC1213 ipInDelivers`_ 29 30.. _RFC1213 ipInDelivers: https://tools.ietf.org/html/rfc1213#page-28 31 32The number of packets delivers to the upper layer protocols. E.g. TCP, UDP, 33ICMP and so on. If no one listens on a raw socket, only kernel 34supported protocols will be delivered, if someone listens on the raw 35socket, all valid IP packets will be delivered. 36 37* IpOutRequests 38 39Defined in `RFC1213 ipOutRequests`_ 40 41.. _RFC1213 ipOutRequests: https://tools.ietf.org/html/rfc1213#page-28 42 43The number of packets sent via IP layer, for both single cast and 44multicast packets, and would always be updated together with 45IpExtOutOctets. 46 47* IpExtInOctets and IpExtOutOctets 48 49They are Linux kernel extensions, no RFC definitions. Please note, 50RFC1213 indeed defines ifInOctets and ifOutOctets, but they 51are different things. The ifInOctets and ifOutOctets include the MAC 52layer header size but IpExtInOctets and IpExtOutOctets don't, they 53only include the IP layer header and the IP layer data. 54 55* IpExtInNoECTPkts, IpExtInECT1Pkts, IpExtInECT0Pkts, IpExtInCEPkts 56 57They indicate the number of four kinds of ECN IP packets, please refer 58`Explicit Congestion Notification`_ for more details. 59 60.. _Explicit Congestion Notification: https://tools.ietf.org/html/rfc3168#page-6 61 62These 4 counters calculate how many packets received per ECN 63status. They count the real frame number regardless the LRO/GRO. So 64for the same packet, you might find that IpInReceives count 1, but 65IpExtInNoECTPkts counts 2 or more. 66 67* IpInHdrErrors 68 69Defined in `RFC1213 ipInHdrErrors`_. It indicates the packet is 70dropped due to the IP header error. It might happen in both IP input 71and IP forward paths. 72 73.. _RFC1213 ipInHdrErrors: https://tools.ietf.org/html/rfc1213#page-27 74 75* IpInAddrErrors 76 77Defined in `RFC1213 ipInAddrErrors`_. It will be increased in two 78scenarios: (1) The IP address is invalid. (2) The destination IP 79address is not a local address and IP forwarding is not enabled 80 81.. _RFC1213 ipInAddrErrors: https://tools.ietf.org/html/rfc1213#page-27 82 83* IpExtInNoRoutes 84 85This counter means the packet is dropped when the IP stack receives a 86packet and can't find a route for it from the route table. It might 87happen when IP forwarding is enabled and the destination IP address is 88not a local address and there is no route for the destination IP 89address. 90 91* IpInUnknownProtos 92 93Defined in `RFC1213 ipInUnknownProtos`_. It will be increased if the 94layer 4 protocol is unsupported by kernel. If an application is using 95raw socket, kernel will always deliver the packet to the raw socket 96and this counter won't be increased. 97 98.. _RFC1213 ipInUnknownProtos: https://tools.ietf.org/html/rfc1213#page-27 99 100* IpExtInTruncatedPkts 101 102For IPv4 packet, it means the actual data size is smaller than the 103"Total Length" field in the IPv4 header. 104 105* IpInDiscards 106 107Defined in `RFC1213 ipInDiscards`_. It indicates the packet is dropped 108in the IP receiving path and due to kernel internal reasons (e.g. no 109enough memory). 110 111.. _RFC1213 ipInDiscards: https://tools.ietf.org/html/rfc1213#page-28 112 113* IpOutDiscards 114 115Defined in `RFC1213 ipOutDiscards`_. It indicates the packet is 116dropped in the IP sending path and due to kernel internal reasons. 117 118.. _RFC1213 ipOutDiscards: https://tools.ietf.org/html/rfc1213#page-28 119 120* IpOutNoRoutes 121 122Defined in `RFC1213 ipOutNoRoutes`_. It indicates the packet is 123dropped in the IP sending path and no route is found for it. 124 125.. _RFC1213 ipOutNoRoutes: https://tools.ietf.org/html/rfc1213#page-29 126 127ICMP counters 128============= 129* IcmpInMsgs and IcmpOutMsgs 130 131Defined by `RFC1213 icmpInMsgs`_ and `RFC1213 icmpOutMsgs`_ 132 133.. _RFC1213 icmpInMsgs: https://tools.ietf.org/html/rfc1213#page-41 134.. _RFC1213 icmpOutMsgs: https://tools.ietf.org/html/rfc1213#page-43 135 136As mentioned in the RFC1213, these two counters include errors, they 137would be increased even if the ICMP packet has an invalid type. The 138ICMP output path will check the header of a raw socket, so the 139IcmpOutMsgs would still be updated if the IP header is constructed by 140a userspace program. 141 142* ICMP named types 143 144| These counters include most of common ICMP types, they are: 145| IcmpInDestUnreachs: `RFC1213 icmpInDestUnreachs`_ 146| IcmpInTimeExcds: `RFC1213 icmpInTimeExcds`_ 147| IcmpInParmProbs: `RFC1213 icmpInParmProbs`_ 148| IcmpInSrcQuenchs: `RFC1213 icmpInSrcQuenchs`_ 149| IcmpInRedirects: `RFC1213 icmpInRedirects`_ 150| IcmpInEchos: `RFC1213 icmpInEchos`_ 151| IcmpInEchoReps: `RFC1213 icmpInEchoReps`_ 152| IcmpInTimestamps: `RFC1213 icmpInTimestamps`_ 153| IcmpInTimestampReps: `RFC1213 icmpInTimestampReps`_ 154| IcmpInAddrMasks: `RFC1213 icmpInAddrMasks`_ 155| IcmpInAddrMaskReps: `RFC1213 icmpInAddrMaskReps`_ 156| IcmpOutDestUnreachs: `RFC1213 icmpOutDestUnreachs`_ 157| IcmpOutTimeExcds: `RFC1213 icmpOutTimeExcds`_ 158| IcmpOutParmProbs: `RFC1213 icmpOutParmProbs`_ 159| IcmpOutSrcQuenchs: `RFC1213 icmpOutSrcQuenchs`_ 160| IcmpOutRedirects: `RFC1213 icmpOutRedirects`_ 161| IcmpOutEchos: `RFC1213 icmpOutEchos`_ 162| IcmpOutEchoReps: `RFC1213 icmpOutEchoReps`_ 163| IcmpOutTimestamps: `RFC1213 icmpOutTimestamps`_ 164| IcmpOutTimestampReps: `RFC1213 icmpOutTimestampReps`_ 165| IcmpOutAddrMasks: `RFC1213 icmpOutAddrMasks`_ 166| IcmpOutAddrMaskReps: `RFC1213 icmpOutAddrMaskReps`_ 167 168.. _RFC1213 icmpInDestUnreachs: https://tools.ietf.org/html/rfc1213#page-41 169.. _RFC1213 icmpInTimeExcds: https://tools.ietf.org/html/rfc1213#page-41 170.. _RFC1213 icmpInParmProbs: https://tools.ietf.org/html/rfc1213#page-42 171.. _RFC1213 icmpInSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-42 172.. _RFC1213 icmpInRedirects: https://tools.ietf.org/html/rfc1213#page-42 173.. _RFC1213 icmpInEchos: https://tools.ietf.org/html/rfc1213#page-42 174.. _RFC1213 icmpInEchoReps: https://tools.ietf.org/html/rfc1213#page-42 175.. _RFC1213 icmpInTimestamps: https://tools.ietf.org/html/rfc1213#page-42 176.. _RFC1213 icmpInTimestampReps: https://tools.ietf.org/html/rfc1213#page-43 177.. _RFC1213 icmpInAddrMasks: https://tools.ietf.org/html/rfc1213#page-43 178.. _RFC1213 icmpInAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-43 179 180.. _RFC1213 icmpOutDestUnreachs: https://tools.ietf.org/html/rfc1213#page-44 181.. _RFC1213 icmpOutTimeExcds: https://tools.ietf.org/html/rfc1213#page-44 182.. _RFC1213 icmpOutParmProbs: https://tools.ietf.org/html/rfc1213#page-44 183.. _RFC1213 icmpOutSrcQuenchs: https://tools.ietf.org/html/rfc1213#page-44 184.. _RFC1213 icmpOutRedirects: https://tools.ietf.org/html/rfc1213#page-44 185.. _RFC1213 icmpOutEchos: https://tools.ietf.org/html/rfc1213#page-45 186.. _RFC1213 icmpOutEchoReps: https://tools.ietf.org/html/rfc1213#page-45 187.. _RFC1213 icmpOutTimestamps: https://tools.ietf.org/html/rfc1213#page-45 188.. _RFC1213 icmpOutTimestampReps: https://tools.ietf.org/html/rfc1213#page-45 189.. _RFC1213 icmpOutAddrMasks: https://tools.ietf.org/html/rfc1213#page-45 190.. _RFC1213 icmpOutAddrMaskReps: https://tools.ietf.org/html/rfc1213#page-46 191 192Every ICMP type has two counters: 'In' and 'Out'. E.g., for the ICMP 193Echo packet, they are IcmpInEchos and IcmpOutEchos. Their meanings are 194straightforward. The 'In' counter means kernel receives such a packet 195and the 'Out' counter means kernel sends such a packet. 196 197* ICMP numeric types 198 199They are IcmpMsgInType[N] and IcmpMsgOutType[N], the [N] indicates the 200ICMP type number. These counters track all kinds of ICMP packets. The 201ICMP type number definition could be found in the `ICMP parameters`_ 202document. 203 204.. _ICMP parameters: https://www.iana.org/assignments/icmp-parameters/icmp-parameters.xhtml 205 206For example, if the Linux kernel sends an ICMP Echo packet, the 207IcmpMsgOutType8 would increase 1. And if kernel gets an ICMP Echo Reply 208packet, IcmpMsgInType0 would increase 1. 209 210* IcmpInCsumErrors 211 212This counter indicates the checksum of the ICMP packet is 213wrong. Kernel verifies the checksum after updating the IcmpInMsgs and 214before updating IcmpMsgInType[N]. If a packet has bad checksum, the 215IcmpInMsgs would be updated but none of IcmpMsgInType[N] would be updated. 216 217* IcmpInErrors and IcmpOutErrors 218 219Defined by `RFC1213 icmpInErrors`_ and `RFC1213 icmpOutErrors`_ 220 221.. _RFC1213 icmpInErrors: https://tools.ietf.org/html/rfc1213#page-41 222.. _RFC1213 icmpOutErrors: https://tools.ietf.org/html/rfc1213#page-43 223 224When an error occurs in the ICMP packet handler path, these two 225counters would be updated. The receiving packet path use IcmpInErrors 226and the sending packet path use IcmpOutErrors. When IcmpInCsumErrors 227is increased, IcmpInErrors would always be increased too. 228 229relationship of the ICMP counters 230--------------------------------- 231The sum of IcmpMsgOutType[N] is always equal to IcmpOutMsgs, as they 232are updated at the same time. The sum of IcmpMsgInType[N] plus 233IcmpInErrors should be equal or larger than IcmpInMsgs. When kernel 234receives an ICMP packet, kernel follows below logic: 235 2361. increase IcmpInMsgs 2372. if has any error, update IcmpInErrors and finish the process 2383. update IcmpMsgOutType[N] 2394. handle the packet depending on the type, if has any error, update 240 IcmpInErrors and finish the process 241 242So if all errors occur in step (2), IcmpInMsgs should be equal to the 243sum of IcmpMsgOutType[N] plus IcmpInErrors. If all errors occur in 244step (4), IcmpInMsgs should be equal to the sum of 245IcmpMsgOutType[N]. If the errors occur in both step (2) and step (4), 246IcmpInMsgs should be less than the sum of IcmpMsgOutType[N] plus 247IcmpInErrors. 248 249General TCP counters 250==================== 251* TcpInSegs 252 253Defined in `RFC1213 tcpInSegs`_ 254 255.. _RFC1213 tcpInSegs: https://tools.ietf.org/html/rfc1213#page-48 256 257The number of packets received by the TCP layer. As mentioned in 258RFC1213, it includes the packets received in error, such as checksum 259error, invalid TCP header and so on. Only one error won't be included: 260if the layer 2 destination address is not the NIC's layer 2 261address. It might happen if the packet is a multicast or broadcast 262packet, or the NIC is in promiscuous mode. In these situations, the 263packets would be delivered to the TCP layer, but the TCP layer will discard 264these packets before increasing TcpInSegs. The TcpInSegs counter 265isn't aware of GRO. So if two packets are merged by GRO, the TcpInSegs 266counter would only increase 1. 267 268* TcpOutSegs 269 270Defined in `RFC1213 tcpOutSegs`_ 271 272.. _RFC1213 tcpOutSegs: https://tools.ietf.org/html/rfc1213#page-48 273 274The number of packets sent by the TCP layer. As mentioned in RFC1213, 275it excludes the retransmitted packets. But it includes the SYN, ACK 276and RST packets. Doesn't like TcpInSegs, the TcpOutSegs is aware of 277GSO, so if a packet would be split to 2 by GSO, TcpOutSegs will 278increase 2. 279 280* TcpActiveOpens 281 282Defined in `RFC1213 tcpActiveOpens`_ 283 284.. _RFC1213 tcpActiveOpens: https://tools.ietf.org/html/rfc1213#page-47 285 286It means the TCP layer sends a SYN, and come into the SYN-SENT 287state. Every time TcpActiveOpens increases 1, TcpOutSegs should always 288increase 1. 289 290* TcpPassiveOpens 291 292Defined in `RFC1213 tcpPassiveOpens`_ 293 294.. _RFC1213 tcpPassiveOpens: https://tools.ietf.org/html/rfc1213#page-47 295 296It means the TCP layer receives a SYN, replies a SYN+ACK, come into 297the SYN-RCVD state. 298 299* TcpExtTCPRcvCoalesce 300 301When packets are received by the TCP layer and are not be read by the 302application, the TCP layer will try to merge them. This counter 303indicate how many packets are merged in such situation. If GRO is 304enabled, lots of packets would be merged by GRO, these packets 305wouldn't be counted to TcpExtTCPRcvCoalesce. 306 307* TcpExtTCPAutoCorking 308 309When sending packets, the TCP layer will try to merge small packets to 310a bigger one. This counter increase 1 for every packet merged in such 311situation. Please refer to the LWN article for more details: 312https://lwn.net/Articles/576263/ 313 314* TcpExtTCPOrigDataSent 315 316This counter is explained by `kernel commit f19c29e3e391`_, I pasted the 317explanation below:: 318 319 TCPOrigDataSent: number of outgoing packets with original data (excluding 320 retransmission but including data-in-SYN). This counter is different from 321 TcpOutSegs because TcpOutSegs also tracks pure ACKs. TCPOrigDataSent is 322 more useful to track the TCP retransmission rate. 323 324* TCPSynRetrans 325 326This counter is explained by `kernel commit f19c29e3e391`_, I pasted the 327explanation below:: 328 329 TCPSynRetrans: number of SYN and SYN/ACK retransmits to break down 330 retransmissions into SYN, fast-retransmits, timeout retransmits, etc. 331 332* TCPFastOpenActiveFail 333 334This counter is explained by `kernel commit f19c29e3e391`_, I pasted the 335explanation below:: 336 337 TCPFastOpenActiveFail: Fast Open attempts (SYN/data) failed because 338 the remote does not accept it or the attempts timed out. 339 340.. _kernel commit f19c29e3e391: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=f19c29e3e391a66a273e9afebaf01917245148cd 341 342* TcpExtListenOverflows and TcpExtListenDrops 343 344When kernel receives a SYN from a client, and if the TCP accept queue 345is full, kernel will drop the SYN and add 1 to TcpExtListenOverflows. 346At the same time kernel will also add 1 to TcpExtListenDrops. When a 347TCP socket is in LISTEN state, and kernel need to drop a packet, 348kernel would always add 1 to TcpExtListenDrops. So increase 349TcpExtListenOverflows would let TcpExtListenDrops increasing at the 350same time, but TcpExtListenDrops would also increase without 351TcpExtListenOverflows increasing, e.g. a memory allocation fail would 352also let TcpExtListenDrops increase. 353 354Note: The above explanation is based on kernel 4.10 or above version, on 355an old kernel, the TCP stack has different behavior when TCP accept 356queue is full. On the old kernel, TCP stack won't drop the SYN, it 357would complete the 3-way handshake. As the accept queue is full, TCP 358stack will keep the socket in the TCP half-open queue. As it is in the 359half open queue, TCP stack will send SYN+ACK on an exponential backoff 360timer, after client replies ACK, TCP stack checks whether the accept 361queue is still full, if it is not full, moves the socket to the accept 362queue, if it is full, keeps the socket in the half-open queue, at next 363time client replies ACK, this socket will get another chance to move 364to the accept queue. 365 366 367TCP Fast Open 368============= 369* TcpEstabResets 370 371Defined in `RFC1213 tcpEstabResets`_. 372 373.. _RFC1213 tcpEstabResets: https://tools.ietf.org/html/rfc1213#page-48 374 375* TcpAttemptFails 376 377Defined in `RFC1213 tcpAttemptFails`_. 378 379.. _RFC1213 tcpAttemptFails: https://tools.ietf.org/html/rfc1213#page-48 380 381* TcpOutRsts 382 383Defined in `RFC1213 tcpOutRsts`_. The RFC says this counter indicates 384the 'segments sent containing the RST flag', but in linux kernel, this 385counter indicates the segments kernel tried to send. The sending 386process might be failed due to some errors (e.g. memory alloc failed). 387 388.. _RFC1213 tcpOutRsts: https://tools.ietf.org/html/rfc1213#page-52 389 390* TcpExtTCPSpuriousRtxHostQueues 391 392When the TCP stack wants to retransmit a packet, and finds that packet 393is not lost in the network, but the packet is not sent yet, the TCP 394stack would give up the retransmission and update this counter. It 395might happen if a packet stays too long time in a qdisc or driver 396queue. 397 398* TcpEstabResets 399 400The socket receives a RST packet in Establish or CloseWait state. 401 402* TcpExtTCPKeepAlive 403 404This counter indicates many keepalive packets were sent. The keepalive 405won't be enabled by default. A userspace program could enable it by 406setting the SO_KEEPALIVE socket option. 407 408* TcpExtTCPSpuriousRTOs 409 410The spurious retransmission timeout detected by the `F-RTO`_ 411algorithm. 412 413.. _F-RTO: https://tools.ietf.org/html/rfc5682 414 415TCP Fast Path 416============= 417When kernel receives a TCP packet, it has two paths to handler the 418packet, one is fast path, another is slow path. The comment in kernel 419code provides a good explanation of them, I pasted them below:: 420 421 It is split into a fast path and a slow path. The fast path is 422 disabled when: 423 424 - A zero window was announced from us 425 - zero window probing 426 is only handled properly on the slow path. 427 - Out of order segments arrived. 428 - Urgent data is expected. 429 - There is no buffer space left 430 - Unexpected TCP flags/window values/header lengths are received 431 (detected by checking the TCP header against pred_flags) 432 - Data is sent in both directions. The fast path only supports pure senders 433 or pure receivers (this means either the sequence number or the ack 434 value must stay constant) 435 - Unexpected TCP option. 436 437Kernel will try to use fast path unless any of the above conditions 438are satisfied. If the packets are out of order, kernel will handle 439them in slow path, which means the performance might be not very 440good. Kernel would also come into slow path if the "Delayed ack" is 441used, because when using "Delayed ack", the data is sent in both 442directions. When the TCP window scale option is not used, kernel will 443try to enable fast path immediately when the connection comes into the 444established state, but if the TCP window scale option is used, kernel 445will disable the fast path at first, and try to enable it after kernel 446receives packets. 447 448* TcpExtTCPPureAcks and TcpExtTCPHPAcks 449 450If a packet set ACK flag and has no data, it is a pure ACK packet, if 451kernel handles it in the fast path, TcpExtTCPHPAcks will increase 1, 452if kernel handles it in the slow path, TcpExtTCPPureAcks will 453increase 1. 454 455* TcpExtTCPHPHits 456 457If a TCP packet has data (which means it is not a pure ACK packet), 458and this packet is handled in the fast path, TcpExtTCPHPHits will 459increase 1. 460 461 462TCP abort 463========= 464* TcpExtTCPAbortOnData 465 466It means TCP layer has data in flight, but need to close the 467connection. So TCP layer sends a RST to the other side, indicate the 468connection is not closed very graceful. An easy way to increase this 469counter is using the SO_LINGER option. Please refer to the SO_LINGER 470section of the `socket man page`_: 471 472.. _socket man page: http://man7.org/linux/man-pages/man7/socket.7.html 473 474By default, when an application closes a connection, the close function 475will return immediately and kernel will try to send the in-flight data 476async. If you use the SO_LINGER option, set l_onoff to 1, and l_linger 477to a positive number, the close function won't return immediately, but 478wait for the in-flight data are acked by the other side, the max wait 479time is l_linger seconds. If set l_onoff to 1 and set l_linger to 0, 480when the application closes a connection, kernel will send a RST 481immediately and increase the TcpExtTCPAbortOnData counter. 482 483* TcpExtTCPAbortOnClose 484 485This counter means the application has unread data in the TCP layer when 486the application wants to close the TCP connection. In such a situation, 487kernel will send a RST to the other side of the TCP connection. 488 489* TcpExtTCPAbortOnMemory 490 491When an application closes a TCP connection, kernel still need to track 492the connection, let it complete the TCP disconnect process. E.g. an 493app calls the close method of a socket, kernel sends fin to the other 494side of the connection, then the app has no relationship with the 495socket any more, but kernel need to keep the socket, this socket 496becomes an orphan socket, kernel waits for the reply of the other side, 497and would come to the TIME_WAIT state finally. When kernel has no 498enough memory to keep the orphan socket, kernel would send an RST to 499the other side, and delete the socket, in such situation, kernel will 500increase 1 to the TcpExtTCPAbortOnMemory. Two conditions would trigger 501TcpExtTCPAbortOnMemory: 502 5031. the memory used by the TCP protocol is higher than the third value of 504the tcp_mem. Please refer the tcp_mem section in the `TCP man page`_: 505 506.. _TCP man page: http://man7.org/linux/man-pages/man7/tcp.7.html 507 5082. the orphan socket count is higher than net.ipv4.tcp_max_orphans 509 510 511* TcpExtTCPAbortOnTimeout 512 513This counter will increase when any of the TCP timers expire. In such 514situation, kernel won't send RST, just give up the connection. 515 516* TcpExtTCPAbortOnLinger 517 518When a TCP connection comes into FIN_WAIT_2 state, instead of waiting 519for the fin packet from the other side, kernel could send a RST and 520delete the socket immediately. This is not the default behavior of 521Linux kernel TCP stack. By configuring the TCP_LINGER2 socket option, 522you could let kernel follow this behavior. 523 524* TcpExtTCPAbortFailed 525 526The kernel TCP layer will send RST if the `RFC2525 2.17 section`_ is 527satisfied. If an internal error occurs during this process, 528TcpExtTCPAbortFailed will be increased. 529 530.. _RFC2525 2.17 section: https://tools.ietf.org/html/rfc2525#page-50 531 532TCP Hybrid Slow Start 533===================== 534The Hybrid Slow Start algorithm is an enhancement of the traditional 535TCP congestion window Slow Start algorithm. It uses two pieces of 536information to detect whether the max bandwidth of the TCP path is 537approached. The two pieces of information are ACK train length and 538increase in packet delay. For detail information, please refer the 539`Hybrid Slow Start paper`_. Either ACK train length or packet delay 540hits a specific threshold, the congestion control algorithm will come 541into the Congestion Avoidance state. Until v4.20, two congestion 542control algorithms are using Hybrid Slow Start, they are cubic (the 543default congestion control algorithm) and cdg. Four snmp counters 544relate with the Hybrid Slow Start algorithm. 545 546.. _Hybrid Slow Start paper: https://pdfs.semanticscholar.org/25e9/ef3f03315782c7f1cbcd31b587857adae7d1.pdf 547 548* TcpExtTCPHystartTrainDetect 549 550How many times the ACK train length threshold is detected 551 552* TcpExtTCPHystartTrainCwnd 553 554The sum of CWND detected by ACK train length. Dividing this value by 555TcpExtTCPHystartTrainDetect is the average CWND which detected by the 556ACK train length. 557 558* TcpExtTCPHystartDelayDetect 559 560How many times the packet delay threshold is detected. 561 562* TcpExtTCPHystartDelayCwnd 563 564The sum of CWND detected by packet delay. Dividing this value by 565TcpExtTCPHystartDelayDetect is the average CWND which detected by the 566packet delay. 567 568TCP retransmission and congestion control 569========================================= 570The TCP protocol has two retransmission mechanisms: SACK and fast 571recovery. They are exclusive with each other. When SACK is enabled, 572the kernel TCP stack would use SACK, or kernel would use fast 573recovery. The SACK is a TCP option, which is defined in `RFC2018`_, 574the fast recovery is defined in `RFC6582`_, which is also called 575'Reno'. 576 577The TCP congestion control is a big and complex topic. To understand 578the related snmp counter, we need to know the states of the congestion 579control state machine. There are 5 states: Open, Disorder, CWR, 580Recovery and Loss. For details about these states, please refer page 5 581and page 6 of this document: 582https://pdfs.semanticscholar.org/0e9c/968d09ab2e53e24c4dca5b2d67c7f7140f8e.pdf 583 584.. _RFC2018: https://tools.ietf.org/html/rfc2018 585.. _RFC6582: https://tools.ietf.org/html/rfc6582 586 587* TcpExtTCPRenoRecovery and TcpExtTCPSackRecovery 588 589When the congestion control comes into Recovery state, if sack is 590used, TcpExtTCPSackRecovery increases 1, if sack is not used, 591TcpExtTCPRenoRecovery increases 1. These two counters mean the TCP 592stack begins to retransmit the lost packets. 593 594* TcpExtTCPSACKReneging 595 596A packet was acknowledged by SACK, but the receiver has dropped this 597packet, so the sender needs to retransmit this packet. In this 598situation, the sender adds 1 to TcpExtTCPSACKReneging. A receiver 599could drop a packet which has been acknowledged by SACK, although it is 600unusual, it is allowed by the TCP protocol. The sender doesn't really 601know what happened on the receiver side. The sender just waits until 602the RTO expires for this packet, then the sender assumes this packet 603has been dropped by the receiver. 604 605* TcpExtTCPRenoReorder 606 607The reorder packet is detected by fast recovery. It would only be used 608if SACK is disabled. The fast recovery algorithm detects recorder by 609the duplicate ACK number. E.g., if retransmission is triggered, and 610the original retransmitted packet is not lost, it is just out of 611order, the receiver would acknowledge multiple times, one for the 612retransmitted packet, another for the arriving of the original out of 613order packet. Thus the sender would find more ACks than its 614expectation, and the sender knows out of order occurs. 615 616* TcpExtTCPTSReorder 617 618The reorder packet is detected when a hole is filled. E.g., assume the 619sender sends packet 1,2,3,4,5, and the receiving order is 6201,2,4,5,3. When the sender receives the ACK of packet 3 (which will 621fill the hole), two conditions will let TcpExtTCPTSReorder increase 6221: (1) if the packet 3 is not re-retransmitted yet. (2) if the packet 6233 is retransmitted but the timestamp of the packet 3's ACK is earlier 624than the retransmission timestamp. 625 626* TcpExtTCPSACKReorder 627 628The reorder packet detected by SACK. The SACK has two methods to 629detect reorder: (1) DSACK is received by the sender. It means the 630sender sends the same packet more than one times. And the only reason 631is the sender believes an out of order packet is lost so it sends the 632packet again. (2) Assume packet 1,2,3,4,5 are sent by the sender, and 633the sender has received SACKs for packet 2 and 5, now the sender 634receives SACK for packet 4 and the sender doesn't retransmit the 635packet yet, the sender would know packet 4 is out of order. The TCP 636stack of kernel will increase TcpExtTCPSACKReorder for both of the 637above scenarios. 638 639* TcpExtTCPSlowStartRetrans 640 641The TCP stack wants to retransmit a packet and the congestion control 642state is 'Loss'. 643 644* TcpExtTCPFastRetrans 645 646The TCP stack wants to retransmit a packet and the congestion control 647state is not 'Loss'. 648 649* TcpExtTCPLostRetransmit 650 651A SACK points out that a retransmission packet is lost again. 652 653* TcpExtTCPRetransFail 654 655The TCP stack tries to deliver a retransmission packet to lower layers 656but the lower layers return an error. 657 658* TcpExtTCPSynRetrans 659 660The TCP stack retransmits a SYN packet. 661 662DSACK 663===== 664The DSACK is defined in `RFC2883`_. The receiver uses DSACK to report 665duplicate packets to the sender. There are two kinds of 666duplications: (1) a packet which has been acknowledged is 667duplicate. (2) an out of order packet is duplicate. The TCP stack 668counts these two kinds of duplications on both receiver side and 669sender side. 670 671.. _RFC2883 : https://tools.ietf.org/html/rfc2883 672 673* TcpExtTCPDSACKOldSent 674 675The TCP stack receives a duplicate packet which has been acked, so it 676sends a DSACK to the sender. 677 678* TcpExtTCPDSACKOfoSent 679 680The TCP stack receives an out of order duplicate packet, so it sends a 681DSACK to the sender. 682 683* TcpExtTCPDSACKRecv 684 685The TCP stack receives a DSACK, which indicates an acknowledged 686duplicate packet is received. 687 688* TcpExtTCPDSACKOfoRecv 689 690The TCP stack receives a DSACK, which indicate an out of order 691duplicate packet is received. 692 693invalid SACK and DSACK 694====================== 695When a SACK (or DSACK) block is invalid, a corresponding counter would 696be updated. The validation method is base on the start/end sequence 697number of the SACK block. For more details, please refer the comment 698of the function tcp_is_sackblock_valid in the kernel source code. A 699SACK option could have up to 4 blocks, they are checked 700individually. E.g., if 3 blocks of a SACk is invalid, the 701corresponding counter would be updated 3 times. The comment of the 702`Add counters for discarded SACK blocks`_ patch has additional 703explanation: 704 705.. _Add counters for discarded SACK blocks: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=18f02545a9a16c9a89778b91a162ad16d510bb32 706 707* TcpExtTCPSACKDiscard 708 709This counter indicates how many SACK blocks are invalid. If the invalid 710SACK block is caused by ACK recording, the TCP stack will only ignore 711it and won't update this counter. 712 713* TcpExtTCPDSACKIgnoredOld and TcpExtTCPDSACKIgnoredNoUndo 714 715When a DSACK block is invalid, one of these two counters would be 716updated. Which counter will be updated depends on the undo_marker flag 717of the TCP socket. If the undo_marker is not set, the TCP stack isn't 718likely to re-transmit any packets, and we still receive an invalid 719DSACK block, the reason might be that the packet is duplicated in the 720middle of the network. In such scenario, TcpExtTCPDSACKIgnoredNoUndo 721will be updated. If the undo_marker is set, TcpExtTCPDSACKIgnoredOld 722will be updated. As implied in its name, it might be an old packet. 723 724SACK shift 725========== 726The linux networking stack stores data in sk_buff struct (skb for 727short). If a SACK block acrosses multiple skb, the TCP stack will try 728to re-arrange data in these skb. E.g. if a SACK block acknowledges seq 72910 to 15, skb1 has seq 10 to 13, skb2 has seq 14 to 20. The seq 14 and 73015 in skb2 would be moved to skb1. This operation is 'shift'. If a 731SACK block acknowledges seq 10 to 20, skb1 has seq 10 to 13, skb2 has 732seq 14 to 20. All data in skb2 will be moved to skb1, and skb2 will be 733discard, this operation is 'merge'. 734 735* TcpExtTCPSackShifted 736 737A skb is shifted 738 739* TcpExtTCPSackMerged 740 741A skb is merged 742 743* TcpExtTCPSackShiftFallback 744 745A skb should be shifted or merged, but the TCP stack doesn't do it for 746some reasons. 747 748TCP out of order 749================ 750* TcpExtTCPOFOQueue 751 752The TCP layer receives an out of order packet and has enough memory 753to queue it. 754 755* TcpExtTCPOFODrop 756 757The TCP layer receives an out of order packet but doesn't have enough 758memory, so drops it. Such packets won't be counted into 759TcpExtTCPOFOQueue. 760 761* TcpExtTCPOFOMerge 762 763The received out of order packet has an overlay with the previous 764packet. the overlay part will be dropped. All of TcpExtTCPOFOMerge 765packets will also be counted into TcpExtTCPOFOQueue. 766 767TCP PAWS 768======== 769PAWS (Protection Against Wrapped Sequence numbers) is an algorithm 770which is used to drop old packets. It depends on the TCP 771timestamps. For detail information, please refer the `timestamp wiki`_ 772and the `RFC of PAWS`_. 773 774.. _RFC of PAWS: https://tools.ietf.org/html/rfc1323#page-17 775.. _timestamp wiki: https://en.wikipedia.org/wiki/Transmission_Control_Protocol#TCP_timestamps 776 777* TcpExtPAWSActive 778 779Packets are dropped by PAWS in Syn-Sent status. 780 781* TcpExtPAWSEstab 782 783Packets are dropped by PAWS in any status other than Syn-Sent. 784 785TCP ACK skip 786============ 787In some scenarios, kernel would avoid sending duplicate ACKs too 788frequently. Please find more details in the tcp_invalid_ratelimit 789section of the `sysctl document`_. When kernel decides to skip an ACK 790due to tcp_invalid_ratelimit, kernel would update one of below 791counters to indicate the ACK is skipped in which scenario. The ACK 792would only be skipped if the received packet is either a SYN packet or 793it has no data. 794 795.. _sysctl document: https://www.kernel.org/doc/Documentation/networking/ip-sysctl.rst 796 797* TcpExtTCPACKSkippedSynRecv 798 799The ACK is skipped in Syn-Recv status. The Syn-Recv status means the 800TCP stack receives a SYN and replies SYN+ACK. Now the TCP stack is 801waiting for an ACK. Generally, the TCP stack doesn't need to send ACK 802in the Syn-Recv status. But in several scenarios, the TCP stack need 803to send an ACK. E.g., the TCP stack receives the same SYN packet 804repeately, the received packet does not pass the PAWS check, or the 805received packet sequence number is out of window. In these scenarios, 806the TCP stack needs to send ACK. If the ACk sending frequency is higher than 807tcp_invalid_ratelimit allows, the TCP stack will skip sending ACK and 808increase TcpExtTCPACKSkippedSynRecv. 809 810 811* TcpExtTCPACKSkippedPAWS 812 813The ACK is skipped due to PAWS (Protect Against Wrapped Sequence 814numbers) check fails. If the PAWS check fails in Syn-Recv, Fin-Wait-2 815or Time-Wait statuses, the skipped ACK would be counted to 816TcpExtTCPACKSkippedSynRecv, TcpExtTCPACKSkippedFinWait2 or 817TcpExtTCPACKSkippedTimeWait. In all other statuses, the skipped ACK 818would be counted to TcpExtTCPACKSkippedPAWS. 819 820* TcpExtTCPACKSkippedSeq 821 822The sequence number is out of window and the timestamp passes the PAWS 823check and the TCP status is not Syn-Recv, Fin-Wait-2, and Time-Wait. 824 825* TcpExtTCPACKSkippedFinWait2 826 827The ACK is skipped in Fin-Wait-2 status, the reason would be either 828PAWS check fails or the received sequence number is out of window. 829 830* TcpExtTCPACKSkippedTimeWait 831 832The ACK is skipped in Time-Wait status, the reason would be either 833PAWS check failed or the received sequence number is out of window. 834 835* TcpExtTCPACKSkippedChallenge 836 837The ACK is skipped if the ACK is a challenge ACK. The RFC 5961 defines 8383 kind of challenge ACK, please refer `RFC 5961 section 3.2`_, 839`RFC 5961 section 4.2`_ and `RFC 5961 section 5.2`_. Besides these 840three scenarios, In some TCP status, the linux TCP stack would also 841send challenge ACKs if the ACK number is before the first 842unacknowledged number (more strict than `RFC 5961 section 5.2`_). 843 844.. _RFC 5961 section 3.2: https://tools.ietf.org/html/rfc5961#page-7 845.. _RFC 5961 section 4.2: https://tools.ietf.org/html/rfc5961#page-9 846.. _RFC 5961 section 5.2: https://tools.ietf.org/html/rfc5961#page-11 847 848TCP receive window 849================== 850* TcpExtTCPWantZeroWindowAdv 851 852Depending on current memory usage, the TCP stack tries to set receive 853window to zero. But the receive window might still be a no-zero 854value. For example, if the previous window size is 10, and the TCP 855stack receives 3 bytes, the current window size would be 7 even if the 856window size calculated by the memory usage is zero. 857 858* TcpExtTCPToZeroWindowAdv 859 860The TCP receive window is set to zero from a no-zero value. 861 862* TcpExtTCPFromZeroWindowAdv 863 864The TCP receive window is set to no-zero value from zero. 865 866 867Delayed ACK 868=========== 869The TCP Delayed ACK is a technique which is used for reducing the 870packet count in the network. For more details, please refer the 871`Delayed ACK wiki`_ 872 873.. _Delayed ACK wiki: https://en.wikipedia.org/wiki/TCP_delayed_acknowledgment 874 875* TcpExtDelayedACKs 876 877A delayed ACK timer expires. The TCP stack will send a pure ACK packet 878and exit the delayed ACK mode. 879 880* TcpExtDelayedACKLocked 881 882A delayed ACK timer expires, but the TCP stack can't send an ACK 883immediately due to the socket is locked by a userspace program. The 884TCP stack will send a pure ACK later (after the userspace program 885unlock the socket). When the TCP stack sends the pure ACK later, the 886TCP stack will also update TcpExtDelayedACKs and exit the delayed ACK 887mode. 888 889* TcpExtDelayedACKLost 890 891It will be updated when the TCP stack receives a packet which has been 892ACKed. A Delayed ACK loss might cause this issue, but it would also be 893triggered by other reasons, such as a packet is duplicated in the 894network. 895 896Tail Loss Probe (TLP) 897===================== 898TLP is an algorithm which is used to detect TCP packet loss. For more 899details, please refer the `TLP paper`_. 900 901.. _TLP paper: https://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01 902 903* TcpExtTCPLossProbes 904 905A TLP probe packet is sent. 906 907* TcpExtTCPLossProbeRecovery 908 909A packet loss is detected and recovered by TLP. 910 911TCP Fast Open description 912========================= 913TCP Fast Open is a technology which allows data transfer before the 9143-way handshake complete. Please refer the `TCP Fast Open wiki`_ for a 915general description. 916 917.. _TCP Fast Open wiki: https://en.wikipedia.org/wiki/TCP_Fast_Open 918 919* TcpExtTCPFastOpenActive 920 921When the TCP stack receives an ACK packet in the SYN-SENT status, and 922the ACK packet acknowledges the data in the SYN packet, the TCP stack 923understand the TFO cookie is accepted by the other side, then it 924updates this counter. 925 926* TcpExtTCPFastOpenActiveFail 927 928This counter indicates that the TCP stack initiated a TCP Fast Open, 929but it failed. This counter would be updated in three scenarios: (1) 930the other side doesn't acknowledge the data in the SYN packet. (2) The 931SYN packet which has the TFO cookie is timeout at least once. (3) 932after the 3-way handshake, the retransmission timeout happens 933net.ipv4.tcp_retries1 times, because some middle-boxes may black-hole 934fast open after the handshake. 935 936* TcpExtTCPFastOpenPassive 937 938This counter indicates how many times the TCP stack accepts the fast 939open request. 940 941* TcpExtTCPFastOpenPassiveFail 942 943This counter indicates how many times the TCP stack rejects the fast 944open request. It is caused by either the TFO cookie is invalid or the 945TCP stack finds an error during the socket creating process. 946 947* TcpExtTCPFastOpenListenOverflow 948 949When the pending fast open request number is larger than 950fastopenq->max_qlen, the TCP stack will reject the fast open request 951and update this counter. When this counter is updated, the TCP stack 952won't update TcpExtTCPFastOpenPassive or 953TcpExtTCPFastOpenPassiveFail. The fastopenq->max_qlen is set by the 954TCP_FASTOPEN socket operation and it could not be larger than 955net.core.somaxconn. For example: 956 957setsockopt(sfd, SOL_TCP, TCP_FASTOPEN, &qlen, sizeof(qlen)); 958 959* TcpExtTCPFastOpenCookieReqd 960 961This counter indicates how many times a client wants to request a TFO 962cookie. 963 964SYN cookies 965=========== 966SYN cookies are used to mitigate SYN flood, for details, please refer 967the `SYN cookies wiki`_. 968 969.. _SYN cookies wiki: https://en.wikipedia.org/wiki/SYN_cookies 970 971* TcpExtSyncookiesSent 972 973It indicates how many SYN cookies are sent. 974 975* TcpExtSyncookiesRecv 976 977How many reply packets of the SYN cookies the TCP stack receives. 978 979* TcpExtSyncookiesFailed 980 981The MSS decoded from the SYN cookie is invalid. When this counter is 982updated, the received packet won't be treated as a SYN cookie and the 983TcpExtSyncookiesRecv counter wont be updated. 984 985Challenge ACK 986============= 987For details of challenge ACK, please refer the explanation of 988TcpExtTCPACKSkippedChallenge. 989 990* TcpExtTCPChallengeACK 991 992The number of challenge acks sent. 993 994* TcpExtTCPSYNChallenge 995 996The number of challenge acks sent in response to SYN packets. After 997updates this counter, the TCP stack might send a challenge ACK and 998update the TcpExtTCPChallengeACK counter, or it might also skip to 999send the challenge and update the TcpExtTCPACKSkippedChallenge. 1000 1001prune 1002===== 1003When a socket is under memory pressure, the TCP stack will try to 1004reclaim memory from the receiving queue and out of order queue. One of 1005the reclaiming method is 'collapse', which means allocate a big skb, 1006copy the contiguous skbs to the single big skb, and free these 1007contiguous skbs. 1008 1009* TcpExtPruneCalled 1010 1011The TCP stack tries to reclaim memory for a socket. After updates this 1012counter, the TCP stack will try to collapse the out of order queue and 1013the receiving queue. If the memory is still not enough, the TCP stack 1014will try to discard packets from the out of order queue (and update the 1015TcpExtOfoPruned counter) 1016 1017* TcpExtOfoPruned 1018 1019The TCP stack tries to discard packet on the out of order queue. 1020 1021* TcpExtRcvPruned 1022 1023After 'collapse' and discard packets from the out of order queue, if 1024the actually used memory is still larger than the max allowed memory, 1025this counter will be updated. It means the 'prune' fails. 1026 1027* TcpExtTCPRcvCollapsed 1028 1029This counter indicates how many skbs are freed during 'collapse'. 1030 1031examples 1032======== 1033 1034ping test 1035--------- 1036Run the ping command against the public dns server 8.8.8.8:: 1037 1038 nstatuser@nstat-a:~$ ping 8.8.8.8 -c 1 1039 PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data. 1040 64 bytes from 8.8.8.8: icmp_seq=1 ttl=119 time=17.8 ms 1041 1042 --- 8.8.8.8 ping statistics --- 1043 1 packets transmitted, 1 received, 0% packet loss, time 0ms 1044 rtt min/avg/max/mdev = 17.875/17.875/17.875/0.000 ms 1045 1046The nstayt result:: 1047 1048 nstatuser@nstat-a:~$ nstat 1049 #kernel 1050 IpInReceives 1 0.0 1051 IpInDelivers 1 0.0 1052 IpOutRequests 1 0.0 1053 IcmpInMsgs 1 0.0 1054 IcmpInEchoReps 1 0.0 1055 IcmpOutMsgs 1 0.0 1056 IcmpOutEchos 1 0.0 1057 IcmpMsgInType0 1 0.0 1058 IcmpMsgOutType8 1 0.0 1059 IpExtInOctets 84 0.0 1060 IpExtOutOctets 84 0.0 1061 IpExtInNoECTPkts 1 0.0 1062 1063The Linux server sent an ICMP Echo packet, so IpOutRequests, 1064IcmpOutMsgs, IcmpOutEchos and IcmpMsgOutType8 were increased 1. The 1065server got ICMP Echo Reply from 8.8.8.8, so IpInReceives, IcmpInMsgs, 1066IcmpInEchoReps and IcmpMsgInType0 were increased 1. The ICMP Echo Reply 1067was passed to the ICMP layer via IP layer, so IpInDelivers was 1068increased 1. The default ping data size is 48, so an ICMP Echo packet 1069and its corresponding Echo Reply packet are constructed by: 1070 1071* 14 bytes MAC header 1072* 20 bytes IP header 1073* 16 bytes ICMP header 1074* 48 bytes data (default value of the ping command) 1075 1076So the IpExtInOctets and IpExtOutOctets are 20+16+48=84. 1077 1078tcp 3-way handshake 1079------------------- 1080On server side, we run:: 1081 1082 nstatuser@nstat-b:~$ nc -lknv 0.0.0.0 9000 1083 Listening on [0.0.0.0] (family 0, port 9000) 1084 1085On client side, we run:: 1086 1087 nstatuser@nstat-a:~$ nc -nv 192.168.122.251 9000 1088 Connection to 192.168.122.251 9000 port [tcp/*] succeeded! 1089 1090The server listened on tcp 9000 port, the client connected to it, they 1091completed the 3-way handshake. 1092 1093On server side, we can find below nstat output:: 1094 1095 nstatuser@nstat-b:~$ nstat | grep -i tcp 1096 TcpPassiveOpens 1 0.0 1097 TcpInSegs 2 0.0 1098 TcpOutSegs 1 0.0 1099 TcpExtTCPPureAcks 1 0.0 1100 1101On client side, we can find below nstat output:: 1102 1103 nstatuser@nstat-a:~$ nstat | grep -i tcp 1104 TcpActiveOpens 1 0.0 1105 TcpInSegs 1 0.0 1106 TcpOutSegs 2 0.0 1107 1108When the server received the first SYN, it replied a SYN+ACK, and came into 1109SYN-RCVD state, so TcpPassiveOpens increased 1. The server received 1110SYN, sent SYN+ACK, received ACK, so server sent 1 packet, received 2 1111packets, TcpInSegs increased 2, TcpOutSegs increased 1. The last ACK 1112of the 3-way handshake is a pure ACK without data, so 1113TcpExtTCPPureAcks increased 1. 1114 1115When the client sent SYN, the client came into the SYN-SENT state, so 1116TcpActiveOpens increased 1, the client sent SYN, received SYN+ACK, sent 1117ACK, so client sent 2 packets, received 1 packet, TcpInSegs increased 11181, TcpOutSegs increased 2. 1119 1120TCP normal traffic 1121------------------ 1122Run nc on server:: 1123 1124 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 1125 Listening on [0.0.0.0] (family 0, port 9000) 1126 1127Run nc on client:: 1128 1129 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1130 Connection to nstat-b 9000 port [tcp/*] succeeded! 1131 1132Input a string in the nc client ('hello' in our example):: 1133 1134 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1135 Connection to nstat-b 9000 port [tcp/*] succeeded! 1136 hello 1137 1138The client side nstat output:: 1139 1140 nstatuser@nstat-a:~$ nstat 1141 #kernel 1142 IpInReceives 1 0.0 1143 IpInDelivers 1 0.0 1144 IpOutRequests 1 0.0 1145 TcpInSegs 1 0.0 1146 TcpOutSegs 1 0.0 1147 TcpExtTCPPureAcks 1 0.0 1148 TcpExtTCPOrigDataSent 1 0.0 1149 IpExtInOctets 52 0.0 1150 IpExtOutOctets 58 0.0 1151 IpExtInNoECTPkts 1 0.0 1152 1153The server side nstat output:: 1154 1155 nstatuser@nstat-b:~$ nstat 1156 #kernel 1157 IpInReceives 1 0.0 1158 IpInDelivers 1 0.0 1159 IpOutRequests 1 0.0 1160 TcpInSegs 1 0.0 1161 TcpOutSegs 1 0.0 1162 IpExtInOctets 58 0.0 1163 IpExtOutOctets 52 0.0 1164 IpExtInNoECTPkts 1 0.0 1165 1166Input a string in nc client side again ('world' in our example):: 1167 1168 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1169 Connection to nstat-b 9000 port [tcp/*] succeeded! 1170 hello 1171 world 1172 1173Client side nstat output:: 1174 1175 nstatuser@nstat-a:~$ nstat 1176 #kernel 1177 IpInReceives 1 0.0 1178 IpInDelivers 1 0.0 1179 IpOutRequests 1 0.0 1180 TcpInSegs 1 0.0 1181 TcpOutSegs 1 0.0 1182 TcpExtTCPHPAcks 1 0.0 1183 TcpExtTCPOrigDataSent 1 0.0 1184 IpExtInOctets 52 0.0 1185 IpExtOutOctets 58 0.0 1186 IpExtInNoECTPkts 1 0.0 1187 1188 1189Server side nstat output:: 1190 1191 nstatuser@nstat-b:~$ nstat 1192 #kernel 1193 IpInReceives 1 0.0 1194 IpInDelivers 1 0.0 1195 IpOutRequests 1 0.0 1196 TcpInSegs 1 0.0 1197 TcpOutSegs 1 0.0 1198 TcpExtTCPHPHits 1 0.0 1199 IpExtInOctets 58 0.0 1200 IpExtOutOctets 52 0.0 1201 IpExtInNoECTPkts 1 0.0 1202 1203Compare the first client-side nstat and the second client-side nstat, 1204we could find one difference: the first one had a 'TcpExtTCPPureAcks', 1205but the second one had a 'TcpExtTCPHPAcks'. The first server-side 1206nstat and the second server-side nstat had a difference too: the 1207second server-side nstat had a TcpExtTCPHPHits, but the first 1208server-side nstat didn't have it. The network traffic patterns were 1209exactly the same: the client sent a packet to the server, the server 1210replied an ACK. But kernel handled them in different ways. When the 1211TCP window scale option is not used, kernel will try to enable fast 1212path immediately when the connection comes into the established state, 1213but if the TCP window scale option is used, kernel will disable the 1214fast path at first, and try to enable it after kernel receives 1215packets. We could use the 'ss' command to verify whether the window 1216scale option is used. e.g. run below command on either server or 1217client:: 1218 1219 nstatuser@nstat-a:~$ ss -o state established -i '( dport = :9000 or sport = :9000 ) 1220 Netid Recv-Q Send-Q Local Address:Port Peer Address:Port 1221 tcp 0 0 192.168.122.250:40654 192.168.122.251:9000 1222 ts sack cubic wscale:7,7 rto:204 rtt:0.98/0.49 mss:1448 pmtu:1500 rcvmss:536 advmss:1448 cwnd:10 bytes_acked:1 segs_out:2 segs_in:1 send 118.2Mbps lastsnd:46572 lastrcv:46572 lastack:46572 pacing_rate 236.4Mbps rcv_space:29200 rcv_ssthresh:29200 minrtt:0.98 1223 1224The 'wscale:7,7' means both server and client set the window scale 1225option to 7. Now we could explain the nstat output in our test: 1226 1227In the first nstat output of client side, the client sent a packet, server 1228reply an ACK, when kernel handled this ACK, the fast path was not 1229enabled, so the ACK was counted into 'TcpExtTCPPureAcks'. 1230 1231In the second nstat output of client side, the client sent a packet again, 1232and received another ACK from the server, in this time, the fast path is 1233enabled, and the ACK was qualified for fast path, so it was handled by 1234the fast path, so this ACK was counted into TcpExtTCPHPAcks. 1235 1236In the first nstat output of server side, fast path was not enabled, 1237so there was no 'TcpExtTCPHPHits'. 1238 1239In the second nstat output of server side, the fast path was enabled, 1240and the packet received from client qualified for fast path, so it 1241was counted into 'TcpExtTCPHPHits'. 1242 1243TcpExtTCPAbortOnClose 1244--------------------- 1245On the server side, we run below python script:: 1246 1247 import socket 1248 import time 1249 1250 port = 9000 1251 1252 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1253 s.bind(('0.0.0.0', port)) 1254 s.listen(1) 1255 sock, addr = s.accept() 1256 while True: 1257 time.sleep(9999999) 1258 1259This python script listen on 9000 port, but doesn't read anything from 1260the connection. 1261 1262On the client side, we send the string "hello" by nc:: 1263 1264 nstatuser@nstat-a:~$ echo "hello" | nc nstat-b 9000 1265 1266Then, we come back to the server side, the server has received the "hello" 1267packet, and the TCP layer has acked this packet, but the application didn't 1268read it yet. We type Ctrl-C to terminate the server script. Then we 1269could find TcpExtTCPAbortOnClose increased 1 on the server side:: 1270 1271 nstatuser@nstat-b:~$ nstat | grep -i abort 1272 TcpExtTCPAbortOnClose 1 0.0 1273 1274If we run tcpdump on the server side, we could find the server sent a 1275RST after we type Ctrl-C. 1276 1277TcpExtTCPAbortOnMemory and TcpExtTCPAbortOnTimeout 1278--------------------------------------------------- 1279Below is an example which let the orphan socket count be higher than 1280net.ipv4.tcp_max_orphans. 1281Change tcp_max_orphans to a smaller value on client:: 1282 1283 sudo bash -c "echo 10 > /proc/sys/net/ipv4/tcp_max_orphans" 1284 1285Client code (create 64 connection to server):: 1286 1287 nstatuser@nstat-a:~$ cat client_orphan.py 1288 import socket 1289 import time 1290 1291 server = 'nstat-b' # server address 1292 port = 9000 1293 1294 count = 64 1295 1296 connection_list = [] 1297 1298 for i in range(64): 1299 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1300 s.connect((server, port)) 1301 connection_list.append(s) 1302 print("connection_count: %d" % len(connection_list)) 1303 1304 while True: 1305 time.sleep(99999) 1306 1307Server code (accept 64 connection from client):: 1308 1309 nstatuser@nstat-b:~$ cat server_orphan.py 1310 import socket 1311 import time 1312 1313 port = 9000 1314 count = 64 1315 1316 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1317 s.bind(('0.0.0.0', port)) 1318 s.listen(count) 1319 connection_list = [] 1320 while True: 1321 sock, addr = s.accept() 1322 connection_list.append((sock, addr)) 1323 print("connection_count: %d" % len(connection_list)) 1324 1325Run the python scripts on server and client. 1326 1327On server:: 1328 1329 python3 server_orphan.py 1330 1331On client:: 1332 1333 python3 client_orphan.py 1334 1335Run iptables on server:: 1336 1337 sudo iptables -A INPUT -i ens3 -p tcp --destination-port 9000 -j DROP 1338 1339Type Ctrl-C on client, stop client_orphan.py. 1340 1341Check TcpExtTCPAbortOnMemory on client:: 1342 1343 nstatuser@nstat-a:~$ nstat | grep -i abort 1344 TcpExtTCPAbortOnMemory 54 0.0 1345 1346Check orphaned socket count on client:: 1347 1348 nstatuser@nstat-a:~$ ss -s 1349 Total: 131 (kernel 0) 1350 TCP: 14 (estab 1, closed 0, orphaned 10, synrecv 0, timewait 0/0), ports 0 1351 1352 Transport Total IP IPv6 1353 * 0 - - 1354 RAW 1 0 1 1355 UDP 1 1 0 1356 TCP 14 13 1 1357 INET 16 14 2 1358 FRAG 0 0 0 1359 1360The explanation of the test: after run server_orphan.py and 1361client_orphan.py, we set up 64 connections between server and 1362client. Run the iptables command, the server will drop all packets from 1363the client, type Ctrl-C on client_orphan.py, the system of the client 1364would try to close these connections, and before they are closed 1365gracefully, these connections became orphan sockets. As the iptables 1366of the server blocked packets from the client, the server won't receive fin 1367from the client, so all connection on clients would be stuck on FIN_WAIT_1 1368stage, so they will keep as orphan sockets until timeout. We have echo 136910 to /proc/sys/net/ipv4/tcp_max_orphans, so the client system would 1370only keep 10 orphan sockets, for all other orphan sockets, the client 1371system sent RST for them and delete them. We have 64 connections, so 1372the 'ss -s' command shows the system has 10 orphan sockets, and the 1373value of TcpExtTCPAbortOnMemory was 54. 1374 1375An additional explanation about orphan socket count: You could find the 1376exactly orphan socket count by the 'ss -s' command, but when kernel 1377decide whither increases TcpExtTCPAbortOnMemory and sends RST, kernel 1378doesn't always check the exactly orphan socket count. For increasing 1379performance, kernel checks an approximate count firstly, if the 1380approximate count is more than tcp_max_orphans, kernel checks the 1381exact count again. So if the approximate count is less than 1382tcp_max_orphans, but exactly count is more than tcp_max_orphans, you 1383would find TcpExtTCPAbortOnMemory is not increased at all. If 1384tcp_max_orphans is large enough, it won't occur, but if you decrease 1385tcp_max_orphans to a small value like our test, you might find this 1386issue. So in our test, the client set up 64 connections although the 1387tcp_max_orphans is 10. If the client only set up 11 connections, we 1388can't find the change of TcpExtTCPAbortOnMemory. 1389 1390Continue the previous test, we wait for several minutes. Because of the 1391iptables on the server blocked the traffic, the server wouldn't receive 1392fin, and all the client's orphan sockets would timeout on the 1393FIN_WAIT_1 state finally. So we wait for a few minutes, we could find 139410 timeout on the client:: 1395 1396 nstatuser@nstat-a:~$ nstat | grep -i abort 1397 TcpExtTCPAbortOnTimeout 10 0.0 1398 1399TcpExtTCPAbortOnLinger 1400---------------------- 1401The server side code:: 1402 1403 nstatuser@nstat-b:~$ cat server_linger.py 1404 import socket 1405 import time 1406 1407 port = 9000 1408 1409 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1410 s.bind(('0.0.0.0', port)) 1411 s.listen(1) 1412 sock, addr = s.accept() 1413 while True: 1414 time.sleep(9999999) 1415 1416The client side code:: 1417 1418 nstatuser@nstat-a:~$ cat client_linger.py 1419 import socket 1420 import struct 1421 1422 server = 'nstat-b' # server address 1423 port = 9000 1424 1425 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1426 s.setsockopt(socket.SOL_SOCKET, socket.SO_LINGER, struct.pack('ii', 1, 10)) 1427 s.setsockopt(socket.SOL_TCP, socket.TCP_LINGER2, struct.pack('i', -1)) 1428 s.connect((server, port)) 1429 s.close() 1430 1431Run server_linger.py on server:: 1432 1433 nstatuser@nstat-b:~$ python3 server_linger.py 1434 1435Run client_linger.py on client:: 1436 1437 nstatuser@nstat-a:~$ python3 client_linger.py 1438 1439After run client_linger.py, check the output of nstat:: 1440 1441 nstatuser@nstat-a:~$ nstat | grep -i abort 1442 TcpExtTCPAbortOnLinger 1 0.0 1443 1444TcpExtTCPRcvCoalesce 1445-------------------- 1446On the server, we run a program which listen on TCP port 9000, but 1447doesn't read any data:: 1448 1449 import socket 1450 import time 1451 port = 9000 1452 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1453 s.bind(('0.0.0.0', port)) 1454 s.listen(1) 1455 sock, addr = s.accept() 1456 while True: 1457 time.sleep(9999999) 1458 1459Save the above code as server_coalesce.py, and run:: 1460 1461 python3 server_coalesce.py 1462 1463On the client, save below code as client_coalesce.py:: 1464 1465 import socket 1466 server = 'nstat-b' 1467 port = 9000 1468 s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) 1469 s.connect((server, port)) 1470 1471Run:: 1472 1473 nstatuser@nstat-a:~$ python3 -i client_coalesce.py 1474 1475We use '-i' to come into the interactive mode, then a packet:: 1476 1477 >>> s.send(b'foo') 1478 3 1479 1480Send a packet again:: 1481 1482 >>> s.send(b'bar') 1483 3 1484 1485On the server, run nstat:: 1486 1487 ubuntu@nstat-b:~$ nstat 1488 #kernel 1489 IpInReceives 2 0.0 1490 IpInDelivers 2 0.0 1491 IpOutRequests 2 0.0 1492 TcpInSegs 2 0.0 1493 TcpOutSegs 2 0.0 1494 TcpExtTCPRcvCoalesce 1 0.0 1495 IpExtInOctets 110 0.0 1496 IpExtOutOctets 104 0.0 1497 IpExtInNoECTPkts 2 0.0 1498 1499The client sent two packets, server didn't read any data. When 1500the second packet arrived at server, the first packet was still in 1501the receiving queue. So the TCP layer merged the two packets, and we 1502could find the TcpExtTCPRcvCoalesce increased 1. 1503 1504TcpExtListenOverflows and TcpExtListenDrops 1505------------------------------------------- 1506On server, run the nc command, listen on port 9000:: 1507 1508 nstatuser@nstat-b:~$ nc -lkv 0.0.0.0 9000 1509 Listening on [0.0.0.0] (family 0, port 9000) 1510 1511On client, run 3 nc commands in different terminals:: 1512 1513 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1514 Connection to nstat-b 9000 port [tcp/*] succeeded! 1515 1516The nc command only accepts 1 connection, and the accept queue length 1517is 1. On current linux implementation, set queue length to n means the 1518actual queue length is n+1. Now we create 3 connections, 1 is accepted 1519by nc, 2 in accepted queue, so the accept queue is full. 1520 1521Before running the 4th nc, we clean the nstat history on the server:: 1522 1523 nstatuser@nstat-b:~$ nstat -n 1524 1525Run the 4th nc on the client:: 1526 1527 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1528 1529If the nc server is running on kernel 4.10 or higher version, you 1530won't see the "Connection to ... succeeded!" string, because kernel 1531will drop the SYN if the accept queue is full. If the nc client is running 1532on an old kernel, you would see that the connection is succeeded, 1533because kernel would complete the 3 way handshake and keep the socket 1534on half open queue. I did the test on kernel 4.15. Below is the nstat 1535on the server:: 1536 1537 nstatuser@nstat-b:~$ nstat 1538 #kernel 1539 IpInReceives 4 0.0 1540 IpInDelivers 4 0.0 1541 TcpInSegs 4 0.0 1542 TcpExtListenOverflows 4 0.0 1543 TcpExtListenDrops 4 0.0 1544 IpExtInOctets 240 0.0 1545 IpExtInNoECTPkts 4 0.0 1546 1547Both TcpExtListenOverflows and TcpExtListenDrops were 4. If the time 1548between the 4th nc and the nstat was longer, the value of 1549TcpExtListenOverflows and TcpExtListenDrops would be larger, because 1550the SYN of the 4th nc was dropped, the client was retrying. 1551 1552IpInAddrErrors, IpExtInNoRoutes and IpOutNoRoutes 1553------------------------------------------------- 1554server A IP address: 192.168.122.250 1555server B IP address: 192.168.122.251 1556Prepare on server A, add a route to server B:: 1557 1558 $ sudo ip route add 8.8.8.8/32 via 192.168.122.251 1559 1560Prepare on server B, disable send_redirects for all interfaces:: 1561 1562 $ sudo sysctl -w net.ipv4.conf.all.send_redirects=0 1563 $ sudo sysctl -w net.ipv4.conf.ens3.send_redirects=0 1564 $ sudo sysctl -w net.ipv4.conf.lo.send_redirects=0 1565 $ sudo sysctl -w net.ipv4.conf.default.send_redirects=0 1566 1567We want to let sever A send a packet to 8.8.8.8, and route the packet 1568to server B. When server B receives such packet, it might send a ICMP 1569Redirect message to server A, set send_redirects to 0 will disable 1570this behavior. 1571 1572First, generate InAddrErrors. On server B, we disable IP forwarding:: 1573 1574 $ sudo sysctl -w net.ipv4.conf.all.forwarding=0 1575 1576On server A, we send packets to 8.8.8.8:: 1577 1578 $ nc -v 8.8.8.8 53 1579 1580On server B, we check the output of nstat:: 1581 1582 $ nstat 1583 #kernel 1584 IpInReceives 3 0.0 1585 IpInAddrErrors 3 0.0 1586 IpExtInOctets 180 0.0 1587 IpExtInNoECTPkts 3 0.0 1588 1589As we have let server A route 8.8.8.8 to server B, and we disabled IP 1590forwarding on server B, Server A sent packets to server B, then server B 1591dropped packets and increased IpInAddrErrors. As the nc command would 1592re-send the SYN packet if it didn't receive a SYN+ACK, we could find 1593multiple IpInAddrErrors. 1594 1595Second, generate IpExtInNoRoutes. On server B, we enable IP 1596forwarding:: 1597 1598 $ sudo sysctl -w net.ipv4.conf.all.forwarding=1 1599 1600Check the route table of server B and remove the default route:: 1601 1602 $ ip route show 1603 default via 192.168.122.1 dev ens3 proto static 1604 192.168.122.0/24 dev ens3 proto kernel scope link src 192.168.122.251 1605 $ sudo ip route delete default via 192.168.122.1 dev ens3 proto static 1606 1607On server A, we contact 8.8.8.8 again:: 1608 1609 $ nc -v 8.8.8.8 53 1610 nc: connect to 8.8.8.8 port 53 (tcp) failed: Network is unreachable 1611 1612On server B, run nstat:: 1613 1614 $ nstat 1615 #kernel 1616 IpInReceives 1 0.0 1617 IpOutRequests 1 0.0 1618 IcmpOutMsgs 1 0.0 1619 IcmpOutDestUnreachs 1 0.0 1620 IcmpMsgOutType3 1 0.0 1621 IpExtInNoRoutes 1 0.0 1622 IpExtInOctets 60 0.0 1623 IpExtOutOctets 88 0.0 1624 IpExtInNoECTPkts 1 0.0 1625 1626We enabled IP forwarding on server B, when server B received a packet 1627which destination IP address is 8.8.8.8, server B will try to forward 1628this packet. We have deleted the default route, there was no route for 16298.8.8.8, so server B increase IpExtInNoRoutes and sent the "ICMP 1630Destination Unreachable" message to server A. 1631 1632Third, generate IpOutNoRoutes. Run ping command on server B:: 1633 1634 $ ping -c 1 8.8.8.8 1635 connect: Network is unreachable 1636 1637Run nstat on server B:: 1638 1639 $ nstat 1640 #kernel 1641 IpOutNoRoutes 1 0.0 1642 1643We have deleted the default route on server B. Server B couldn't find 1644a route for the 8.8.8.8 IP address, so server B increased 1645IpOutNoRoutes. 1646 1647TcpExtTCPACKSkippedSynRecv 1648-------------------------- 1649In this test, we send 3 same SYN packets from client to server. The 1650first SYN will let server create a socket, set it to Syn-Recv status, 1651and reply a SYN/ACK. The second SYN will let server reply the SYN/ACK 1652again, and record the reply time (the duplicate ACK reply time). The 1653third SYN will let server check the previous duplicate ACK reply time, 1654and decide to skip the duplicate ACK, then increase the 1655TcpExtTCPACKSkippedSynRecv counter. 1656 1657Run tcpdump to capture a SYN packet:: 1658 1659 nstatuser@nstat-a:~$ sudo tcpdump -c 1 -w /tmp/syn.pcap port 9000 1660 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 1661 1662Open another terminal, run nc command:: 1663 1664 nstatuser@nstat-a:~$ nc nstat-b 9000 1665 1666As the nstat-b didn't listen on port 9000, it should reply a RST, and 1667the nc command exited immediately. It was enough for the tcpdump 1668command to capture a SYN packet. A linux server might use hardware 1669offload for the TCP checksum, so the checksum in the /tmp/syn.pcap 1670might be not correct. We call tcprewrite to fix it:: 1671 1672 nstatuser@nstat-a:~$ tcprewrite --infile=/tmp/syn.pcap --outfile=/tmp/syn_fixcsum.pcap --fixcsum 1673 1674On nstat-b, we run nc to listen on port 9000:: 1675 1676 nstatuser@nstat-b:~$ nc -lkv 9000 1677 Listening on [0.0.0.0] (family 0, port 9000) 1678 1679On nstat-a, we blocked the packet from port 9000, or nstat-a would send 1680RST to nstat-b:: 1681 1682 nstatuser@nstat-a:~$ sudo iptables -A INPUT -p tcp --sport 9000 -j DROP 1683 1684Send 3 SYN repeatly to nstat-b:: 1685 1686 nstatuser@nstat-a:~$ for i in {1..3}; do sudo tcpreplay -i ens3 /tmp/syn_fixcsum.pcap; done 1687 1688Check snmp counter on nstat-b:: 1689 1690 nstatuser@nstat-b:~$ nstat | grep -i skip 1691 TcpExtTCPACKSkippedSynRecv 1 0.0 1692 1693As we expected, TcpExtTCPACKSkippedSynRecv is 1. 1694 1695TcpExtTCPACKSkippedPAWS 1696----------------------- 1697To trigger PAWS, we could send an old SYN. 1698 1699On nstat-b, let nc listen on port 9000:: 1700 1701 nstatuser@nstat-b:~$ nc -lkv 9000 1702 Listening on [0.0.0.0] (family 0, port 9000) 1703 1704On nstat-a, run tcpdump to capture a SYN:: 1705 1706 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/paws_pre.pcap -c 1 port 9000 1707 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 1708 1709On nstat-a, run nc as a client to connect nstat-b:: 1710 1711 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1712 Connection to nstat-b 9000 port [tcp/*] succeeded! 1713 1714Now the tcpdump has captured the SYN and exit. We should fix the 1715checksum:: 1716 1717 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/paws_pre.pcap --outfile /tmp/paws.pcap --fixcsum 1718 1719Send the SYN packet twice:: 1720 1721 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/paws.pcap; done 1722 1723On nstat-b, check the snmp counter:: 1724 1725 nstatuser@nstat-b:~$ nstat | grep -i skip 1726 TcpExtTCPACKSkippedPAWS 1 0.0 1727 1728We sent two SYN via tcpreplay, both of them would let PAWS check 1729failed, the nstat-b replied an ACK for the first SYN, skipped the ACK 1730for the second SYN, and updated TcpExtTCPACKSkippedPAWS. 1731 1732TcpExtTCPACKSkippedSeq 1733---------------------- 1734To trigger TcpExtTCPACKSkippedSeq, we send packets which have valid 1735timestamp (to pass PAWS check) but the sequence number is out of 1736window. The linux TCP stack would avoid to skip if the packet has 1737data, so we need a pure ACK packet. To generate such a packet, we 1738could create two sockets: one on port 9000, another on port 9001. Then 1739we capture an ACK on port 9001, change the source/destination port 1740numbers to match the port 9000 socket. Then we could trigger 1741TcpExtTCPACKSkippedSeq via this packet. 1742 1743On nstat-b, open two terminals, run two nc commands to listen on both 1744port 9000 and port 9001:: 1745 1746 nstatuser@nstat-b:~$ nc -lkv 9000 1747 Listening on [0.0.0.0] (family 0, port 9000) 1748 1749 nstatuser@nstat-b:~$ nc -lkv 9001 1750 Listening on [0.0.0.0] (family 0, port 9001) 1751 1752On nstat-a, run two nc clients:: 1753 1754 nstatuser@nstat-a:~$ nc -v nstat-b 9000 1755 Connection to nstat-b 9000 port [tcp/*] succeeded! 1756 1757 nstatuser@nstat-a:~$ nc -v nstat-b 9001 1758 Connection to nstat-b 9001 port [tcp/*] succeeded! 1759 1760On nstat-a, run tcpdump to capture an ACK:: 1761 1762 nstatuser@nstat-a:~$ sudo tcpdump -w /tmp/seq_pre.pcap -c 1 dst port 9001 1763 tcpdump: listening on ens3, link-type EN10MB (Ethernet), capture size 262144 bytes 1764 1765On nstat-b, send a packet via the port 9001 socket. E.g. we sent a 1766string 'foo' in our example:: 1767 1768 nstatuser@nstat-b:~$ nc -lkv 9001 1769 Listening on [0.0.0.0] (family 0, port 9001) 1770 Connection from nstat-a 42132 received! 1771 foo 1772 1773On nstat-a, the tcpdump should have captured the ACK. We should check 1774the source port numbers of the two nc clients:: 1775 1776 nstatuser@nstat-a:~$ ss -ta '( dport = :9000 || dport = :9001 )' | tee 1777 State Recv-Q Send-Q Local Address:Port Peer Address:Port 1778 ESTAB 0 0 192.168.122.250:50208 192.168.122.251:9000 1779 ESTAB 0 0 192.168.122.250:42132 192.168.122.251:9001 1780 1781Run tcprewrite, change port 9001 to port 9000, change port 42132 to 1782port 50208:: 1783 1784 nstatuser@nstat-a:~$ tcprewrite --infile /tmp/seq_pre.pcap --outfile /tmp/seq.pcap -r 9001:9000 -r 42132:50208 --fixcsum 1785 1786Now the /tmp/seq.pcap is the packet we need. Send it to nstat-b:: 1787 1788 nstatuser@nstat-a:~$ for i in {1..2}; do sudo tcpreplay -i ens3 /tmp/seq.pcap; done 1789 1790Check TcpExtTCPACKSkippedSeq on nstat-b:: 1791 1792 nstatuser@nstat-b:~$ nstat | grep -i skip 1793 TcpExtTCPACKSkippedSeq 1 0.0