Acceptable performance of cxgbe(4), and TOE almost didn’t crash the system
After upgrading this system to stable/12 r348672, my Chelsio T6225-CR NIC finally provided acceptable performance.
For the past year or so, the performance was at half speed when talking to other 1 Gbit/s systems. Now, the NIC is running at top speed again.
This system is a bit lonely being the only one running at 10 Gbit/s, but another system will have its 10 Gbit/s NIC delivered sometime next week. I’m looking forward to do some iperf3(1) tests between these two hosts.
I even dared to enable the TCP Offload Engine, but sadly the system crashed after a little more than 24 hours of continued service.
kldload t4_tom ifconfig cc0 toe sysctl dev.t6nex.0.toe.ddp=1 sysctl dev.t6nex.0.toe.tx_zcopy=1
The downside of enabling TOE is that iftop(8) from net-mgmt/iftop doesn’t see the traffic being handled by the TOE, limiting the insight to non-TCP packets only.
Here’s the stacktrace of the latest crash:
#0 __curthread () at /usr/src/sys/amd64/include/pcpu.h:234
#1 doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:371
#2 0xffffffff808af4ed in kern_reboot (howto=260)
at /usr/src/sys/kern/kern_shutdown.c:451
#3 0xffffffff808af979 in vpanic (fmt=<optimized out>, ap=<optimized out>)
at /usr/src/sys/kern/kern_shutdown.c:877
#4 0xffffffff808af773 in panic (fmt=<unavailable>)
at /usr/src/sys/kern/kern_shutdown.c:804
#5 0xffffffff80c062b4 in trap_fatal (frame=0xfffffe00004882a0, eva=24)
at /usr/src/sys/amd64/amd64/trap.c:946
#6 0xffffffff80c06319 in trap_pfault (frame=0xfffffe00004882a0, usermode=0)
at /usr/src/sys/amd64/amd64/trap.c:765
#7 0xffffffff80c058ff in trap (frame=0xfffffe00004882a0)
at /usr/src/sys/amd64/amd64/trap.c:441
#8 <signal handler called>
#9 offload_socket (so=0xfffff801a5ea96d0, toep=0x0)
at /usr/src/sys/dev/cxgbe/tom/t4_tom.c:199
#10 0xffffffff82a2bb00 in t4_offload_socket (tod=<optimized out>,
arg=0xfffff80539778f00, so=0xfffff801a5ea96d0)
at /usr/src/sys/dev/cxgbe/tom/t4_listen.c:958
#11 0xffffffff80a925eb in syncache_socket (sc=0xfffff8036f552348,
lso=0xfffff801ad29ea38, m=<optimized out>)
at /usr/src/sys/netinet/tcp_syncache.c:989
#12 0xffffffff80a91bde in syncache_expand (inc=0xfffffe0000488618,
to=0xfffffe0000488590, th=<optimized out>, lsop=<optimized out>,
m=0xfffff8003b5efd00) at /usr/src/sys/netinet/tcp_syncache.c:1235
#13 0xffffffff80a7bbb3 in tcp_input (mp=<optimized out>,
offp=<optimized out>, proto=<optimized out>)
at /usr/src/sys/netinet/tcp_input.c:1086
#14 0xffffffff809f3e83 in ip_input (m=0x0)
at /usr/src/sys/netinet/ip_input.c:828
#15 0xffffffff809cbc3f in netisr_dispatch_src (proto=1,
source=<optimized out>, m=0x1) at /usr/src/sys/net/netisr.c:1122
#16 0xffffffff809bf619 in ether_demux (ifp=0xfffff8000b7e1800, m=0x0)
at /usr/src/sys/net/if_ethersubr.c:879
#17 0xffffffff809c0896 in ether_input_internal (ifp=0xfffff8000b7e1800, m=0x0)
at /usr/src/sys/net/if_ethersubr.c:667
#18 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:697
#19 0xffffffff809cbc3f in netisr_dispatch_src (proto=5,
source=<optimized out>, m=0x1) at /usr/src/sys/net/netisr.c:1122
#20 0xffffffff809bfa2b in ether_input (ifp=0xfffff8000b7e1800, m=0x0)
at /usr/src/sys/net/if_ethersubr.c:787
#21 0xffffffff80a847d8 in tcp_lro_flush (lc=0xfffffe0004506130,
le=0xfffff80015007bf0) at /usr/src/sys/netinet/tcp_lro.c:397
#22 0xffffffff80a8494f in tcp_lro_rx_done (lc=<optimized out>)
at /usr/src/sys/netinet/tcp_lro.c:287
#23 tcp_lro_flush_all (lc=0xfffffe0004506130)
at /usr/src/sys/netinet/tcp_lro.c:535
#24 0xffffffff805dc574 in service_iq_fl (iq=<optimized out>, budget=0)
at /usr/src/sys/dev/cxgbe/t4_sge.c:1763
#25 0xffffffff805dc00d in t4_intr (arg=0xfffffe0004506000)
at /usr/src/sys/dev/cxgbe/t4_sge.c:1432
#26 0xffffffff80872484 in intr_event_execute_handlers (p=<optimized out>,
ie=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1129
#27 ithread_execute_handlers (p=<optimized out>, ie=<optimized out>)
at /usr/src/sys/kern/kern_intr.c:1142
#28 ithread_loop (arg=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1222
#29 0xffffffff8086eee3 in fork_exit (
callout=0xffffffff808722b0 <ithread_loop>, arg=0xfffff8000b7cd500,
frame=0xfffffe0000488ac0) at /usr/src/sys/kern/kern_fork.c:1063
#30 <signal handler called>
I think it’s time to reenable options INVARIANTS and options INVARIANT_SUPPORT in the kernel.
#notsponsored
Update 2019-06-10
Running kgdb -n 1 was fruitless:
root@enterprise:/var/crash>kgdb -n 1
GNU gdb (GDB) 8.3 [GDB v8.3 for FreeBSD]
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/obj/usr/src/amd64.amd64/sys/ENTERPRISE/kernel.full...
Unread portion of the kernel message buffer:
[89041]
[89041]
[89041] Fatal trap 12: page fault while in kernel mode
[89041] cpuid = 1; apic id = 02
[89041] fault virtual address = 0x18
[89041] fault code = supervisor read data, page not present
[89041] instruction pointer = 0x20:0xffffffff82a30654
[89041] stack pointer = 0x28:0xfffffe0000488360
[89041] frame pointer = 0x28:0xfffffe0000488390
[89041] code segment = base 0x0, limit 0xfffff, type 0x1b
[89041] = DPL 0, pres 1, long 1, def32 0, gran 1
[89041] processor eflags = interrupt enabled, resume, IOPL = 0
[89041] current process = 12 (irq266: t6nex0:0a0)
[89041] trap number = 12
[89041] panic: page fault
[89041] cpuid = 1
[89041] time = 1559921538
[89041] KDB: stack backtrace:
[89041] db_trace_self_wrapper() at 0xffffffff8057b6ab = db_trace_self_wrapper+0x2b/frame 0xfffffe0000488010
[89041] vpanic() at 0xffffffff808af91d = vpanic+0x19d/frame 0xfffffe0000488060
[89041] panic() at 0xffffffff808af773 = panic+0x43/frame 0xfffffe00004880c0
[89041] trap_fatal() at 0xffffffff80c062b4 = trap_fatal+0x394/frame 0xfffffe0000488120
[89041] trap_pfault() at 0xffffffff80c06319 = trap_pfault+0x49/frame 0xfffffe0000488180
[89041] trap() at 0xffffffff80c058ff = trap+0x29f/frame 0xfffffe0000488290
[89041] calltrap() at 0xffffffff80be1685 = calltrap+0x8/frame 0xfffffe0000488290
[89041] --- trap 0xc, rip = 0xffffffff82a30654, rsp = 0xfffffe0000488360, rbp = 0xfffffe0000488390 ---
[89041] offload_socket() at 0xffffffff82a30654 = offload_socket+0x14/frame 0xfffffe0000488390
[89041] t4_offload_socket() at 0xffffffff82a2bb00 = t4_offload_socket+0x20/frame 0xfffffe00004883c0
[89041] syncache_socket() at 0xffffffff80a925eb = syncache_socket+0x7ab/frame 0xfffffe0000488450
[89041] syncache_expand() at 0xffffffff80a91bde = syncache_expand+0x9ae/frame 0xfffffe0000488580
[89041] tcp_input() at 0xffffffff80a7bbb3 = tcp_input+0x1143/frame 0xfffffe00004886d0
[89041] ip_input() at 0xffffffff809f3e83 = ip_input+0x143/frame 0xfffffe0000488790
[89041] netisr_dispatch_src() at 0xffffffff809cbc3f = netisr_dispatch_src+0xcf/frame 0xfffffe00004887e0
[89041] ether_demux() at 0xffffffff809bf619 = ether_demux+0x139/frame 0xfffffe0000488810
[89041] ether_nh_input() at 0xffffffff809c0896 = ether_nh_input+0x346/frame 0xfffffe0000488870
[89041] netisr_dispatch_src() at 0xffffffff809cbc3f = netisr_dispatch_src+0xcf/frame 0xfffffe00004888c0
[89041] ether_input() at 0xffffffff809bfa2b = ether_input+0x4b/frame 0xfffffe00004888f0
[89041] tcp_lro_flush() at 0xffffffff80a847d8 = tcp_lro_flush+0x228/frame 0xfffffe0000488910
[89041] tcp_lro_flush_all() at 0xffffffff80a8494f = tcp_lro_flush_all+0x11f/frame 0xfffffe0000488950
[89041] service_iq_fl() at 0xffffffff805dc574 = service_iq_fl+0x554/frame 0xfffffe00004889f0
[89041] t4_intr() at 0xffffffff805dc00d = t4_intr+0x2d/frame 0xfffffe0000488a10
[89041] ithread_loop() at 0xffffffff80872484 = ithread_loop+0x1d4/frame 0xfffffe0000488a70
[89041] fork_exit() at 0xffffffff8086eee3 = fork_exit+0x83/frame 0xfffffe0000488ab0
[89041] fork_trampoline() at 0xffffffff80be267e = fork_trampoline+0xe/frame 0xfffffe0000488ab0
[89041] --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
[89041] Uptime: 1d0h44m1s
[89041] Dumping 10016 out of 32677 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
inferior.c:287: internal-error: struct inferior *find_inferior_pid(int): Assertion `pid != 0' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) n
This is a bug, please report it. For instructions, see:
<http://www.gnu.org/software/gdb/bugs/>.
inferior.c:287: internal-error: struct inferior *find_inferior_pid(int): Assertion `pid != 0' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Create a core file of GDB? (y or n) n
Command aborted.
(kgdb)
Running gdb /boot/kernel/kernel /var/crash/vmcore.last isn’t any better:
root@enterprise:/var/crash>gdb /boot/kernel/kernel vmcore.last
GNU gdb (GDB) 8.3 [GDB v8.3 for FreeBSD]
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...
"/var/crash/vmcore.last" is not a core dump: file format not recognized
(gdb)
On the other hand, running /usr/libexec/kgdb /boot/kernel/kernel /var/crash/vmcore.last does actually take me somewhere:
root@enterprise:~>/usr/libexec/kgdb /boot/kernel/kernel /var/crash/vmcore.last
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
[89041]
[89041]
[89041] Fatal trap 12: page fault while in kernel mode
[89041] cpuid = 1; apic id = 02
[89041] fault virtual address = 0x18
[89041] fault code = supervisor read data, page not present
[89041] instruction pointer = 0x20:0xffffffff82a30654
[89041] stack pointer = 0x28:0xfffffe0000488360
[89041] frame pointer = 0x28:0xfffffe0000488390
[89041] code segment = base 0x0, limit 0xfffff, type 0x1b
[89041] = DPL 0, pres 1, long 1, def32 0, gran 1
[89041] processor eflags = interrupt enabled, resume, IOPL = 0
[89041] current process = 12 (irq266: t6nex0:0a0)
[89041] trap number = 12
[89041] panic: page fault
[89041] cpuid = 1
[89041] time = 1559921538
[89041] KDB: stack backtrace:
[89041] db_trace_self_wrapper() at 0xffffffff8057b6ab = db_trace_self_wrapper+0x2b/frame 0xfffffe0000488010
[89041] vpanic() at 0xffffffff808af91d = vpanic+0x19d/frame 0xfffffe0000488060
[89041] panic() at 0xffffffff808af773 = panic+0x43/frame 0xfffffe00004880c0
[89041] trap_fatal() at 0xffffffff80c062b4 = trap_fatal+0x394/frame 0xfffffe0000488120
[89041] trap_pfault() at 0xffffffff80c06319 = trap_pfault+0x49/frame 0xfffffe0000488180
[89041] trap() at 0xffffffff80c058ff = trap+0x29f/frame 0xfffffe0000488290
[89041] calltrap() at 0xffffffff80be1685 = calltrap+0x8/frame 0xfffffe0000488290
[89041] --- trap 0xc, rip = 0xffffffff82a30654, rsp = 0xfffffe0000488360, rbp = 0xfffffe0000488390 ---
[89041] offload_socket() at 0xffffffff82a30654 = offload_socket+0x14/frame 0xfffffe0000488390
[89041] t4_offload_socket() at 0xffffffff82a2bb00 = t4_offload_socket+0x20/frame 0xfffffe00004883c0
[89041] syncache_socket() at 0xffffffff80a925eb = syncache_socket+0x7ab/frame 0xfffffe0000488450
[89041] syncache_expand() at 0xffffffff80a91bde = syncache_expand+0x9ae/frame 0xfffffe0000488580
[89041] tcp_input() at 0xffffffff80a7bbb3 = tcp_input+0x1143/frame 0xfffffe00004886d0
[89041] ip_input() at 0xffffffff809f3e83 = ip_input+0x143/frame 0xfffffe0000488790
[89041] netisr_dispatch_src() at 0xffffffff809cbc3f = netisr_dispatch_src+0xcf/frame 0xfffffe00004887e0
[89041] ether_demux() at 0xffffffff809bf619 = ether_demux+0x139/frame 0xfffffe0000488810
[89041] ether_nh_input() at 0xffffffff809c0896 = ether_nh_input+0x346/frame 0xfffffe0000488870
[89041] netisr_dispatch_src() at 0xffffffff809cbc3f = netisr_dispatch_src+0xcf/frame 0xfffffe00004888c0
[89041] ether_input() at 0xffffffff809bfa2b = ether_input+0x4b/frame 0xfffffe00004888f0
[89041] tcp_lro_flush() at 0xffffffff80a847d8 = tcp_lro_flush+0x228/frame 0xfffffe0000488910
[89041] tcp_lro_flush_all() at 0xffffffff80a8494f = tcp_lro_flush_all+0x11f/frame 0xfffffe0000488950
[89041] service_iq_fl() at 0xffffffff805dc574 = service_iq_fl+0x554/frame 0xfffffe00004889f0
[89041] t4_intr() at 0xffffffff805dc00d = t4_intr+0x2d/frame 0xfffffe0000488a10
[89041] ithread_loop() at 0xffffffff80872484 = ithread_loop+0x1d4/frame 0xfffffe0000488a70
[89041] fork_exit() at 0xffffffff8086eee3 = fork_exit+0x83/frame 0xfffffe0000488ab0
[89041] fork_trampoline() at 0xffffffff80be267e = fork_trampoline+0xe/frame 0xfffffe0000488ab0
[89041] --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
[89041] Uptime: 1d0h44m1s
[89041] Dumping 10016 out of 32677 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%
Reading symbols from /boot/kernel/dtraceall.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtraceall.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtraceall.ko
Reading symbols from /boot/kernel/profile.ko...Reading symbols from /usr/lib/debug//boot/kernel/profile.ko.debug...done.
done.
Loaded symbols for /boot/kernel/profile.ko
Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtrace.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtrace.ko
Reading symbols from /boot/kernel/systrace_freebsd32.ko...Reading symbols from /usr/lib/debug//boot/kernel/systrace_freebsd32.ko.debug...done.
done.
Loaded symbols for /boot/kernel/systrace_freebsd32.ko
Reading symbols from /boot/kernel/systrace.ko...Reading symbols from /usr/lib/debug//boot/kernel/systrace.ko.debug...done.
done.
Loaded symbols for /boot/kernel/systrace.ko
Reading symbols from /boot/kernel/sdt.ko...Reading symbols from /usr/lib/debug//boot/kernel/sdt.ko.debug...done.
done.
Loaded symbols for /boot/kernel/sdt.ko
Reading symbols from /boot/kernel/fasttrap.ko...Reading symbols from /usr/lib/debug//boot/kernel/fasttrap.ko.debug...done.
done.
Loaded symbols for /boot/kernel/fasttrap.ko
Reading symbols from /boot/kernel/fbt.ko...Reading symbols from /usr/lib/debug//boot/kernel/fbt.ko.debug...done.
done.
Loaded symbols for /boot/kernel/fbt.ko
Reading symbols from /boot/kernel/dtnfscl.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtnfscl.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtnfscl.ko
Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtmalloc.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtmalloc.ko
Reading symbols from /boot/kernel/cc_htcp.ko...Reading symbols from /usr/lib/debug//boot/kernel/cc_htcp.ko.debug...done.
done.
Loaded symbols for /boot/kernel/cc_htcp.ko
Reading symbols from /boot/kernel/t4_tom.ko...Reading symbols from /usr/lib/debug//boot/kernel/t4_tom.ko.debug...done.
done.
Loaded symbols for /boot/kernel/t4_tom.ko
Reading symbols from /boot/kernel/toecore.ko...Reading symbols from /usr/lib/debug//boot/kernel/toecore.ko.debug...done.
done.
Loaded symbols for /boot/kernel/toecore.ko
Reading symbols from /boot/kernel/mac_ntpd.ko...Reading symbols from /usr/lib/debug//boot/kernel/mac_ntpd.ko.debug...done.
done.
Loaded symbols for /boot/kernel/mac_ntpd.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /usr/lib/debug//boot/kernel/accf_http.ko.debug...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/accf_data.ko...Reading symbols from /usr/lib/debug//boot/kernel/accf_data.ko.debug...done.
done.
Loaded symbols for /boot/kernel/accf_data.ko
#0 doadump (textdump=1) at src/sys/amd64/include/pcpu.h:234
234 __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (OFFSETOF_CURTHREAD));
(kgdb) bt
#0 doadump (textdump=1) at src/sys/amd64/include/pcpu.h:234
#1 0xffffffff808af4ed in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:451
#2 0xffffffff808af979 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:877
#3 0xffffffff808af773 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:804
#4 0xffffffff80c062b4 in trap_fatal (frame=0xfffffe00004882a0, eva=24) at /usr/src/sys/amd64/amd64/trap.c:946
#5 0xffffffff80c06319 in trap_pfault (frame=0xfffffe00004882a0, usermode=0) at src/sys/amd64/include/pcpu.h:234
#6 0xffffffff80c058ff in trap (frame=0xfffffe00004882a0) at /usr/src/sys/amd64/amd64/trap.c:441
#7 0xffffffff80be1685 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232
#8 0xffffffff82a30654 in offload_socket (so=0xfffff801a5ea96d0, toep=0x0) at /usr/src/sys/dev/cxgbe/tom/t4_tom.c:198
#9 0xffffffff82a2bb00 in t4_offload_socket (tod=<value optimized out>, arg=0xfffff80539778f00, so=<value optimized out>) at /usr/src/sys/dev/cxgbe/tom/t4_listen.c:958
#10 0xffffffff80a925eb in syncache_socket (sc=0xfffff8036f552348, lso=0xfffff801ad29ea38, m=<value optimized out>) at /usr/src/sys/netinet/tcp_syncache.c:989
#11 0xffffffff80a91bde in syncache_expand (inc=0xfffffe0000488618, to=0xfffffe0000488590, th=<value optimized out>, lsop=<value optimized out>, m=0xfffff8003b5efd00) at /usr/src/sys/netinet/tcp_syncache.c:1235
#12 0xffffffff80a7bbb3 in tcp_input (mp=<value optimized out>, offp=<value optimized out>) at /usr/src/sys/netinet/tcp_input.c:1086
#13 0xffffffff809f3e83 in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:828
#14 0xffffffff809cbc3f in netisr_dispatch_src (proto=1, source=<value optimized out>, m=<value optimized out>) at /usr/src/sys/net/netisr.c:1122
#15 0xffffffff809bf619 in ether_demux (ifp=0xfffff8000b7e1800, m=<value optimized out>) at /usr/src/sys/net/if_ethersubr.c:879
#16 0xffffffff809c0896 in ether_nh_input (m=<value optimized out>) at /usr/src/sys/net/if_ethersubr.c:667
#17 0xffffffff809cbc3f in netisr_dispatch_src (proto=5, source=<value optimized out>, m=<value optimized out>) at /usr/src/sys/net/netisr.c:1122
#18 0xffffffff809bfa2b in ether_input (ifp=0xfffff8000b7e1800, m=0x0) at /usr/src/sys/net/if_ethersubr.c:787
#19 0xffffffff80a847d8 in tcp_lro_flush (lc=0xfffffe0004506130, le=0xfffff80015007bf0) at /usr/src/sys/netinet/tcp_lro.c:397
#20 0xffffffff80a8494f in tcp_lro_flush_all (lc=0xfffffe0004506130) at /usr/src/sys/netinet/tcp_lro.c:287
#21 0xffffffff805dc574 in service_iq_fl (iq=<value optimized out>, budget=0) at /usr/src/sys/dev/cxgbe/t4_sge.c:1763
#22 0xffffffff805dc00d in t4_intr (arg=0xfffffe0004506000) at /usr/src/sys/dev/cxgbe/t4_sge.c:1432
#23 0xffffffff80872484 in ithread_loop (arg=<value optimized out>) at /usr/src/sys/kern/kern_intr.c:1129
#24 0xffffffff8086eee3 in fork_exit (callout=0xffffffff808722b0 <ithread_loop>, arg=0xfffff8000b7cd500, frame=0xfffffe0000488ac0) at /usr/src/sys/kern/kern_fork.c:1063
#25 0xffffffff80be267e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:996
#26 0x0000000000000000 in ?? ()
Current language: auto; currently minimal
(kgdb) up 9
#9 0xffffffff82a2bb00 in t4_offload_socket (tod=<value optimized out>, arg=0xfffff80539778f00, so=<value optimized out>) at /usr/src/sys/dev/cxgbe/tom/t4_listen.c:958
958 offload_socket(so, toep);
(kgdb) list t4_offload_socket
939 return (0);
940 }
941
942 void
943 t4_offload_socket(struct toedev *tod, void *arg, struct socket *so)
944 {
945 struct adapter *sc = tod->tod_softc;
946 struct synq_entry *synqe = arg;
947 #ifdef INVARIANTS
948 struct inpcb *inp = sotoinpcb(so);
(kgdb)
949 #endif
950 struct toepcb *toep = synqe->toep;
951
952 INP_INFO_RLOCK_ASSERT(&V_tcbinfo); /* prevents bad race with accept() */
953 INP_WLOCK_ASSERT(inp);
954 KASSERT(synqe->flags & TPF_SYNQE,
955 ("%s: %p not a synq_entry?", __func__, arg));
956 MPASS(toep->tid == synqe->tid);
957
958 offload_socket(so, toep);
(kgdb)
959 make_established(toep, synqe->iss, synqe->irs, synqe->tcp_opt);
960 toep->flags |= TPF_CPL_PENDING;
961 update_tid(sc, synqe->tid, toep);
962 synqe->flags |= TPF_SYNQE_EXPANDED;
963 }
964
965 static inline void
966 save_qids_in_synqe(struct synq_entry *synqe, struct vi_info *vi,
967 struct offload_settings *s)
968 {
(kgdb) up
#10 0xffffffff80a925eb in syncache_socket (sc=0xfffff8036f552348, lso=0xfffff801ad29ea38, m=<value optimized out>) at /usr/src/sys/netinet/tcp_syncache.c:989
989 tod->tod_offload_socket(tod, sc->sc_todctx, so);
(kgdb) list syncache_socket
702 *
703 * On success return the newly created socket with its underlying inp locked.
704 */
705 static struct socket *
706 syncache_socket(struct syncache *sc, struct socket *lso, struct mbuf *m)
707 {
708 struct tcp_function_block *blk;
709 struct inpcb *inp = NULL;
710 struct socket *so;
711 struct tcpcb *tp;
(kgdb) up
#11 0xffffffff80a91bde in syncache_expand (inc=0xfffffe0000488618, to=0xfffffe0000488590, th=<value optimized out>, lsop=<value optimized out>, m=0xfffff8003b5efd00) at /usr/src/sys/netinet/tcp_syncache.c:1235
1235 *lsop = syncache_socket(sc, *lsop, m);
(kgdb) list syncache_expand
1020 * has its underlying inp locked.
1021 */
1022 int
1023 syncache_expand(struct in_conninfo *inc, struct tcpopt *to, struct tcphdr *th,
1024 struct socket **lsop, struct mbuf *m)
1025 {
1026 struct syncache *sc;
1027 struct syncache_head *sch;
1028 struct syncache scs;
1029 char *s;
(kgdb)
1030
1031 /*
1032 * Global TCP locks are held because we manipulate the PCB lists
1033 * and create a new socket.
1034 */
1035 INP_INFO_RLOCK_ASSERT(&V_tcbinfo);
1036 KASSERT((th->th_flags & (TH_RST|TH_ACK|TH_SYN)) == TH_ACK,
1037 ("%s: can handle only ACK", __func__));
1038
1039 sc = syncache_lookup(inc, &sch); /* returns locked sch */
(kgdb)
1040 SCH_LOCK_ASSERT(sch);
1041
1042 #ifdef INVARIANTS
1043 /*
1044 * Test code for syncookies comparing the syncache stored
1045 * values with the reconstructed values from the cookie.
1046 */
1047 if (sc != NULL)
1048 syncookie_cmp(inc, sch, sc, th, to, *lsop);
1049 #endif
(kgdb) print *sc
$1 = {sc_hash = {tqe_next = 0x0, tqe_prev = 0xfffffe00a2f07f00}, sc_inc = {inc_flags = 0 '\0', inc_len = 0 '\0', inc_fibnum = 0, inc_ie = {ie_fport = 49107, ie_lport = 20480, ie_dependfaddr = {id46_addr = {ia46_pad32 = 0xfffff8036f552360, ia46_addr4 = {
s_addr = 2162321455}}, id6_addr = {__u6_addr = {__u6_addr8 = 0xfffff8036f552360 "", __u6_addr16 = 0xfffff8036f552360, __u6_addr32 = 0xfffff8036f552360}}}, ie_dependladdr = {id46_addr = {ia46_pad32 = 0xfffff8036f552370, ia46_addr4 = {s_addr = 145631104}},
id6_addr = {__u6_addr = {__u6_addr8 = 0xfffff8036f552370 "", __u6_addr16 = 0xfffff8036f552370, __u6_addr32 = 0xfffff8036f552370}}}, ie6_zoneid = 0}}, sc_rxttime = -2059042049, sc_rxmits = 1, sc_tsreflect = 0, sc_tsoff = 0, sc_flowlabel = 0, sc_irs = 1571556941,
sc_iss = 1893493902, sc_ipopts = 0x0, sc_peer_mss = 1460, sc_wnd = 65535, sc_ip_ttl = 64 '@', sc_ip_tos = 0 '\0', sc_requested_s_scale = 8 '\b', sc_requested_r_scale = 9 '\t', sc_flags = 130, sc_tod = 0xfffff80072ab6600, sc_todctx = 0xfffff80539778f00, sc_label = 0x0,
sc_cred = 0xfffff8007938ed00, sc_tfo_cookie = 0x0, sc_pspare = 0x0, sc_spare = 0xfffff8036f5523e8}
(kgdb) print sc->sc_todctx
$2 = (void *) 0xfffff80539778f00
(kgdb) print *(struct synq_entry *)sc->sc_todctx
$3 = {lctx = 0x0, syn = 0x0, flags = 0, ok_to_respond = 0, refcnt = 1111638594, tid = 1111638594, iss = 1111638594, irs = 1111638594, ts = 0, txqid = 2, rxqid = 1, l2e_idx = 0, ulp_mode = 5, rcv_bufsize = 64, tcp_opt = 32778, toep = 0x0}
The cxgbe(4) code isn’t defensive enough to fend off any invalid addresses it receives. The real issue might be in the overall syncache handling.