After upgrading this system to stable/12 r348672, my Chelsio T6225-CR NIC finally provided acceptable performance.

For the past year or so, the performance was at half speed when talking to other 1 Gbit/s systems. Now, the NIC is running at top speed again.

This system is a bit lonely being the only one running at 10 Gbit/s, but another system will have its 10 Gbit/s NIC delivered sometime next week. I’m looking forward to do some iperf3(1) tests between these two hosts.

I even dared to enable the TCP Offload Engine, but sadly the system crashed after a little more than 24 hours of continued service.

kldload t4_tom
ifconfig cc0 toe
sysctl dev.t6nex.0.toe.ddp=1
sysctl dev.t6nex.0.toe.tx_zcopy=1

The downside of enabling TOE is that iftop(8) from net-mgmt/iftop doesn’t see the traffic being handled by the TOE, limiting the insight to non-TCP packets only.

Here’s the stacktrace of the latest crash:

#0  __curthread () at /usr/src/sys/amd64/include/pcpu.h:234
#1  doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:371
#2  0xffffffff808af4ed in kern_reboot (howto=260)
    at /usr/src/sys/kern/kern_shutdown.c:451
#3  0xffffffff808af979 in vpanic (fmt=<optimized out>, ap=<optimized out>)
    at /usr/src/sys/kern/kern_shutdown.c:877
#4  0xffffffff808af773 in panic (fmt=<unavailable>)
    at /usr/src/sys/kern/kern_shutdown.c:804
#5  0xffffffff80c062b4 in trap_fatal (frame=0xfffffe00004882a0, eva=24)
    at /usr/src/sys/amd64/amd64/trap.c:946
#6  0xffffffff80c06319 in trap_pfault (frame=0xfffffe00004882a0, usermode=0)
    at /usr/src/sys/amd64/amd64/trap.c:765
#7  0xffffffff80c058ff in trap (frame=0xfffffe00004882a0)
    at /usr/src/sys/amd64/amd64/trap.c:441
#8  <signal handler called>
#9  offload_socket (so=0xfffff801a5ea96d0, toep=0x0)
    at /usr/src/sys/dev/cxgbe/tom/t4_tom.c:199
#10 0xffffffff82a2bb00 in t4_offload_socket (tod=<optimized out>,
    arg=0xfffff80539778f00, so=0xfffff801a5ea96d0)
    at /usr/src/sys/dev/cxgbe/tom/t4_listen.c:958
#11 0xffffffff80a925eb in syncache_socket (sc=0xfffff8036f552348,
    lso=0xfffff801ad29ea38, m=<optimized out>)
    at /usr/src/sys/netinet/tcp_syncache.c:989
#12 0xffffffff80a91bde in syncache_expand (inc=0xfffffe0000488618,
    to=0xfffffe0000488590, th=<optimized out>, lsop=<optimized out>,
    m=0xfffff8003b5efd00) at /usr/src/sys/netinet/tcp_syncache.c:1235
#13 0xffffffff80a7bbb3 in tcp_input (mp=<optimized out>,
    offp=<optimized out>, proto=<optimized out>)
    at /usr/src/sys/netinet/tcp_input.c:1086
#14 0xffffffff809f3e83 in ip_input (m=0x0)
    at /usr/src/sys/netinet/ip_input.c:828
#15 0xffffffff809cbc3f in netisr_dispatch_src (proto=1,
    source=<optimized out>, m=0x1) at /usr/src/sys/net/netisr.c:1122
#16 0xffffffff809bf619 in ether_demux (ifp=0xfffff8000b7e1800, m=0x0)
    at /usr/src/sys/net/if_ethersubr.c:879
#17 0xffffffff809c0896 in ether_input_internal (ifp=0xfffff8000b7e1800, m=0x0)
    at /usr/src/sys/net/if_ethersubr.c:667
#18 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:697
#19 0xffffffff809cbc3f in netisr_dispatch_src (proto=5,
    source=<optimized out>, m=0x1) at /usr/src/sys/net/netisr.c:1122
#20 0xffffffff809bfa2b in ether_input (ifp=0xfffff8000b7e1800, m=0x0)
    at /usr/src/sys/net/if_ethersubr.c:787
#21 0xffffffff80a847d8 in tcp_lro_flush (lc=0xfffffe0004506130,
    le=0xfffff80015007bf0) at /usr/src/sys/netinet/tcp_lro.c:397
#22 0xffffffff80a8494f in tcp_lro_rx_done (lc=<optimized out>)
    at /usr/src/sys/netinet/tcp_lro.c:287
#23 tcp_lro_flush_all (lc=0xfffffe0004506130)
    at /usr/src/sys/netinet/tcp_lro.c:535
#24 0xffffffff805dc574 in service_iq_fl (iq=<optimized out>, budget=0)
    at /usr/src/sys/dev/cxgbe/t4_sge.c:1763
#25 0xffffffff805dc00d in t4_intr (arg=0xfffffe0004506000)
    at /usr/src/sys/dev/cxgbe/t4_sge.c:1432
#26 0xffffffff80872484 in intr_event_execute_handlers (p=<optimized out>,
    ie=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1129
#27 ithread_execute_handlers (p=<optimized out>, ie=<optimized out>)
    at /usr/src/sys/kern/kern_intr.c:1142
#28 ithread_loop (arg=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1222
#29 0xffffffff8086eee3 in fork_exit (
    callout=0xffffffff808722b0 <ithread_loop>, arg=0xfffff8000b7cd500,
    frame=0xfffffe0000488ac0) at /usr/src/sys/kern/kern_fork.c:1063
#30 <signal handler called>

I think it’s time to reenable options INVARIANTS and options INVARIANT_SUPPORT in the kernel.

#notsponsored


Update 2019-06-10

Running kgdb -n 1 was fruitless:

root@enterprise:/var/crash>kgdb -n 1
GNU gdb (GDB) 8.3 [GDB v8.3 for FreeBSD]
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/obj/usr/src/amd64.amd64/sys/ENTERPRISE/kernel.full...

Unread portion of the kernel message buffer:
[89041]
[89041]
[89041] Fatal trap 12: page fault while in kernel mode
[89041] cpuid = 1; apic id = 02
[89041] fault virtual address   = 0x18
[89041] fault code              = supervisor read data, page not present
[89041] instruction pointer     = 0x20:0xffffffff82a30654
[89041] stack pointer           = 0x28:0xfffffe0000488360
[89041] frame pointer           = 0x28:0xfffffe0000488390
[89041] code segment            = base 0x0, limit 0xfffff, type 0x1b
[89041]                         = DPL 0, pres 1, long 1, def32 0, gran 1
[89041] processor eflags        = interrupt enabled, resume, IOPL = 0
[89041] current process         = 12 (irq266: t6nex0:0a0)
[89041] trap number             = 12
[89041] panic: page fault
[89041] cpuid = 1
[89041] time = 1559921538
[89041] KDB: stack backtrace:
[89041] db_trace_self_wrapper() at 0xffffffff8057b6ab = db_trace_self_wrapper+0x2b/frame 0xfffffe0000488010
[89041] vpanic() at 0xffffffff808af91d = vpanic+0x19d/frame 0xfffffe0000488060
[89041] panic() at 0xffffffff808af773 = panic+0x43/frame 0xfffffe00004880c0
[89041] trap_fatal() at 0xffffffff80c062b4 = trap_fatal+0x394/frame 0xfffffe0000488120
[89041] trap_pfault() at 0xffffffff80c06319 = trap_pfault+0x49/frame 0xfffffe0000488180
[89041] trap() at 0xffffffff80c058ff = trap+0x29f/frame 0xfffffe0000488290
[89041] calltrap() at 0xffffffff80be1685 = calltrap+0x8/frame 0xfffffe0000488290
[89041] --- trap 0xc, rip = 0xffffffff82a30654, rsp = 0xfffffe0000488360, rbp = 0xfffffe0000488390 ---
[89041] offload_socket() at 0xffffffff82a30654 = offload_socket+0x14/frame 0xfffffe0000488390
[89041] t4_offload_socket() at 0xffffffff82a2bb00 = t4_offload_socket+0x20/frame 0xfffffe00004883c0
[89041] syncache_socket() at 0xffffffff80a925eb = syncache_socket+0x7ab/frame 0xfffffe0000488450
[89041] syncache_expand() at 0xffffffff80a91bde = syncache_expand+0x9ae/frame 0xfffffe0000488580
[89041] tcp_input() at 0xffffffff80a7bbb3 = tcp_input+0x1143/frame 0xfffffe00004886d0
[89041] ip_input() at 0xffffffff809f3e83 = ip_input+0x143/frame 0xfffffe0000488790
[89041] netisr_dispatch_src() at 0xffffffff809cbc3f = netisr_dispatch_src+0xcf/frame 0xfffffe00004887e0
[89041] ether_demux() at 0xffffffff809bf619 = ether_demux+0x139/frame 0xfffffe0000488810
[89041] ether_nh_input() at 0xffffffff809c0896 = ether_nh_input+0x346/frame 0xfffffe0000488870
[89041] netisr_dispatch_src() at 0xffffffff809cbc3f = netisr_dispatch_src+0xcf/frame 0xfffffe00004888c0
[89041] ether_input() at 0xffffffff809bfa2b = ether_input+0x4b/frame 0xfffffe00004888f0
[89041] tcp_lro_flush() at 0xffffffff80a847d8 = tcp_lro_flush+0x228/frame 0xfffffe0000488910
[89041] tcp_lro_flush_all() at 0xffffffff80a8494f = tcp_lro_flush_all+0x11f/frame 0xfffffe0000488950
[89041] service_iq_fl() at 0xffffffff805dc574 = service_iq_fl+0x554/frame 0xfffffe00004889f0
[89041] t4_intr() at 0xffffffff805dc00d = t4_intr+0x2d/frame 0xfffffe0000488a10
[89041] ithread_loop() at 0xffffffff80872484 = ithread_loop+0x1d4/frame 0xfffffe0000488a70
[89041] fork_exit() at 0xffffffff8086eee3 = fork_exit+0x83/frame 0xfffffe0000488ab0
[89041] fork_trampoline() at 0xffffffff80be267e = fork_trampoline+0xe/frame 0xfffffe0000488ab0
[89041] --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
[89041] Uptime: 1d0h44m1s
[89041] Dumping 10016 out of 32677 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

inferior.c:287: internal-error: struct inferior *find_inferior_pid(int): Assertion `pid != 0' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) n

This is a bug, please report it.  For instructions, see:
<http://www.gnu.org/software/gdb/bugs/>.

inferior.c:287: internal-error: struct inferior *find_inferior_pid(int): Assertion `pid != 0' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Create a core file of GDB? (y or n) n
Command aborted.
(kgdb)

Running gdb /boot/kernel/kernel /var/crash/vmcore.last isn’t any better:

root@enterprise:/var/crash>gdb /boot/kernel/kernel vmcore.last
GNU gdb (GDB) 8.3 [GDB v8.3 for FreeBSD]
Copyright (C) 2019 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-portbld-freebsd12.0".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
    <http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /boot/kernel/kernel...
Reading symbols from /usr/lib/debug//boot/kernel/kernel.debug...
"/var/crash/vmcore.last" is not a core dump: file format not recognized
(gdb)

On the other hand, running /usr/libexec/kgdb /boot/kernel/kernel /var/crash/vmcore.last does actually take me somewhere:

root@enterprise:~>/usr/libexec/kgdb /boot/kernel/kernel /var/crash/vmcore.last
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "amd64-marcel-freebsd"...

Unread portion of the kernel message buffer:
[89041]
[89041]
[89041] Fatal trap 12: page fault while in kernel mode
[89041] cpuid = 1; apic id = 02
[89041] fault virtual address   = 0x18
[89041] fault code              = supervisor read data, page not present
[89041] instruction pointer     = 0x20:0xffffffff82a30654
[89041] stack pointer           = 0x28:0xfffffe0000488360
[89041] frame pointer           = 0x28:0xfffffe0000488390
[89041] code segment            = base 0x0, limit 0xfffff, type 0x1b
[89041]                         = DPL 0, pres 1, long 1, def32 0, gran 1
[89041] processor eflags        = interrupt enabled, resume, IOPL = 0
[89041] current process         = 12 (irq266: t6nex0:0a0)
[89041] trap number             = 12
[89041] panic: page fault
[89041] cpuid = 1
[89041] time = 1559921538
[89041] KDB: stack backtrace:
[89041] db_trace_self_wrapper() at 0xffffffff8057b6ab = db_trace_self_wrapper+0x2b/frame 0xfffffe0000488010
[89041] vpanic() at 0xffffffff808af91d = vpanic+0x19d/frame 0xfffffe0000488060
[89041] panic() at 0xffffffff808af773 = panic+0x43/frame 0xfffffe00004880c0
[89041] trap_fatal() at 0xffffffff80c062b4 = trap_fatal+0x394/frame 0xfffffe0000488120
[89041] trap_pfault() at 0xffffffff80c06319 = trap_pfault+0x49/frame 0xfffffe0000488180
[89041] trap() at 0xffffffff80c058ff = trap+0x29f/frame 0xfffffe0000488290
[89041] calltrap() at 0xffffffff80be1685 = calltrap+0x8/frame 0xfffffe0000488290
[89041] --- trap 0xc, rip = 0xffffffff82a30654, rsp = 0xfffffe0000488360, rbp = 0xfffffe0000488390 ---
[89041] offload_socket() at 0xffffffff82a30654 = offload_socket+0x14/frame 0xfffffe0000488390
[89041] t4_offload_socket() at 0xffffffff82a2bb00 = t4_offload_socket+0x20/frame 0xfffffe00004883c0
[89041] syncache_socket() at 0xffffffff80a925eb = syncache_socket+0x7ab/frame 0xfffffe0000488450
[89041] syncache_expand() at 0xffffffff80a91bde = syncache_expand+0x9ae/frame 0xfffffe0000488580
[89041] tcp_input() at 0xffffffff80a7bbb3 = tcp_input+0x1143/frame 0xfffffe00004886d0
[89041] ip_input() at 0xffffffff809f3e83 = ip_input+0x143/frame 0xfffffe0000488790
[89041] netisr_dispatch_src() at 0xffffffff809cbc3f = netisr_dispatch_src+0xcf/frame 0xfffffe00004887e0
[89041] ether_demux() at 0xffffffff809bf619 = ether_demux+0x139/frame 0xfffffe0000488810
[89041] ether_nh_input() at 0xffffffff809c0896 = ether_nh_input+0x346/frame 0xfffffe0000488870
[89041] netisr_dispatch_src() at 0xffffffff809cbc3f = netisr_dispatch_src+0xcf/frame 0xfffffe00004888c0
[89041] ether_input() at 0xffffffff809bfa2b = ether_input+0x4b/frame 0xfffffe00004888f0
[89041] tcp_lro_flush() at 0xffffffff80a847d8 = tcp_lro_flush+0x228/frame 0xfffffe0000488910
[89041] tcp_lro_flush_all() at 0xffffffff80a8494f = tcp_lro_flush_all+0x11f/frame 0xfffffe0000488950
[89041] service_iq_fl() at 0xffffffff805dc574 = service_iq_fl+0x554/frame 0xfffffe00004889f0
[89041] t4_intr() at 0xffffffff805dc00d = t4_intr+0x2d/frame 0xfffffe0000488a10
[89041] ithread_loop() at 0xffffffff80872484 = ithread_loop+0x1d4/frame 0xfffffe0000488a70
[89041] fork_exit() at 0xffffffff8086eee3 = fork_exit+0x83/frame 0xfffffe0000488ab0
[89041] fork_trampoline() at 0xffffffff80be267e = fork_trampoline+0xe/frame 0xfffffe0000488ab0
[89041] --- trap 0, rip = 0, rsp = 0, rbp = 0 ---
[89041] Uptime: 1d0h44m1s
[89041] Dumping 10016 out of 32677 MB:..1%..11%..21%..31%..41%..51%..61%..71%..81%..91%

Reading symbols from /boot/kernel/dtraceall.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtraceall.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtraceall.ko
Reading symbols from /boot/kernel/profile.ko...Reading symbols from /usr/lib/debug//boot/kernel/profile.ko.debug...done.
done.
Loaded symbols for /boot/kernel/profile.ko
Reading symbols from /boot/kernel/dtrace.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtrace.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtrace.ko
Reading symbols from /boot/kernel/systrace_freebsd32.ko...Reading symbols from /usr/lib/debug//boot/kernel/systrace_freebsd32.ko.debug...done.
done.
Loaded symbols for /boot/kernel/systrace_freebsd32.ko
Reading symbols from /boot/kernel/systrace.ko...Reading symbols from /usr/lib/debug//boot/kernel/systrace.ko.debug...done.
done.
Loaded symbols for /boot/kernel/systrace.ko
Reading symbols from /boot/kernel/sdt.ko...Reading symbols from /usr/lib/debug//boot/kernel/sdt.ko.debug...done.
done.
Loaded symbols for /boot/kernel/sdt.ko
Reading symbols from /boot/kernel/fasttrap.ko...Reading symbols from /usr/lib/debug//boot/kernel/fasttrap.ko.debug...done.
done.
Loaded symbols for /boot/kernel/fasttrap.ko
Reading symbols from /boot/kernel/fbt.ko...Reading symbols from /usr/lib/debug//boot/kernel/fbt.ko.debug...done.
done.
Loaded symbols for /boot/kernel/fbt.ko
Reading symbols from /boot/kernel/dtnfscl.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtnfscl.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtnfscl.ko
Reading symbols from /boot/kernel/dtmalloc.ko...Reading symbols from /usr/lib/debug//boot/kernel/dtmalloc.ko.debug...done.
done.
Loaded symbols for /boot/kernel/dtmalloc.ko
Reading symbols from /boot/kernel/cc_htcp.ko...Reading symbols from /usr/lib/debug//boot/kernel/cc_htcp.ko.debug...done.
done.
Loaded symbols for /boot/kernel/cc_htcp.ko
Reading symbols from /boot/kernel/t4_tom.ko...Reading symbols from /usr/lib/debug//boot/kernel/t4_tom.ko.debug...done.
done.
Loaded symbols for /boot/kernel/t4_tom.ko
Reading symbols from /boot/kernel/toecore.ko...Reading symbols from /usr/lib/debug//boot/kernel/toecore.ko.debug...done.
done.
Loaded symbols for /boot/kernel/toecore.ko
Reading symbols from /boot/kernel/mac_ntpd.ko...Reading symbols from /usr/lib/debug//boot/kernel/mac_ntpd.ko.debug...done.
done.
Loaded symbols for /boot/kernel/mac_ntpd.ko
Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /usr/lib/debug//boot/kernel/accf_http.ko.debug...done.
done.
Loaded symbols for /boot/kernel/accf_http.ko
Reading symbols from /boot/kernel/accf_data.ko...Reading symbols from /usr/lib/debug//boot/kernel/accf_data.ko.debug...done.
done.
Loaded symbols for /boot/kernel/accf_data.ko
#0  doadump (textdump=1) at src/sys/amd64/include/pcpu.h:234
234             __asm("movq %%gs:%P1,%0" : "=r" (td) : "n" (OFFSETOF_CURTHREAD));
(kgdb) bt
#0  doadump (textdump=1) at src/sys/amd64/include/pcpu.h:234
#1  0xffffffff808af4ed in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:451
#2  0xffffffff808af979 in vpanic (fmt=<value optimized out>, ap=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:877
#3  0xffffffff808af773 in panic (fmt=<value optimized out>) at /usr/src/sys/kern/kern_shutdown.c:804
#4  0xffffffff80c062b4 in trap_fatal (frame=0xfffffe00004882a0, eva=24) at /usr/src/sys/amd64/amd64/trap.c:946
#5  0xffffffff80c06319 in trap_pfault (frame=0xfffffe00004882a0, usermode=0) at src/sys/amd64/include/pcpu.h:234
#6  0xffffffff80c058ff in trap (frame=0xfffffe00004882a0) at /usr/src/sys/amd64/amd64/trap.c:441
#7  0xffffffff80be1685 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:232
#8  0xffffffff82a30654 in offload_socket (so=0xfffff801a5ea96d0, toep=0x0) at /usr/src/sys/dev/cxgbe/tom/t4_tom.c:198
#9  0xffffffff82a2bb00 in t4_offload_socket (tod=<value optimized out>, arg=0xfffff80539778f00, so=<value optimized out>) at /usr/src/sys/dev/cxgbe/tom/t4_listen.c:958
#10 0xffffffff80a925eb in syncache_socket (sc=0xfffff8036f552348, lso=0xfffff801ad29ea38, m=<value optimized out>) at /usr/src/sys/netinet/tcp_syncache.c:989
#11 0xffffffff80a91bde in syncache_expand (inc=0xfffffe0000488618, to=0xfffffe0000488590, th=<value optimized out>, lsop=<value optimized out>, m=0xfffff8003b5efd00) at /usr/src/sys/netinet/tcp_syncache.c:1235
#12 0xffffffff80a7bbb3 in tcp_input (mp=<value optimized out>, offp=<value optimized out>) at /usr/src/sys/netinet/tcp_input.c:1086
#13 0xffffffff809f3e83 in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:828
#14 0xffffffff809cbc3f in netisr_dispatch_src (proto=1, source=<value optimized out>, m=<value optimized out>) at /usr/src/sys/net/netisr.c:1122
#15 0xffffffff809bf619 in ether_demux (ifp=0xfffff8000b7e1800, m=<value optimized out>) at /usr/src/sys/net/if_ethersubr.c:879
#16 0xffffffff809c0896 in ether_nh_input (m=<value optimized out>) at /usr/src/sys/net/if_ethersubr.c:667
#17 0xffffffff809cbc3f in netisr_dispatch_src (proto=5, source=<value optimized out>, m=<value optimized out>) at /usr/src/sys/net/netisr.c:1122
#18 0xffffffff809bfa2b in ether_input (ifp=0xfffff8000b7e1800, m=0x0) at /usr/src/sys/net/if_ethersubr.c:787
#19 0xffffffff80a847d8 in tcp_lro_flush (lc=0xfffffe0004506130, le=0xfffff80015007bf0) at /usr/src/sys/netinet/tcp_lro.c:397
#20 0xffffffff80a8494f in tcp_lro_flush_all (lc=0xfffffe0004506130) at /usr/src/sys/netinet/tcp_lro.c:287
#21 0xffffffff805dc574 in service_iq_fl (iq=<value optimized out>, budget=0) at /usr/src/sys/dev/cxgbe/t4_sge.c:1763
#22 0xffffffff805dc00d in t4_intr (arg=0xfffffe0004506000) at /usr/src/sys/dev/cxgbe/t4_sge.c:1432
#23 0xffffffff80872484 in ithread_loop (arg=<value optimized out>) at /usr/src/sys/kern/kern_intr.c:1129
#24 0xffffffff8086eee3 in fork_exit (callout=0xffffffff808722b0 <ithread_loop>, arg=0xfffff8000b7cd500, frame=0xfffffe0000488ac0) at /usr/src/sys/kern/kern_fork.c:1063
#25 0xffffffff80be267e in fork_trampoline () at /usr/src/sys/amd64/amd64/exception.S:996
#26 0x0000000000000000 in ?? ()
Current language:  auto; currently minimal
(kgdb) up 9
#9  0xffffffff82a2bb00 in t4_offload_socket (tod=<value optimized out>, arg=0xfffff80539778f00, so=<value optimized out>) at /usr/src/sys/dev/cxgbe/tom/t4_listen.c:958
958             offload_socket(so, toep);
(kgdb) list t4_offload_socket
939             return (0);
940     }
941
942     void
943     t4_offload_socket(struct toedev *tod, void *arg, struct socket *so)
944     {
945             struct adapter *sc = tod->tod_softc;
946             struct synq_entry *synqe = arg;
947     #ifdef INVARIANTS
948             struct inpcb *inp = sotoinpcb(so);
(kgdb)
949     #endif
950             struct toepcb *toep = synqe->toep;
951
952             INP_INFO_RLOCK_ASSERT(&V_tcbinfo); /* prevents bad race with accept() */
953             INP_WLOCK_ASSERT(inp);
954             KASSERT(synqe->flags & TPF_SYNQE,
955                 ("%s: %p not a synq_entry?", __func__, arg));
956             MPASS(toep->tid == synqe->tid);
957
958             offload_socket(so, toep);
(kgdb)
959             make_established(toep, synqe->iss, synqe->irs, synqe->tcp_opt);
960             toep->flags |= TPF_CPL_PENDING;
961             update_tid(sc, synqe->tid, toep);
962             synqe->flags |= TPF_SYNQE_EXPANDED;
963     }
964
965     static inline void
966     save_qids_in_synqe(struct synq_entry *synqe, struct vi_info *vi,
967         struct offload_settings *s)
968     {
(kgdb) up
#10 0xffffffff80a925eb in syncache_socket (sc=0xfffff8036f552348, lso=0xfffff801ad29ea38, m=<value optimized out>) at /usr/src/sys/netinet/tcp_syncache.c:989
989                     tod->tod_offload_socket(tod, sc->sc_todctx, so);
(kgdb) list syncache_socket
702      *
703      * On success return the newly created socket with its underlying inp locked.
704      */
705     static struct socket *
706     syncache_socket(struct syncache *sc, struct socket *lso, struct mbuf *m)
707     {
708             struct tcp_function_block *blk;
709             struct inpcb *inp = NULL;
710             struct socket *so;
711             struct tcpcb *tp;
(kgdb) up
#11 0xffffffff80a91bde in syncache_expand (inc=0xfffffe0000488618, to=0xfffffe0000488590, th=<value optimized out>, lsop=<value optimized out>, m=0xfffff8003b5efd00) at /usr/src/sys/netinet/tcp_syncache.c:1235
1235            *lsop = syncache_socket(sc, *lsop, m);
(kgdb) list syncache_expand
1020     * has its underlying inp locked.
1021     */
1022    int
1023    syncache_expand(struct in_conninfo *inc, struct tcpopt *to, struct tcphdr *th,
1024        struct socket **lsop, struct mbuf *m)
1025    {
1026            struct syncache *sc;
1027            struct syncache_head *sch;
1028            struct syncache scs;
1029            char *s;
(kgdb)
1030
1031            /*
1032             * Global TCP locks are held because we manipulate the PCB lists
1033             * and create a new socket.
1034             */
1035            INP_INFO_RLOCK_ASSERT(&V_tcbinfo);
1036            KASSERT((th->th_flags & (TH_RST|TH_ACK|TH_SYN)) == TH_ACK,
1037                ("%s: can handle only ACK", __func__));
1038
1039            sc = syncache_lookup(inc, &sch);        /* returns locked sch */
(kgdb)
1040            SCH_LOCK_ASSERT(sch);
1041
1042    #ifdef INVARIANTS
1043            /*
1044             * Test code for syncookies comparing the syncache stored
1045             * values with the reconstructed values from the cookie.
1046             */
1047            if (sc != NULL)
1048                    syncookie_cmp(inc, sch, sc, th, to, *lsop);
1049    #endif
(kgdb) print *sc
$1 = {sc_hash = {tqe_next = 0x0, tqe_prev = 0xfffffe00a2f07f00}, sc_inc = {inc_flags = 0 '\0', inc_len = 0 '\0', inc_fibnum = 0, inc_ie = {ie_fport = 49107, ie_lport = 20480, ie_dependfaddr = {id46_addr = {ia46_pad32 = 0xfffff8036f552360, ia46_addr4 = {
            s_addr = 2162321455}}, id6_addr = {__u6_addr = {__u6_addr8 = 0xfffff8036f552360 "", __u6_addr16 = 0xfffff8036f552360, __u6_addr32 = 0xfffff8036f552360}}}, ie_dependladdr = {id46_addr = {ia46_pad32 = 0xfffff8036f552370, ia46_addr4 = {s_addr = 145631104}},
        id6_addr = {__u6_addr = {__u6_addr8 = 0xfffff8036f552370 "", __u6_addr16 = 0xfffff8036f552370, __u6_addr32 = 0xfffff8036f552370}}}, ie6_zoneid = 0}}, sc_rxttime = -2059042049, sc_rxmits = 1, sc_tsreflect = 0, sc_tsoff = 0, sc_flowlabel = 0, sc_irs = 1571556941,
  sc_iss = 1893493902, sc_ipopts = 0x0, sc_peer_mss = 1460, sc_wnd = 65535, sc_ip_ttl = 64 '@', sc_ip_tos = 0 '\0', sc_requested_s_scale = 8 '\b', sc_requested_r_scale = 9 '\t', sc_flags = 130, sc_tod = 0xfffff80072ab6600, sc_todctx = 0xfffff80539778f00, sc_label = 0x0,
  sc_cred = 0xfffff8007938ed00, sc_tfo_cookie = 0x0, sc_pspare = 0x0, sc_spare = 0xfffff8036f5523e8}
(kgdb) print sc->sc_todctx
$2 = (void *) 0xfffff80539778f00
(kgdb) print *(struct synq_entry *)sc->sc_todctx
$3 = {lctx = 0x0, syn = 0x0, flags = 0, ok_to_respond = 0, refcnt = 1111638594, tid = 1111638594, iss = 1111638594, irs = 1111638594, ts = 0, txqid = 2, rxqid = 1, l2e_idx = 0, ulp_mode = 5, rcv_bufsize = 64, tcp_opt = 32778, toep = 0x0}

The cxgbe(4) code isn’t defensive enough to fend off any invalid addresses it receives. The real issue might be in the overall syncache handling.

Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes:

<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>