Chelsio T6225-CR and TOE

After upgrading to a recent base/stable/11, I noticed ifconfig -m now displays two more bits:

cc0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
             options=ec07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,          VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=ecc7bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,TOE4,TOE6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:07:43:XX:XX:X0
        hwaddr 00:07:43:XX:XX:X0
        nd6 options=8021<PERFORMNUD,AUTO_LINKLOCAL,DEFAULTIF>
        media: Ethernet 10Gbase-SR <full-duplex,rxpause,txpause>
        status: active
        supported media:
                media 10Gbase-SR mediaopt full-duplex,rxpause,txpause
                media 10Gbase-SR mediaopt full-duplex,rxpause
                media 10Gbase-SR mediaopt full-duplex,txpause
                media 10Gbase-SR mediaopt full-duplex
                media 1000baseSX mediaopt full-duplex,rxpause,txpause
                media 1000baseSX mediaopt full-duplex,rxpause
                media 1000baseSX mediaopt full-duplex,txpause
                media 1000baseSX mediaopt full-duplex

TOE4 and TOE6 appear in the capabilities word, but not in the options word. (TOE in this case being TCP Offload Engine.)

I found Chelsio’s Unified Wire documentation for FreeBSD, and it all became apparent.

kldload t4_tom
ifconfig cc0 toe
sysctl dev.t6nex.0.toe.tx_zcopy=1
sysctl dev.t6nex.0.toe.ddp=1

From previous experiments, I have these settings in /etc/sysctl.conf, thanks to https://calomel.org/freebsd_network_tuning.html and https://fasterdata.es.net/host-tuning/freebsd/:

kern.ipc.maxsockbuf=16777216
net.inet.tcp.recvbuf_max=16777216
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.sendbuf_inc=16384
net.inet.tcp.recvbuf_inc=524288
net.inet.tcp.sendbuf_auto=1
net.inet.tcp.recvbuf_auto=1
net.inet.tcp.cc.algorithm=htcp

I should increase the MTU to 9000 octets, but that will require me to reconfigure the entire network.

For some reason (licensing?), we can’t place t4_tom in the static kernel.

Now, ifconfig -m shows the same bits for both options and capabilities:

     options=ecc7bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,TOE4,TOE6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
capabilities=ecc7bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,TOE4,TOE6,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>

Looking at sysctl(8) dev.t6nex.0.misc.tids, I see some usage:

dev.t6nex.0.misc.tids: ATID range: 0-8191, in use: 0
TID range: 0-511, 3072-19455, in use: 3
STID range: 512-1023, in use: 8
FTID range: 1024-3055
HW TID usage: 2 IP users, 1 IPv6 users

I made the TOE changes permanent by editing /boot/loader.conf, /etc/rc.conf, and /etc/sysctl.conf.

#notsponsored

Update 2018-09-08

The Chelsio TOE driver isn’t perfect. Here are some details from last night’s crash.

Fatal trap 9: general protection fault while in kernel mode
cpuid = 3; apic id = 06
instruction pointer     = 0x20:0xffffffff8242cfc4
stack pointer           = 0x28:0xfffffe085f209420
frame pointer           = 0x28:0xfffffe085f209450
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (irq271: t6nex0:0a2)
trap number             = 9
panic: general protection fault
cpuid = 3
KDB: stack backtrace:
db_trace_self_wrapper() at 0xffffffff804c2e3b = db_trace_self_wrapper+0x2b/frame 0xfffffe085f209130
vpanic() at 0xffffffff807a0e37 = vpanic+0x177/frame 0xfffffe085f209190
panic() at 0xffffffff807a0cb3 = panic+0x43/frame 0xfffffe085f2091f0
trap_fatal() at 0xffffffff80ab0edf = trap_fatal+0x35f/frame 0xfffffe085f209240
trap() at 0xffffffff80ab049e = trap+0x5e/frame 0xfffffe085f209350
calltrap() at 0xffffffff80a916cc = calltrap+0x8/frame 0xfffffe085f209350
--- trap 0x9, rip = 0xffffffff8242cfc4, rsp = 0xfffffe085f209420, rbp = 0xfffffe085f209450 ---
offload_socket() at 0xffffffff8242cfc4 = offload_socket+0x14/frame 0xfffffe085f209450
t4_offload_socket() at 0xffffffff8242b9d9 = t4_offload_socket+0x29/frame 0xfffffe085f209480
syncache_expand() at 0xffffffff809632cc = syncache_expand+0x123c/frame 0xfffffe085f2095d0
tcp_input() at 0xffffffff809544f6 = tcp_input+0xfc6/frame 0xfffffe085f209700
ip_input() at 0xffffffff808d77c3 = ip_input+0x113/frame 0xfffffe085f209760
netisr_dispatch_src() at 0xffffffff808b7548 = netisr_dispatch_src+0xa8/frame 0xfffffe085f2097b0
ether_demux() at 0xffffffff808a0600 = ether_demux+0x140/frame 0xfffffe085f2097e0
ether_nh_input() at 0xffffffff808a1716 = ether_nh_input+0x336/frame 0xfffffe085f209840
netisr_dispatch_src() at 0xffffffff808b7548 = netisr_dispatch_src+0xa8/frame 0xfffffe085f209890
ether_input() at 0xffffffff808a09a6 = ether_input+0x26/frame 0xfffffe085f2098b0
tcp_lro_flush() at 0xffffffff80958df8 = tcp_lro_flush+0x208/frame 0xfffffe085f2098d0
tcp_lro_flush_all() at 0xffffffff80958f3f = tcp_lro_flush_all+0xff/frame 0xfffffe085f209910
service_iq() at 0xffffffff80521431 = service_iq+0x621/frame 0xfffffe085f2099c0
t4_intr() at 0xffffffff80521780 = t4_intr+0x30/frame 0xfffffe085f2099e0
intr_event_execute_handlers() at 0xffffffff80765a69 = intr_event_execute_handlers+0xe9/frame 0xfffffe085f209a20
ithread_loop() at 0xffffffff80766157 = ithread_loop+0xe7/frame 0xfffffe085f209a70
fork_exit() at 0xffffffff80762ec3 = fork_exit+0x83/frame 0xfffffe085f209ab0
fork_trampoline() at 0xffffffff80a9268e = fork_trampoline+0xe/frame 0xfffffe085f209ab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

I also learned you must not ever load t4_tom.ko from /boot/loader.conf. It’s being loaded too early in the boot process, or maybe the driver’s simply immature. Instead, I added these lines to /etc/rc.local. I edited /boot/loader.conf, /etc/rc.conf, and /etc/sysctl.conf to follow suit:

if [ -z `/sbin/kldstat | /usr/bin/grep 't4_tom\.ko'` ]; then
  /sbin/kldload t4_tom && \
  /sbin/ifconfig cc0 toe && \
  /sbin/sysctl -q dev.t6nex.0.toe.ddp=1 >/dev/null && \
  /sbin/sysctl -q dev.t6nex.0.toe.tx_zcopy=1 >/dev/null
fi

Update 2018-09-09

I had several crashes last night.

Fatal trap 9: general protection fault while in kernel mode
cpuid = 1; apic id = 02
instruction pointer     = 0x20:0xffffffff8242afc4
stack pointer           = 0x28:0xfffffe085f1ff420
frame pointer           = 0x28:0xfffffe085f1ff450
code segment            = base 0x0, limit 0xfffff, type 0x1b
                        = DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags        = interrupt enabled, resume, IOPL = 0
current process         = 12 (irq269: t6nex0:0a0)
trap number             = 9
panic: general protection fault
cpuid = 1
KDB: stack backtrace:
db_trace_self_wrapper() at 0xffffffff804c2e3b = db_trace_self_wrapper+0x2b/frame 0xfffffe085f1ff130
vpanic() at 0xffffffff807a0e37 = vpanic+0x177/frame 0xfffffe085f1ff190
panic() at 0xffffffff807a0cb3 = panic+0x43/frame 0xfffffe085f1ff1f0
trap_fatal() at 0xffffffff80ab0f0f = trap_fatal+0x35f/frame 0xfffffe085f1ff240
trap() at 0xffffffff80ab04ce = trap+0x5e/frame 0xfffffe085f1ff350
calltrap() at 0xffffffff80a916cc = calltrap+0x8/frame 0xfffffe085f1ff350
--- trap 0x9, rip = 0xffffffff8242afc4, rsp = 0xfffffe085f1ff420, rbp = 0xfffffe085f1ff450 ---
offload_socket() at 0xffffffff8242afc4 = offload_socket+0x14/frame 0xfffffe085f1ff450
t4_offload_socket() at 0xffffffff824299d9 = t4_offload_socket+0x29/frame 0xfffffe085f1ff480
syncache_expand() at 0xffffffff809632cc = syncache_expand+0x123c/frame 0xfffffe085f1ff5d0
tcp_input() at 0xffffffff809544f6 = tcp_input+0xfc6/frame 0xfffffe085f1ff700
ip_input() at 0xffffffff808d77c3 = ip_input+0x113/frame 0xfffffe085f1ff760
netisr_dispatch_src() at 0xffffffff808b7548 = netisr_dispatch_src+0xa8/frame 0xfffffe085f1ff7b0
ether_demux() at 0xffffffff808a0600 = ether_demux+0x140/frame 0xfffffe085f1ff7e0
ether_nh_input() at 0xffffffff808a1716 = ether_nh_input+0x336/frame 0xfffffe085f1ff840
netisr_dispatch_src() at 0xffffffff808b7548 = netisr_dispatch_src+0xa8/frame 0xfffffe085f1ff890
ether_input() at 0xffffffff808a09a6 = ether_input+0x26/frame 0xfffffe085f1ff8b0
tcp_lro_flush() at 0xffffffff80958df8 = tcp_lro_flush+0x208/frame 0xfffffe085f1ff8d0
tcp_lro_flush_all() at 0xffffffff80958f3f = tcp_lro_flush_all+0xff/frame 0xfffffe085f1ff910
service_iq() at 0xffffffff80521431 = service_iq+0x621/frame 0xfffffe085f1ff9c0
t4_intr() at 0xffffffff80521780 = t4_intr+0x30/frame 0xfffffe085f1ff9e0
intr_event_execute_handlers() at 0xffffffff80765a69 = intr_event_execute_handlers+0xe9/frame 0xfffffe085f1ffa20
ithread_loop() at 0xffffffff80766157 = ithread_loop+0xe7/frame 0xfffffe085f1ffa70
fork_exit() at 0xffffffff80762ec3 = fork_exit+0x83/frame 0xfffffe085f1ffab0
fork_trampoline() at 0xffffffff80a9268e = fork_trampoline+0xe/frame 0xfffffe085f1ffab0
--- trap 0, rip = 0, rsp = 0, rbp = 0 ---

Looking at the two backtraces, I see we are 20 octets into the offload_socket() function. At lines 207 through 209, I see three pointers being accessed. Next, I see a writelock being asserted in line 212. Maybe the code is chasing a NULL pointer somewhere in this area.

/*
 * Set up the socket for TCP offload.
 */
void
offload_socket(struct socket *so, struct toepcb *toep)
{
        struct tom_data *td = toep->td;
        struct inpcb *inp = sotoinpcb(so);
        struct tcpcb *tp = intotcpcb(inp);
        struct sockbuf *sb;

        INP_WLOCK_ASSERT(inp);

        /* Update socket */
        sb = &so->so_snd;
        SOCKBUF_LOCK(sb);
        sb->sb_flags |= SB_NOCOALESCE;
        SOCKBUF_UNLOCK(sb);
        sb = &so->so_rcv;
        SOCKBUF_LOCK(sb);
        sb->sb_flags |= SB_NOCOALESCE;
        if (inp->inp_vflag & INP_IPV6)
                so->so_proto = &toe6_protosw;
        else
                so->so_proto = &toe_protosw;
        SOCKBUF_UNLOCK(sb);

        /* Update TCP PCB */
        tp->tod = &td->tod;
        tp->t_toe = toep;
        tp->t_flags |= TF_TOE;

        /* Install an extra hold on inp */
        toep->inp = inp;
        toep->flags |= TPF_ATTACHED;
        in_pcbref(inp);

        /* Add the TOE PCB to the active list */
        mtx_lock(&td->toep_list_lock);
        TAILQ_INSERT_HEAD(&td->toep_list, toep, link);
        mtx_unlock(&td->toep_list_lock);
}

I decided to turn off dev.t6nex.0.toe.ddp and dev.t6nex.0.toe.tx_zcopy, and edited /etc/rc.local accordingly.

sysctl dev.t6nex.0.toe.ddp=0
sysctl dev.t6nex.0.toe.tx_zcopy=0

If I see more crashes over the next 24 hours, I’ll disable encryption on /dev/gpt/swap0, and allow crash dumps to be written on that device. I’d hate to declare the t4_tom kernel module a lost cause, but it may come to that.

More action on 2018-09-09

That’s it. The t4_tom kernel module will not be loaded anytime soon. Another crash happened while I was around, and I managed to get hold of the crash dump.

Running kgdb -n 0 and looking around, revealed this:

(kgdb) bt
#0  __curthread () at ./machine/pcpu.h:230
#1  doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:315
#2  0xffffffff807a0a30 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:383
#3  0xffffffff807a0e71 in vpanic (fmt=<optimized out>, ap=0xfffffe085f2091d0) at /usr/src/sys/kern/kern_shutdown.c:776
#4  0xffffffff807a0cb3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:707
#5  0xffffffff80ab0f0f in trap_fatal (frame=0xfffffe085f209360, eva=0) at /usr/src/sys/amd64/amd64/trap.c:891
#6  0xffffffff80ab04ce in trap (frame=0xfffffe085f209360) at /usr/src/sys/amd64/amd64/trap.c:216
#7  <signal handler called>
#8  offload_socket (so=0xfffff80255ddba20, toep=0xb284e0ba48430700) at /usr/src/sys/dev/cxgbe/tom/t4_tom.c:207
#9  0xffffffff824299d9 in t4_offload_socket (tod=<optimized out>, arg=0xfffff800452dc0c0, so=0xfffff80255ddba20) at /usr/src/sys/dev/cxgbe/tom/t4_listen.c:941
#10 0xffffffff809632cc in syncache_socket (lso=<optimized out>, m=<optimized out>, sc=<optimized out>) at /usr/src/sys/netinet/tcp_syncache.c:933
#11 syncache_expand (inc=0xfffffe085f209678, to=0xfffff80255ddba20, th=<optimized out>, lsop=<optimized out>, m=<optimized out>) at /usr/src/sys/netinet/tcp_syncache.c:1176
#12 0xffffffff809544f6 in tcp_input (mp=<optimized out>, offp=<optimized out>, proto=<optimized out>) at /usr/src/sys/netinet/tcp_input.c:1113
#13 0xffffffff808d77c3 in ip_input (m=0x0) at /usr/src/sys/netinet/ip_input.c:825
#14 0xffffffff808b7548 in netisr_dispatch_src (proto=1, source=<optimized out>, m=0xfffff800452dc000) at /usr/src/sys/net/netisr.c:1120
#15 0xffffffff808a0600 in ether_demux (ifp=0xfffff80011db6800, m=0xb284e0ba48430700) at /usr/src/sys/net/if_ethersubr.c:878
#16 0xffffffff808a1716 in ether_input_internal (ifp=0xfffff80011db6800, m=0xb284e0ba48430700) at /usr/src/sys/net/if_ethersubr.c:666
#17 ether_nh_input (m=<optimized out>) at /usr/src/sys/net/if_ethersubr.c:696
#18 0xffffffff808b7548 in netisr_dispatch_src (proto=5, source=<optimized out>, m=0xfffff800452dc000) at /usr/src/sys/net/netisr.c:1120
#19 0xffffffff808a09a6 in ether_input (ifp=<optimized out>, m=0x0) at /usr/src/sys/net/if_ethersubr.c:786
#20 0xffffffff80958df8 in tcp_lro_flush (lc=0xfffffe0001357540, le=0xfffff8003996f780) at /usr/src/sys/netinet/tcp_lro.c:371
#21 0xffffffff80958f3f in tcp_lro_rx_done (lc=<optimized out>) at /usr/src/sys/netinet/tcp_lro.c:262
#22 tcp_lro_flush_all (lc=0xfffffe0001357540) at /usr/src/sys/netinet/tcp_lro.c:509
#23 0xffffffff80521431 in service_iq (iq=<optimized out>, budget=0) at /usr/src/sys/dev/cxgbe/t4_sge.c:1487
#24 0xffffffff80521780 in t4_intr (arg=0xfffffe0001357400) at /usr/src/sys/dev/cxgbe/t4_sge.c:1250
#25 0xffffffff80765a69 in intr_event_execute_handlers (p=<optimized out>, ie=0xfffff80011d95700) at /usr/src/sys/kern/kern_intr.c:1336
#26 0xffffffff80766157 in ithread_execute_handlers (ie=<optimized out>, p=<optimized out>) at /usr/src/sys/kern/kern_intr.c:1349
#27 ithread_loop (arg=0xfffff80011da0700) at /usr/src/sys/kern/kern_intr.c:1430
#28 0xffffffff80762ec3 in fork_exit (callout=0xffffffff80766070 <ithread_loop>, arg=0xfffff80011da0700, frame=0xfffffe085f209ac0) at /usr/src/sys/kern/kern_fork.c:1072
#29 <signal handler called>
(kgdb) up
#1  doadump (textdump=1) at /usr/src/sys/kern/kern_shutdown.c:315
315             dumptid = curthread->td_tid;
(kgdb)
#2  0xffffffff807a0a30 in kern_reboot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:383
383                     doadump(TRUE);
(kgdb)
#3  0xffffffff807a0e71 in vpanic (fmt=<optimized out>, ap=0xfffffe085f2091d0) at /usr/src/sys/kern/kern_shutdown.c:776
776             kern_reboot(bootopt);
(kgdb)
#4  0xffffffff807a0cb3 in panic (fmt=<unavailable>) at /usr/src/sys/kern/kern_shutdown.c:707
707             vpanic(fmt, ap);
(kgdb)
#5  0xffffffff80ab0f0f in trap_fatal (frame=0xfffffe085f209360, eva=0) at /usr/src/sys/amd64/amd64/trap.c:891
891                     panic("%s", trap_msg[type]);
(kgdb)
#6  0xffffffff80ab04ce in trap (frame=0xfffffe085f209360) at /usr/src/sys/amd64/amd64/trap.c:216
216             if (type == T_RESERVED) {
(kgdb)
#7  <signal handler called>
(kgdb)
#8  offload_socket (so=0xfffff80255ddba20, toep=0xb284e0ba48430700) at /usr/src/sys/dev/cxgbe/tom/t4_tom.c:207
207             struct tom_data *td = toep->td;
(kgdb) list
202      * Set up the socket for TCP offload.
203      */
204     void
205     offload_socket(struct socket *so, struct toepcb *toep)
206     {
207             struct tom_data *td = toep->td;
208             struct inpcb *inp = sotoinpcb(so);
209             struct tcpcb *tp = intotcpcb(inp);
210             struct sockbuf *sb;
211
(kgdb) print *toep
Cannot access memory at address 0xb284e0ba48430700
(kgdb) up
#9  0xffffffff824299d9 in t4_offload_socket (tod=<optimized out>, arg=0xfffff800452dc0c0, so=0xfffff80255ddba20) at /usr/src/sys/dev/cxgbe/tom/t4_listen.c:941
941             offload_socket(so, toep);
(kgdb) print *toep
Cannot access memory at address 0xb284e0ba48430700
(kgdb) list
936             INP_INFO_RLOCK_ASSERT(&V_tcbinfo); /* prevents bad race with accept() */
937             INP_WLOCK_ASSERT(inp);
938             KASSERT(synqe->flags & TPF_SYNQE,
939                 ("%s: %p not a synq_entry?", __func__, arg));
940
941             offload_socket(so, toep);
942             make_established(toep, cpl->snd_isn, cpl->rcv_isn, cpl->tcp_opt);
943             toep->flags |= TPF_CPL_PENDING;
944             update_tid(sc, synqe->tid, toep);
945             synqe->flags |= TPF_SYNQE_EXPANDED;
(kgdb) print *synqe
$1 = {link = {tqe_next = 0x0, tqe_prev = 0xfffff80029c72160}, flags = 4352, tid = 13952, lctx = 0xfffff80029c72100, syn = 0xfffff800452dc000, iss = 4209764294, ts = 0, wr = 0, refcnt = 0, l2e_idx = 0, rcv_bufsize = 64}
(kgdb) print *synqe->syn
$2 = {{m_next = 0x0, m_slist = {sle_next = 0x0}, m_stailq = {stqe_next = 0x0}}, {m_nextpkt = 0x0, m_slistpkt = {sle_next = 0x0}, m_stailqpkt = {stqe_next = 0x0}}, m_data = 0xfffff800452dc058 "D", m_len = 84, m_type = 1, m_flags = 2, {{m_pkthdr = {
        rcvif = 0xfffff80011db6800, tags = {slh_first = 0x0}, len = 84, flowid = 196609, csum_flags = 0, fibnum = 0, cosqos = 0 '\000', rsstype = 0 '\000', l2hlen = 0 '\000', l3hlen = 0 '\000', l4hlen = 0 '\000', l5hlen = 0 '\000', PH_per = {
          eight = "\000\000\000\000\000\000\000", sixteen = {0, 0, 0, 0}, thirtytwo = {0, 0}, sixtyfour = {0}, unintptr = {0}, ptr = 0x0}, PH_loc = {eight = "\000\000\000\000\000\000\000", sixteen = {0, 0, 0, 0}, thirtytwo = {0, 0}, sixtyfour = {0}, unintptr = {0},
          ptr = 0x0}}, {m_ext = {{ext_count = 2151022660, ext_cnt = 0x3c00000080360044}, ext_buf = 0x2ffff0014050e <error: Cannot access memory at address 0x2ffff0014050e>, ext_size = 4009951232, ext_type = 2, ext_flags = 65344, ext_free = 0xb284e0ba48430700,
          ext_arg1 = 0x4500087f2f4461, ext_arg2 = 0x6f50000a9f92800}, m_pktdat = 0xfffff800452dc058 "D"}}, m_dat = 0xfffff800452dc020 ""}}
(kgdb) print cpl
$3 = (struct cpl_pass_establish *) 0xfffff800452dc058
(kgdb) print cpl+1
$4 = (struct cpl_pass_establish *) 0xfffff800452dc070
(kgdb) print *cpl
$5 = {ot = {opcode_tid = 2151022660, opcode = 68 'D'}, rsvd = 1006632960, tos_stid = 1312014, mac_idx = 65535, tcp_opt = 2, snd_isn = 4009951232, rcv_isn = 16728066}
(kgdb) print *(cpl+1)
$6 = {ot = {opcode_tid = 1212352256, opcode = 0 '\000'}, rsvd = 2995052730, tos_stid = 2133804129, mac_idx = 8, tcp_opt = 69, snd_isn = 2851678208, rcv_isn = 116719616}
(kgdb) print (struct toepcb **)(cpl+1)
$7 = (struct toepcb **) 0xfffff800452dc070
(kgdb) print *(struct toepcb **)(cpl+1)
$8 = (struct toepcb *) 0xb284e0ba48430700
(kgdb) print *(struct toepcb *)(cpl+1)
$9 = {link = {tqe_next = 0xb284e0ba48430700, tqe_prev = 0x4500087f2f4461}, flags = 2851678208, refcount = 116719616, td = 0x27806cc3b81fd1ba, inp = 0x5b3a50007af408ae, vnet = 0x25000000000f157, vi = 0x77130004, ofld_txq = 0xe4fc4e0200000000, ofld_rxq = 0x320400000031,
  ctrlq = 0x1f000000540400, l2te = 0x0, ce = 0xfffff80029c72160, tid = 4352, tx_total = 13952, tx_credits = 700915968, tx_nocompl = 4294965248, plen_nocompl = 1160626176, sb_cc = 4294965248, rx_credits = -85203002, ulp_mode = 0, ulpcb = 0x0, ulpcb2 = 0x40000000000000,
  ulp_pduq = {mq_head = {stqh_first = 0x0, stqh_last = 0x0}, mq_len = 1160626464, mq_maxlen = -2048}, ulp_pdu_reclaimq = {mq_head = {stqh_first = 0x10000004c, stqh_last = 0x320400000031}, mq_len = 5506048, mq_maxlen = 2162688}, ddp = {flags = 1819178241, db = {{ps = 0x64,
        job = 0xffff010010000000, cancel_pending = 65535}, {ps = 0x740100000001000b, job = 0x4c45530d00000043, cancel_pending = 542393157}}, cached_pagesets = {tqh_first = 0x649050000, tqh_last = 0x0}, aiojobq = {tqh_first = 0x0, tqh_last = 0x0}, waiting_count = 16777216,
    active_count = 0, cached_count = 0, active_id = 0, requeue_task = {ta_link = {stqe_next = 0xd09ad09a01000000}, ta_pending = 24576, ta_priority = 29440, ta_func = 0x58, ta_context = 0x0}, queueing = 0x100000000, lock = {lock_object = {lo_name = 0x0, lo_flags = 0,
        lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}}, aiotx_jobq = {tqh_first = 0x0, tqh_last = 0x0}, aiotx_task = {ta_link = {stqe_next = 0x0}, ta_pending = 0, ta_priority = 0, ta_func = 0xfffff8044393e500, ta_context = 0x0}, aiotx_task_active = false,
  txsd_total = 176 '\260', txsd_pidx = 252 '\374', txsd_cidx = 37 '%', txsd_avail = 4 '\004', txsd = 0xfffff800452dc218}

So, the crash happens on line 207, the first statement of the offload_socket() function. Is 0xb284e0ba48430700 a vaild address?

Looking at t4_offload_socket(), how well does this function trust or handle the pointers it’s given?

void
t4_offload_socket(struct toedev *tod, void *arg, struct socket *so)
{
        struct adapter *sc = tod->tod_softc;
        struct synq_entry *synqe = arg;
#ifdef INVARIANTS
        struct inpcb *inp = sotoinpcb(so);
#endif
        struct cpl_pass_establish *cpl = mtod(synqe->syn, void *);
        struct toepcb *toep = *(struct toepcb **)(cpl + 1);

        INP_INFO_RLOCK_ASSERT(&V_tcbinfo); /* prevents bad race with accept() */
        INP_WLOCK_ASSERT(inp);
        KASSERT(synqe->flags & TPF_SYNQE,
            ("%s: %p not a synq_entry?", __func__, arg));

        offload_socket(so, toep);
        make_established(toep, cpl->snd_isn, cpl->rcv_isn, cpl->tcp_opt);
        toep->flags |= TPF_CPL_PENDING;
        update_tid(sc, synqe->tid, toep);
        synqe->flags |= TPF_SYNQE_EXPANDED;
}

What’s going on at line 934? It looks more like pointer mayhem. It easy to see how we derive the 0xb284e0ba48430700 address. I’m not certain this is the proper way.

Maybe line 934 should simply read:

        struct toepcb *toep = (struct toepcb *)(cpl + 1);

Or maybe the problem is where and why did (cpl + 1)->link->tqe_next get the value 0xb284e0ba48430700?