Kernel panic due to memory corruption

This is what a kernel panic may look like when facing memory corruption.

kernel: panic: solaris assert: arc_buf_alloc_impl(hdr, private, compressed_read, 1, &buf) == 0 (0x5 == 0x0), file: /usr/src/sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c, line: 5318
kernel: cpuid = 1
kernel: KDB: stack backtrace:
kernel: db_trace_self_wrapper() at 0xffffffff8046b52b = db_trace_self_wrapper+0x2b/frame 0xfffffe0122681380
kernel: vpanic() at 0xffffffff806c5ba6 = vpanic+0x186/frame 0xfffffe0122681400
kernel: panic() at 0xffffffff806c5a13 = panic+0x43/frame 0xfffffe0122681460
kernel: assfail3() at 0xffffffff8030987c = assfail3+0x2c/frame 0xfffffe0122681480
kernel: arc_read() at 0xffffffff80322ddc = arc_read+0x83c/frame 0xfffffe0122681510
kernel: dbuf_read() at 0xffffffff8032e918 = dbuf_read+0x678/frame 0xfffffe01226815c0
kernel: dmu_buf_hold_array_by_dnode() at 0xffffffff80338bf3 = dmu_buf_hold_array_by_dnode+0x203/frame 0xfffffe0122681630
kernel: dmu_read_uio_dnode() at 0xffffffff8033a167 = dmu_read_uio_dnode+0x37/frame 0xfffffe01226816a0
kernel: dmu_read_uio_dbuf() at 0xffffffff8033a10b = dmu_read_uio_dbuf+0x3b/frame 0xfffffe01226816d0
kernel: zfs_freebsd_read() at 0xffffffff803d7e80 = zfs_freebsd_read+0x660/frame 0xfffffe0122681780
kernel: VOP_READ_APV() at 0xffffffff80a7b77c = VOP_READ_APV+0x7c/frame 0xfffffe01226817b0
kernel: vn_read() at 0xffffffff807a0815 = vn_read+0x195/frame 0xfffffe0122681830
kernel: vn_io_fault1() at 0xffffffff8079e4e9 = vn_io_fault1+0x169/frame 0xfffffe0122681970
kernel: vn_io_fault() at 0xffffffff8079c6ac = vn_io_fault+0x18c/frame 0xfffffe01226819e0
kernel: dofileread() at 0xffffffff807291aa = dofileread+0xba/frame 0xfffffe0122681a20
kernel: kern_readv() at 0xffffffff80728db8 = kern_readv+0x68/frame 0xfffffe0122681a70
kernel: sys_read() at 0xffffffff80728d46 = sys_read+0x86/frame 0xfffffe0122681ac0
kernel: amd64_syscall() at 0xffffffff809d5f18 = amd64_syscall+0xa38/frame 0xfffffe0122681bf0
kernel: fast_syscall_common() at 

Upgrading XenServer 7.1 to 7.3 via 7.2

It’s winter break, giving me time to bring systems down for their much needed maintenance. One of my tasks was to upgrade our XenServers. They’re not joined to the same pool for historical reasons. One of them is close to 8 years old, the other one is fairly new. For the first server I chose to reboot into the hypervisor after upgrading to 7.2, before upgrading to 7.3. A wise choice. Just for the fun of it, I chose not to reboot into the hypervisor on the second server between 7.2 and 7.3. This left me with the choice of restoring the old 7.1 version or performing a clean reinstall of 7.3. Neither option was inviting, so I had to reboot into the hypervisor prior to upgrading to 7.3. Lesson learned.

The irony of this story is that XenServer 7.4 was released two days ago. Luckily, I can use the new update feature and save me a trip to the server room.