Your own virtualized desktop FreeBSD lab

Comments Off on Your own virtualized desktop FreeBSD lab

If you’re like me, eager to test new stuff in FreeBSD, you might as well run a virtualized FreeBSD laboratory on your desktop. I use Oracle VirtualBox, but you might as well use Microsoft Hyper-V, real hardware, or some other contraption. Couple this with ZFS and boot environments, and you’re even able to rapidly revert any mishaps, although enabled ZFS features can’t be reverted without some help in advance from the virtualization software.

Running such experiments should be economical in terms of disk space and network bandwidth used for the Subversion working copies for both the OS source code and the ports collection, for the downloaded source code of third party software (/usr/ports/distfiles), for the packages produced, if any, and for the object tree. This paves way for spending less time updating one VM when another VM already has updated say the common working copy for the ports collection and downloaded any missing distfiles, or when one VM has already compiled a new world and kernel that you intend on installing on other VMs.

The only snag is being limited to run a single VM at a time.

If you’re using real hardware, you can avoid this limitation by having a NFS file server export all the needed files to all your individual experiments simultaneously.

Let’s got back to my use of VirtualBox. I use several common virtual harddrives for my VMs:

  1. Working copy of base/head, named common-base-head.vdi
  2. Working copy of base/stable/10, named common-base-stable-10.vdi
  3. Working copy of base/stable/9, named common-base-stable-9.vdi
  4. Working copy of base/stable/8, named common-base-stable-8.vdi
  5. Working copy of ports/head, named common-ports-head.vdi
  6. Storage for /usr/ports/distfiles, named common-ports-distfiles.vdi
  7. Storage for /usr/ports/packages; one for each of the following combinations:
    1. base/head on amd64, named common-packages-amd64-base-head.vdi
    2. base/head on i386, named common-packages-i386-base-head.vdi
    3. base/stable/10 on amd64, named common-packages-amd64-base-stable-10.vdi
    4. base/stable/10 on i386, named common-packages-i386-base-stable-10.vdi
    5. base/stable/9 on amd64, named common-packages-amd64-base-stable-9.vdi
    6. base/stable/9 on i386, named common-packages-i386-base-stable-9.vdi
    7. base/stable/8 on amd64, named common-packages-amd64-base-stable-8.vdi
    8. base/stable/8 on i386, named common-packages-i386-base-stable-8.vdi
  8. Storage for /usr/ports/workdirs, named common-ports-workdirs.vdi (under the assumption you’ll always run rm -Rf /usr/ports/workdirs/* when reusing this drive on other VMs)
  9. Storage for /usr/obj; one for each of the following combinations:
    1. base/head on amd64, named common-usr4-obj-amd64-base-head.vdi
    2. base/head on i386, named common-usr-obj-base-i386-head.vdi
    3. base/stable/10 on amd64, named common-usr-obj-amd64-base-stable-10.vdi
    4. base/stable/10 on i386, named common-usr-obj-i386-base-stable-10.vdi
    5. base/stable/9 on amd64, named common-usr-obj-amd64-base-stable-9.vdi
    6. base/stable/9 on i386, named common-usr-obj-i386-base-stable-9.vdi
    7. base/stable/8 on amd64, named common-usr-obj-amd64-base-stable-8.vdi
    8. base/stable/8 on i386, named common-usr-obj-i386-base-stable-8.vdi

I might have gone a bit overboard with all my virtual harddrives, but I guess it’ll pay up in the long run.

All these virtual harddrives are 5 GiB (5120 MiB) in size. That should be enough space for the foreseeable future. Any of these drives and their filesystems may be expanded at any time.

A question arrives, namely how to get started. I recommend you start by creating a single VM running base/release/10 or some snapshot of base/stable/10. The reason is that svnlite is available straight out of the box in these versions. base/head is simply to noisy with its GENERIC kernel, and after all it’s a moving target.

Start by creating a new VM. Remove the IDE controller, as it has low performance both in real life and when emulated. Add the SATA controller. Reattach the VM’s ordinary virtual harddrives and one DVD drive to the SATA controller. Try not to place the DVD drive beyond the fourth SATA port as VirtualBox doesn’t detect DVD drives that high up in the chain. Add the SCSI and the SAS controllers. Create the first 15 common virtual harddrives and attach them to the SCSI controller, taking care to name the virtual harddrives properly and placing them in a directory not associated with any particular VM. The directory “C:\Users\username\VirtualBox VMs” should be suitable for this purpose. Make sure none of these drives occupies SCSI port 7 as this LUN is reserved for the SCSI controller itself. Any harddrive at SCSI port 7 will NOT be detected by FreeBSD. Create the last 8 common virtual harddrives and attach them to the SAS controller. Install a small system on the VM’s ordinary virtual harddrives. Reboot into the preparation system.

You can also use the command line tool VBoxManage to create the common virtual harddrives:

C:
cd "\Users\username\VirtualBox VMs"
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-base-head.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-base-stable-10.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-base-stable-9.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-base-stable-8.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-ports-head.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-ports-distfiles.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-packages-amd64-base-head.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-packages-i386-base-head.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-packages-amd64-base-stable-10.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-packages-i386-base-stable-10.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-packages-amd64-base-stable-9.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-packages-i386-base-stable-9.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-packages-amd64-base-stable-8.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-packages-i386-base-stable-8.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-ports-workdirs.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-usr-obj-amd64-base-head.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-usr-obj-i386-base-head.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-usr-obj-amd64-base-stable-10.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-usr-obj-i386-base-stable-10.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-usr-obj-amd64-base-stable-9.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-usr-obj-i386-base-stable-9.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-usr-obj-amd64-base-stable-8.vdi
C:\Program Files\Oracle\VirtualBox\VboxManage.exe createhd --size 5120 --filename common-usr-obj-i386-base-stable-8.vdi

The SATA controller can control up to 30 units. The SCSI controller can control up to 15 units. The SAS controller can control up to 8 units. The SATA controller is run by the ahci driver, while the SCSI and the SAS controllers are both run by the mpt driver.

Assuming your common virtual harddrives are named and attached as follows:

Filename SCSI port Device name
common-base-head.vdi 0 da0
common-base-stable-10.vdi 1 da1
common-base-stable-9.vdi 2 da2
common-base-stable-8.vdi 3 da3
common-ports-head.vdi 4 da4
common-ports-distfiles.vdi 5 da5
common-packages-amd64-base-head.vdi 6 da6
Add nothing here 7  
common-packages-i386-base-head.vdi 8 da7
common-packages-amd64-base-stable-10.vdi 9 da8
common-packages-i386-base-stable-10.vdi 10 da9
common-packages-amd64-base-stable-9.vdi 11 da10
common-packages-i386-base-stable-9.vdi 12 da11
common-packages-amd64-base-stable-8.vdi 13 da12
common-packages-i386-base-stable-8.vdi 14 da13
common-ports-workdirs.vdi 15 da14
Filename SAS port Device name
common-usr-obj-amd64-base-head.vdi 0 da15
common-usr-obj-i386-base-head.vdi 1 da16
common-usr-obj-amd64-base-stable-10.vdi 2 da17
common-usr-obj-i386-base-stable-10.vdi 3 da18
common-usr-obj-amd64-base-stable-9.vdi 4 da19
common-usr-obj-i386-base-stable-9.vdi 5 da20
common-usr-obj-amd64-base-stable-8.vdi 6 da21
common-usr-obj-i386-base-stable-8.vdi 7 da22

I chose the SCSI and SAS controllers to make the device names intuitive while we’re preparing the virtual harddrives.

The next step is to create GPT partition tables, UFS partitions, and initialize the filesystems on each virtual harddrive:

for d in 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22; do \
  gpart create -s GPT da$d; \
  gpart add -t freebsd-ufs da$d; \
  newfs /dev/da${d}p1; \
done

GPT labels are a sure thing to keep track of the filesystems no matter what order you attach the drives in your future VMs:

gpart modify -i 1 -l base_head                     da0
gpart modify -i 1 -l base_stable_10                da1
gpart modify -i 1 -l base_stable_9                 da2
gpart modify -i 1 -l base_stable_8                 da3
gpart modify -i 1 -l ports_head                    da4
gpart modify -i 1 -l ports_distfiles               da5
gpart modify -i 1 -l packages_amd64_base_head      da6
gpart modify -i 1 -l packages_i386_base_head       da7
gpart modify -i 1 -l packages_amd64_base_stable_10 da8
gpart modify -i 1 -l packages_i386_base_stable_10  da9
gpart modify -i 1 -l packages_amd64_base_stable_9  da10
gpart modify -i 1 -l packages_i386_base_stable_9   da11
gpart modify -i 1 -l packages_amd64_base_stable_8  da12
gpart modify -i 1 -l packages_i386_base_stable_8   da13
gpart modify -i 1 -l ports_workdirs                da14
gpart modify -i 1 -l usr_obj_amd64_base_head       da15
gpart modify -i 1 -l usr_obj_i386_base_head        da16
gpart modify -i 1 -l usr_obj_amd64_base_stable_10  da17
gpart modify -i 1 -l usr_obj_i386_base_stable_10   da18
gpart modify -i 1 -l usr_obj_amd64_base_stable_9   da19
gpart modify -i 1 -l usr_obj_i386_base_stable_9    da20
gpart modify -i 1 -l usr_obj_amd64_base_stable_8   da21
gpart modify -i 1 -l usr_obj_i386_base_stable_8    da22

Now we need to mount the harddrives responsible for the Subversion working copies and populate them with their contents:

mkdir /mnt/base_head
mkdir /mnt/base_stable_10
mkdir /mnt/base_stable_9
mkdir /mnt/base_stable_8
mkdir /mnt/ports_head
mount -t ufs /dev/gpt/base_head      /mnt/base_head
mount -t ufs /dev/gpt/base_stable_10 /mnt/base_stable_10
mount -t ufs /dev/gpt/base_stable_9  /mnt/base_stable_9
mount -t ufs /dev/gpt/base_stable_8  /mnt/base_stable_8
cd /mnt/base_head
svnlite co svn://svn0.us-west.freebsd.org/base/head .
cd /mnt/base_stable_10
svnlite co svn://svn0.us-west.freebsd.org/base/stable/10 .
cd /mnt/base_stable_9
svnlite co svn://svn0.us-west.freebsd.org/base/stable/9 .
cd /mnt/base_stable_8
svnlite co svn://svn0.us-west.freebsd.org/base/stable/8 .
cd /mnt/ports_head
svnlite co svn://svn0.us-west.freebsd.org/ports/head .
mkdir distfiles local packages workdirs

Feel free to choose any access method and Subversion mirror.

A bit of tidying up might be in order:

cd /
umount /mnt/base_head
umount /mnt/base_stable_10
umount /mnt/base_stable_9
umount /mnt/base_stable_8
umount /mnt/ports_head
rmdir /mnt/base_head
rmdir /mnt/base_stable_10
rmdir /mnt/base_stable_9
rmdir /mnt/base_stable_8
rmdir /mnt/ports_head

Remember to add the following line to the /etc/make.conf file in every VM:

WRKDIRPREFIX=/usr/ports/workdirs

A VM running amd64 of base/head will need these lines in its /etc/fstab file:

/dev/gpt/base_head                /usr/src             ufs rw,late 2 2
/dev/gpt/usr_obj_amd64_base_head  /usr/obj             ufs rw,late 2 2
/dev/gpt/ports_head               /usr/ports           ufs rw,late 2 2
/dev/gpt/ports_distfiles          /usr/ports/distfiles ufs rw,late 2 2
/dev/gpt/packages_amd64_base_head /usr/ports/packages  ufs rw,late 2 2
/dev/gpt/ports_workdirs           /usr/ports/workdirs  ufs rw,late 2 2

A VM running i386 of base/head will need these lines in its /etc/fstab file:

/dev/gpt/base_head               /usr/src             ufs rw,late 2 2
/dev/gpt/usr_obj_i386_base_head  /usr/obj             ufs rw,late 2 2
/dev/gpt/ports_head              /usr/ports           ufs rw,late 2 2
/dev/gpt/ports_distfiles         /usr/ports/distfiles ufs rw,late 2 2
/dev/gpt/packages_i386_base_head /usr/ports/packages  ufs rw,late 2 2
/dev/gpt/ports_workdirs          /usr/ports/workdirs  ufs rw,late 2 2

A VM running amd64 of base/stable/10 will need these lines in its /etc/fstab file:

/dev/gpt/base_stable_10                /usr/src             ufs rw,late 2 2
/dev/gpt/usr_obj_amd64_base_stable_10  /usr/obj             ufs rw,late 2 2
/dev/gpt/ports_head                    /usr/ports           ufs rw,late 2 2
/dev/gpt/ports_distfiles               /usr/ports/distfiles ufs rw,late 2 2
/dev/gpt/packages_amd64_base_stable_10 /usr/ports/packages  ufs rw,late 2 2
/dev/gpt/ports_workdirs                /usr/ports/workdirs  ufs rw,late 2 2

A VM running i386 of base/stable/10 will need these lines in its /etc/fstab file:

/dev/gpt/base_stable_10               /usr/src             ufs rw,late 2 2
/dev/gpt/usr_obj_i386_base_stable_10  /usr/obj             ufs rw,late 2 2
/dev/gpt/ports_head                   /usr/ports           ufs rw,late 2 2
/dev/gpt/ports_distfiles              /usr/ports/distfiles ufs rw,late 2 2
/dev/gpt/packages_i386_base_stable_10 /usr/ports/packages  ufs rw,late 2 2
/dev/gpt/ports_workdirs               /usr/ports/workdirs  ufs rw,late 2 2

A VM running amd64 of base/stable/9 will need these lines in its /etc/fstab file:

/dev/gpt/base_stable_9                /usr/src             ufs rw,late 2 2
/dev/gpt/usr_obj_amd64_base_stable_9  /usr/obj             ufs rw,late 2 2
/dev/gpt/ports_head                   /usr/ports           ufs rw,late 2 2
/dev/gpt/ports_distfiles              /usr/ports/distfiles ufs rw,late 2 2
/dev/gpt/packages_amd64_base_stable_9 /usr/ports/packages  ufs rw,late 2 2
/dev/gpt/ports_workdirs               /usr/ports/workdirs  ufs rw,late 2 2

A VM running i386 of base/stable/9 will need these lines in its /etc/fstab file:

/dev/gpt/base_stable_9               /usr/src             ufs rw,late 2 2
/dev/gpt/usr_obj_i386_base_stable_9  /usr/obj             ufs rw,late 2 2
/dev/gpt/ports_head                  /usr/ports           ufs rw,late 2 2
/dev/gpt/ports_distfiles             /usr/ports/distfiles ufs rw,late 2 2
/dev/gpt/packages_i386_base_stable_9 /usr/ports/packages  ufs rw,late 2 2
/dev/gpt/ports_workdirs              /usr/ports/workdirs  ufs rw,late 2 2

A VM running amd64 of base/stable/8 will need these lines in its /etc/fstab file:

/dev/gpt/base_stable_8                /usr/src             ufs rw,late 2 2
/dev/gpt/usr_obj_amd64_base_stable_8  /usr/obj             ufs rw,late 2 2
/dev/gpt/ports_head                   /usr/ports           ufs rw,late 2 2
/dev/gpt/ports_distfiles              /usr/ports/distfiles ufs rw,late 2 2
/dev/gpt/packages_amd64_base_stable_8 /usr/ports/packages  ufs rw,late 2 2
/dev/gpt/ports_workdirs               /usr/ports/workdirs  ufs rw,late 2 2

A VM running i386 of base/stable/8 will need these lines in its /etc/fstab file:

/dev/gpt/base_stable_8               /usr/src             ufs rw,late 2 2
/dev/gpt/usr_obj_i386_base_stable_8  /usr/obj             ufs rw,late 2 2
/dev/gpt/ports_head                  /usr/ports           ufs rw,late 2 2
/dev/gpt/ports_distfiles             /usr/ports/distfiles ufs rw,late 2 2
/dev/gpt/packages_i386_base_stable_8 /usr/ports/packages  ufs rw,late 2 2
/dev/gpt/ports_workdirs              /usr/ports/workdirs  ufs rw,late 2 2