Abstract

When starting to use FreeBSD, one might become interested in the hypervisor that FreeBSD uses, Bhyve. There are many guides on how to get started with it such as this one one from Klara Systems. However, many of these guides, including this one, teaches using vm-bhyve. While vm-bhyve is a great tool, I want to have a better understanding of vanilla bhyve for another project I am currently working on; these are some notes I have from experimenting with bhyve out of the box.

Getting Started

A natural place to start working with bhyve is going to be the bhyve man page; however, the man pages are quite dense with information, and all of it serves a purpose, but I think it is a bit much to get started. Thankfully there is an entry for Bhyve in the FreeBSD handbook. It has a very good guide on getting started with Bhyve. This blog post is another more concise ‘getting started’ guide for bhyve, however, it is worth noting that this guide only goes over creating Linux based VMs in Bhyve.

Networking

Before setting up any VMs, it would probably be beneficial to get the networking set up so that the VM can access the rest of the network and/or the Internet. The following commands should be run as root, or prefixed by sudo/doas.

# Enable vmm kernel module.
# this isn't really networking related,
# but fits in this sections quite nicely
kldload vmm
# Stuff to make networking work
ifconfig tap0 create
sysctl net.link.tap.up_on_open=1
ifconfig bridge0 create
ifconfig bridge0 addm vtnet0 addm tap0
ifconfig bridge0 up
# Add to pf.conf, only if PF is enabled
pass in on $HOSTIP from $YOURVMIP to $DHCPSERVER port 53 keep state

This should be sufficient to get networking set up in most cases, however, wireless NICs may require a different process as most wireless NICs cannot do bridge networking. In that case, setting up NAT networking is generally the way to approach giving VMs Internet access. This wiki entry from VM-bhyve should be a good place to start when setting it up.

Starting VMs

We can now finally get a VM set up.

# download the FreeBSD 13.2 release boot ISO
wget https://download.freebsd.org/releases/amd64/amd64/ISO-IMAGES/13.2/FreeBSD-13.2-RELEASE-amd64-bootonly.iso
# Load the VM with the ISO for installation
bhyveload -d FreeBSD-13.2-RELEASE-amd64-bootonly.iso -m 1G vm1
# Command to run the VM for installation
bhyve -c 1 -m 1G -A -H -P \
-s 0,hostbridge \
-s 1,lpc \
-s 2,virtio-net,tap0 \
-s 3,ahci-cd,FreeBSD-13.2-RELEASE-amd64-bootonly.iso \
-s 4,virtio-blk,guest.img \
-l com1,stdio vm1

Install FreeBSD as usual, may have to configure a static IP address and DNS server for networking to work though.

# This is required to load the VM into Bhyve
bhyveload -d guest.img -m 1G vm1

# Command to actually run the VM
bhyve -c 1 -m 1G -A -H -P \
-s 0,hostbridge \
-s 1,lpc \
-s 2,virtio-net,tap0 \
-s 3,virtio-blk,guest.img \
-l com1,stdio vm

Stopping VMs

Once a VM is started and you have a console, you can power off the VM by doing a standard shutdown in the guest. However, sometimes that is not an option for one reason or another. So, how can we shut guests down from the host? bhyvectl has us covered on it.

# To forcefully reset VM
bhyvectl --vm=${VM-Name} --force-reset
# To forcefully shutdown VM
bhyvectl --vm=${VM-Name} --force-poweroff
# To delete the VM
bhyvectl --vm=${VM-Name} --destroy

Gathing Information About Bhyve VMs

Occasionally, the system administrator will need or want to look at guests on the system for one reason or another.

All of the VMs that Bhyve knows about of the system will be listed in /dev/vmm. Using the following command will show all the VMs on the system that Bhyve is managing:

ls -al /dev/vmm

However, this doesn’t tell you much about the VMs themselves. How do we show the VMs that are currently running? or how much memory they have allocated to them? Unfortunately the Bhyve’s built in tools are not very user friendly for things like this, but there are still ways of getting such information using bhyvectl and some other built in tools. For example, the ps utility does a great job of showing running VMs:

# I personally like using the '-u' flag as it shows most everything I am looking for
# however there are many other flags that might work better for the reader, the ps manpage is a great place to look
ps -u
USER   PID %CPU %MEM     VSZ   RSS TT  STAT STARTED    TIME COMMAND
root 77792  6.0  1.7 1097588 71120  0  IC+  13:27   6:51.49 bhyve: vm1 (bhyve)
root 61273  0.0  0.1   12840  2296 v0  Is+  12:25   0:00.01 /usr/libexec/getty Pc ttyv0
root 61384  0.0  0.1   12840  2296 v1  Is+  12:25   0:00.01 /usr/libexec/getty Pc ttyv1
root 61499  0.0  0.1   12840  2296 v2  Is+  12:25   0:00.01 /usr/libexec/getty Pc ttyv2
root 61617  0.0  0.1   12840  2292 v3  Is+  12:25   0:00.01 /usr/libexec/getty Pc ttyv3
root 61718  0.0  0.1   12840  2292 v4  Is+  12:25   0:00.01 /usr/libexec/getty Pc ttyv4
root 61724  0.0  0.1   12840  2296 v5  Is+  12:25   0:00.01 /usr/libexec/getty Pc ttyv5
root 61821  0.0  0.1   12840  2296 v6  Is+  12:25   0:00.01 /usr/libexec/getty Pc ttyv6
root 61838  0.0  0.1   12840  2312 v7  Is+  12:25   0:00.01 /usr/libexec/getty Pc ttyv7
root 62367  0.0  0.1   13660  3172  0  I    12:30   0:00.04 su
root 62406  0.0  0.1   13944  3840  0  I    12:30   0:00.10 _su (csh)
root  1084  0.0  0.1   13448  2884  1  R+   14:31   0:00.00 ps -u
root 80435  0.0  0.1   13660  3164  1  I    13:45   0:00.04 su
root 80436  0.0  0.1   13944  3880  1  S    13:45   0:00.27 _su (csh)

# We can grep for bhyve to just see information about our VMs (and the grep process).
ps -u | grep bhyve
root 77792  6.0  1.7 1097588 71120  0  IC+  13:27   6:53.65 bhyve: vm1 (bhyve)
root  1181  0.0  0.1   12816  2264  1  S+   14:32   0:00.00 grep bhyve

Another utility that we can use is bhyvectl. Bhyvectl is geared more towards development rather than for administration, however, it can still give admins useful information on running VMs. The bhyvectl man page is a bit light on information, but as noted in the man page, a full list of arguments can be obtained by running bhyvectl without and arguments. Most of the valid arguments appeared to be things related to CPU registers and hacking on the virtual processor which is not particularly useful for me at this time. However, there are a few arguments that are useful:

# This command will show the CPU topology for the VM
bhyvectl --vm=vm1 --get-cpu-tologoy

# This command give a lot of output, most of which looked like register values
# piping the output through head made it much more readable for me.
cpu_topology: sockets=1, cores=1, threads=1, maxcpus=4
bhyvectl --vm=vm1 --get-all | head -n 3
ID  Length      Name
0   1024MB      sysmem
1   16384KB     bootrom

Closing Thoughts

Hopefully this article was helpful to someone trying to learn more about bhyve. Thankfully there are great tools like vm-bhyve that exist to make things like administration of bhyve VMs much easier on admins than it is with the standard tools that bhyve has. It is, however, important to know how things work on the back-end, and this information will be helpful to me in a future project.