How to run bhyve in a jail

I’ll setup a jail dedicated to run bhyve vms , for jail creation I’ll use bastillebsd

Install bastillebsd to create and manage jails.

# pkg install bastillebsd

Setup bastillebsd

Follow the getting started guide at https://bastillebsd.org/getting-started/
I’m using zfs so /usr/local/etc/bastille/bastille.conf I must edit bastille.conf (this must be done before bootstraping a release).

I used this on bastille.conf.

bastille_zfs_enable="YES"
bastille_zfs_zpool="zroot"

Create a set of rules to allow to run bhyve inside a jail edit /etc/devfs.rules, create it if does not exists.

[devfs_rules_bhyve_jail=25]
add include $devfsrules_jail
add path vmm unhide
add path vmm/* unhide
add path tap* unhide
add path nmdm* unhide

Create a new jail that will be use these rules.

sudo bastille create --vnet test-bhyve 12.2-RELEASE 192.168.1.225 em0

Modify test-bhyve jail.conf for this jail:

sudo bastille edit test-bhyve

Now add

allow.vmm;

So the jail.conf will look like:

test-bhyve {
  devfs_ruleset =25;
  enforce_statfs = 2;
  exec.clean;
  exec.consolelog = /var/log/bastille/test-bhyve_console.log;
  exec.start = '/bin/sh /etc/rc';
  exec.stop = '/bin/sh /etc/rc.shutdown';
  host.hostname = test-bhyve;
  mount.devfs;
  mount.fstab = /usr/local/bastille/jails/test-bhyve/fstab;
  path = /usr/local/bastille/jails/test-bhyve/root;
  securelevel = 2;
  allow.vmm;
  allow.raw_sockets;
  vnet;
  vnet.interface = e0b_bastille0;
  exec.prestart += "jib addm bastille0 em0";
  exec.poststop += "jib destroy bastille0";
}
~

Load the required modules:

kldload vmm
kldload nmdm

Start test-bhyve jail

sudo bastille start test-bhyve

Go inside the new jail

sudo bastille console test-bhyve

Now install vm-bhyve inside the jail follow the vm-bhyve setup at https://github.com/churchers/vm-bhyve

pkg install vm-bhyve
sysrc vm_enable="YES"
mkdir /vms
sysrc vm_dir="/vms"
vm init
cp /usr/local/share/examples/vm-bhyve/* /vms/.templates/
vm switch create public
vm switch add public vtnet0
vm iso https://download.freebsd.org/ftp/releases/ISO-IMAGES/12.2/FreeBSD-12.2-RELEASE-amd64-bootonly.iso
vm create test
vm install -f test FreeBSD-12.2-RELEASE-amd64-bootonly.iso

by default a vm-bhyve vm has 256mb, if you need more you will need to run

vm config test

And configure how much RAM do you need.

Issues

If you are going to use dhcp on the vm, you will need to configure the interface to
use SYNCDHCP.
According to this post using SYNCDHCP works, but we need to reboot the vm first.

This gives me an interface named vnet0 in my jails that i can then configure through the jail’s rc.conf. for some reason SYNCDHCP works but not DHCP in the guest rc.conf.

References

Install WordPress on SmartOS

Installing WordPress on Triton is pretty straight forward, this guide still holds but we will use base-64-lts@18.4.0 image.

$ triton create instance -w --name blog base-64-lts@18.4.0 sample-1G 
$ triton ssh blog
$ pkgin in wordpress
$ pkgin in php71-json-7.1.26.tgz  
   

This will install all packages needed for wordpress, but we will to manually do these steps from this guide http://www.machine-unix.com/how-to-install-your-wordpress-blog-on-your-joyent-smartmachine-smartos/

If this error is found : “Apache is running a threaded MPM, but your PHP Module is not compiled to be threadsafe. You need to recompile PHP.”

Replace

LoadModule mpm_event_module modules/mod_mpm_event.so

With

LoadModule mpm_prefork_module modules/mod_mpm_prefork.so

then svcadm restart apache

References:

https://docs.joyent.com/public-cloud/instances/infrastructure/images/smartos/managing-smartos/using-smf/basic-smf-commands

fs-joyent failed with exit status 95

I encountered this error in my Triton instance

Screenshot from 2019-11-27 18.25.35

 

and the community at irc.freenode.org #smartos helped me solve it.

The error was related to running out of space and my dump device was not big enough, so here are the steps to solve it:

  • Add a new disk to your Triton instance and boot it using a SmartOS usb and select rescue mode then execute:
# zpool import zones
# zfs destroy zones/dump
# zfs create -V <your ram amount>G zones/dump
# dumpadm -y -d /dev/zvol/dsk/zones/dump 
# reboot

Relevant information on how to fix this, is in this irc conversation :

https://echelog.com/logs/browse/smartos/1503957600

Setting up Triton

I’m playing with Triton lately https://github.com/joyent/triton using this awesome guide https://www.daveeddy.com/2019/02/12/smartos-coal-on-linux-kvm-with-virt-manager/

The only gotchas I found are the following:

Connect to pseudo terminal using socat

I needed to connect to a firecracker instance on /dev/pts/2 so this was the solution from https://stackoverflow.com/questions/20532195/socat-with-a-virtual-tty-link-and-fork-removes-my-pty-link , then just nc to the listening port.

$socat -d -d open:/dev/pts/2,nonblock,echo=0,raw TCP-LISTEN:11313,reuseaddr,fork 
OK  ] Started Create Volatile Files and Directories.
Starting Update UTMP about System Boot/Shutdown…
[ OK ] Started Update UTMP about System Boot/Shutdown.
[ OK ] Reached target System Initialization.
[ OK ] Reached target Timers.
[ OK ] Reached target Paths.
[ OK ] Listening on RPCbind Server Activation Socket.
[ OK ] Listening on D-Bus System Message Bus Socket.
[ OK ] Reached target Sockets.
[ OK ] Reached target Basic System.
Starting Permit User Sessions…
Starting Dump dmesg to /var/log/dmesg…
[ OK ] Started OpenSSH server daemon.
Starting OpenSSH server daemon…
Starting LSB: Bring up/down networking…
[ OK ] Started D-Bus System Message Bus.
Starting D-Bus System Message Bus…
Starting Login Service…
[ OK ] Started Permit User Sessions.
[ OK ] Started Dump dmesg to /var/log/dmesg.
[ OK ] Started Login Service.
[ OK ] Started Getty on tty1.
Starting Getty on tty1…
[ OK ] Started Serial Getty on ttyS0.
Starting Serial Getty on ttyS0…
[ OK ] Reached target Login Prompts.
[ OK ] Started LSB: Bring up/down networking.
[ OK ] Reached target Network is Online.
[ OK ] Reached target Multi-User System.
[ OK ] Reached target Graphical Interface.
Starting Update UTMP about System Runlevel Changes…
[ OK ] Started Update UTMP about System Runlevel Changes.

convert CD to iso

$ isoinfo -d -i /dev/cdrom | grep -i -E 'block size|volume size' 
 Logical block size is: 2048
 Volume size is: 1439856

$ dd if=/dev/cdrom of=~/ISOS/marvel_ultimate_alliance.iso bs=2048 count=1439856 status=progress 

2943076352 bytes (2.9 GB, 2.7 GiB) copied, 224 s, 13.1 MB/s 

1439856+0 records in

1439856+0 records out

2948825088 bytes (2.9 GB, 2.7 GiB) copied, 224.303 s, 13.1 MB/s

Alpine Linux create ZVOLS

I was starting to create zvols in Alpine, when I realized that /dev/zvol was not being created, after installing zfs-udev, udev and starting zfs-import it appeared https://github.com/zfsonlinux/zfs/issues/1234

The generic udev block device rules will detect the new zdX devices and create the /dev/zdX nodes.

Finally the udev 60-zvol.rules will create the /dev/zvol/ symlinks to the /dev/zdX nodes using the zvol_id helper.

alpine-build:/etc/init.d$ sudo apk add udev
 (1/4) Installing udev-init-scripts (32-r2)
 Executing udev-init-scripts-32-r2.post-install
 (2/4) Installing udev-init-scripts-openrc (32-r2)
 (3/4) Installing eudev-libs (3.2.7-r0)
 (4/4) Installing eudev (3.2.7-r0)
 Executing busybox-1.29.3-r10.trigger
 OK: 1414 MiB in 223 packages
 alpine-build:/etc/init.d$ ls
 acpid              hwclock            netmount           sysfsconf
 agetty             hwdrivers          networking         syslog
 binfmt             inetd              ntpd               termencoding
 bootmisc           iptables           numlock            udev
 cgroups            killprocs          osclock            udev-postmount
 chronyd            klogd              procfs             udev-settle
 consolefont        kmod-static-nodes  rdate              udev-trigger
 crond              loadkmap           root               udhcpd
 devfs              local              runsvdir           urandom
 dmesg              localmount         s6-svscan          watchdog
 dnsd               loopback           savecache          zfs-import
 docker             mdev               sshd               zfs-mount
 firstboot          modloop            staticroute        zfs-share
 fsck               modules            swap               zfs-zed
 functions.sh       mount-ro           swclock
 hostname           mtab               sysctl
 httpd              net-online         sysfs
 alpine-build:/etc/init.d$ sudo service udev start
 Caching service dependencies …                                      [ ok ]
 Starting udev …                                                     [ ok ]
 alpine-build:/etc/init.d$ ls /dev/z*
 /dev/zd0   /dev/zd16  /dev/zero  /dev/zfs
 alpine-build:/etc/init.d$ sudo service udev-postmount start
 alpine-build:/etc/init.d$ sudo service udev-trigger start
 Generating a rule to create a /dev/root symlink …                   [ ok ]
 Populating /dev with existing devices through uevents …             [ ok ]
 alpine-build:/etc/init.d$ ls /dev/z
 zd0    zd16   zero   zfs    zvol/
 alpine-build:/etc/init.d$ ls /dev/z
 zd0    zd16   zero   zfs    zvol/