chyves

chyves is a bhyve front-end manager. chyves manages type-2 virtualized guests by utilizing hardware virtualization on a base FreeBSD 10.3+ installation. On a base install, only FreeBSD guests can run. However, with the installation of sysutils/grub2-bhyve and sysutils/bhyve-firmware from ports or pkg, most other OSes can run as a guest, including Windows. See DEPENDENCIES section in the man page for more information.

chyves is targeted for beginners as well as power users. Beginners should find chyves relatively easy to use with lots of documentation and demonstrations. While power users should find utility with features such as true ZFS clones, PCI passthrough, rapid execution against many guests, disk images, and snapshot reverted states on boot/reboot to name a few of the advanced features.

The name ‘chyves’ is the pluralized, big endian alphabetic increment of bhyve. ‘chyves’ is pronounced like ‘chives’, part of the Allium genus. The onion is also in the Allium genus.

announcement

Announcement

Released on August 21, 2016.

See below for some of the changes.

demo
What has changed?

Below is a partial list of changes. The man page was updated as changes were made but not as much effort was put into tracking the changes, as the Git log easily does that.

chyves is a direct code descendent of iohyve and a complete code rewrite was started in April 2016. This code was forked from iohyve at version 0.7.5 “Tennessee Cherry Moonshine Edition” release at commit 2ff5b50. Nearly every line of code has changed and that reflects in the user experience. Some of the changes include: the command syntax, user properties, property storage methods, and features just to name a few changes. See the CHANGELOG or commit log for details.

General changes

  • Rewrite of every function, either in part or completely. There are also 108 new functions, primarily for backend handling (user input, repetitive code, etc.).
  • Better error handling.
  • Kernel modules are loaded by default.
  • Library files with some files only loading when needed. Index in header of sbin/chyves.
  • Logs are now kept. There is a master log which is stored on the primary pool in a monthly format (YYYYMM.log). There is also guest log which are specific to that guest (work-in-progress, please report bugs). See LOGS section in man page.
  • The number of parameters for each command is checked and verified for user input. When conditions are not met or user input is out of specification, a warning is displayed and chyves exits before causing damage or spewing errors messages.
  • The man page has been expanded and there is a pretty online HTML version. Thank you to the ronn project for the new man page parsing platform.
  • Multiple pool configurations are completely supported.
    • Pools have roles including primary, secondary, and offline. The offline role is still being developed and refined, soon to be released.
  • Version all the things! Guests and dataset have versions associated with them. This is to allow upgrading from the pre-release versions all the way to version 999.99.999 possible without a hick-up, hopefully.
    • Datasets/pools get a version to ensure the dataset has the latest global and default properties, as well as structure.
    • Each guest has a version associated with it to ensure the properties are the latest. This also makes migrating a guest from another system and version possible.
    • During the upgrade process for guests or dataset, actions are performed on each to ensure compatibility. This makes the UPGRADE.md document a matter of reference.
  • Self upgrades to the latest version on GitHub by running chyves upgrade.
  • Properties are no longer stored as ZFS user properties, this was done with some reluctance. However the performance is much better using key values stored in hidden files and as a side benefit zfs history no longer fills up with noise. There is still a heavy dependency on ZFS due to the use of ZFS volumes as the default storage mechanism for guests’ disks and also due to the way the internal mechanisms are written.
    • Many guest properties have been added, some changed, and some have been deprecated such as name, size, and persist.
      • See:
        • chyves <guest> set <property>=<value>
        • chyves <guest> get <property>
        • chyves <guest> get all
        • chyves list properties
        • chyves list <property>
        • or the Guest Properties section in man page.
    • Default guest properties are user changeable.
    • There are now global properties that influence how chyves operates.
  • Networking changes.
    • bridge0 is no longer hard coded.
    • Complex network designs are supported.
      • Multiple tap interfaces and bridges can be assigned to a guest. Tap interfaces are assigned to guests and bridges are assigned tap interfaces and external interfaces. This information is stored as a global property.
      • Each bridge has a physical or VLAN assigned to it or can be set as a private bridge with no external interface (sensitive database traffic).
    • Two network modes: ‘auto’ and ‘system’. The ‘auto’ mode handles all configurations of the devices above the physical or VLAN interface. The ‘system’ mode relies on the user to correctly configure everything below the tap interface.
    • VALE support.
    • Emulated Intel E1000 support on 12-CURRENT hosts.
  • CPU checks are ran preflight to ensure that the necessary hardware virtualization CPU features are in place such as Second Level Address Translation (SLAT), also known as nested paging.
    • Intel (Xeon 5500) CPUs are restricted to only running one core FreeBSD guests due to the lack of a feature Intel later implemented called unrestricted guest.
    • Before starting guests with PCI-passthrough devices, an IO-MMU check is ran to see if the CPU supports the feature and that the feature is enabled.
    • See DEPENDENCIES section in man page for details.
  • make directives
    • make clean - Removes chyves.8.gz file for developers using git
    • make docs - Runs the ronn command to build the roff and html document files. Requires Ruby gem ronn.
    • make install - Installs chyves to ‘/usr/local’.
    • make installrc - Enables ‘chyves_enable=YES’ in ‘/etc/rc.conf’ using ‘sysrc’.
    • make deinstall - Remove chyves from ‘/usr/local’.
    • make rcremove - Removes ‘chyves_enable=YES’ from ‘/etc/rc.conf’ using ‘sysrc’.
    • make what - Prints available help directives.

Guest changes

  • UEFI guests have been heavily tested and are treated as a first class citizen.
    • Have more functionality compared to the other loader types.
      • UEFI GOP support.
    • Part of this is due to the complete rewrite of the start command.
  • Raw file backed disks are supported. All files in the /chyves/<pool>/guests/<guest>/img folder or child folders are attached to the guest.
  • Clone functionality has been expanded.
    • True ZFS clones are now supported and only the disks volumes are cloned. Independent clones are also possible and copied data using zfs send | zfs recv.
    • Clones can optionally have new unique properties assigned to them while cloning where the (complex) network design is duplicated, a new UUID is assigned, and a new serial console interface number is also assigned. Or clones can keep the exact properties as an offline backup. Clones with the exact properties can be deleted with chyves <guest> delete keepnet to keep the network associations.
  • There is now a feature called “Multi-guest’ which allows for rapid execution of a command over several guests machines in one command.
    • Command compatible with multi-guest are indicated in the SYNOPSIS section in the man page with “MG”.
      • For example chyves gst1,gst2,gst3 set cpu=8 will set the cpu property to “8” for guests “gst1”, “gst2”, and “gst3”.
    • Some commands support the keyword “all” to specify all guests on system.
      • For example chyves all stop or chyves all reclaim.
    • This functionality will be expanded in the future to include the use of group memberships.
  • Deprecated following user properties
    • persist
    • con -> serial
    • name
    • size

All of this does have a cost and the cost is slowness when executing commands.

Command syntax quick reference and comparison:

The man page (html version) has been vastly expanded and is a recommended read. Below is a table containing the old iohyve command and syntax compared to the new chyves commands and syntax (at the time of release.):

iohyve command chyves command Notes
iohyve chyves Obvious.
iohyve setup <pool=poolname> chyves dataset <pool> install  
- chyves dataset <pool> {offline|online|promote|uninstall} Coming soon.
- chyves dataset <pool> upgrade Upgrades a dataset to the current running pool_version version.
- chyves dev Allows for a function to be called direct from command line when developing for chyves. Requires global property dev_mode to be set with something other than ‘off’.
iohyve setup [kmod=0|1] - Deprecated. Kernel modules are loaded by default, loading can be turned off but checking for modules can not be turned off.
iohyve export <name> - Deprecated. Functionality will be written in chyves-utils side project.
iohyve fwlist chyves firmware list Also chyves list firmware works and references same function.
iohyve fetchfw <URL> chyves firmware import {http-URL|ftp-URL|local-path} Now prompts for a hash and heckles you if you do not enter one.
iohyve renamefw <firmware> <newname> chyves firmware rename <firmware> <new-firmware-name>  
iohyve rmfw <firmware> chyves firmware delete <firmware>  
iohyve clone [-r] <name> <clonename> chyves <guest> clone <clonenames>|MG [-ce|-cu|-ie|-iu] [<pool>] Expanded. See man page for details.
iohyve console <name> chyves <guest> console  
iohyve conlist - chyves info -v will list consoles and indicate an “(A)” next to active consoles.
iohyve conreset chyves <guest>|MG|all console reset  
- chyves <guest>|MG|all console tmux  
- chyves <guest>|MG|all console vnc  
iohyve create <name> <size> [pool] chyves <guest>|MG create [<size>] [<pool>] Size parameter is now optional and the default guest properties are referenced when not supplied.
iohyve delete [-f] <name> chyves <guest>|MG|all delete [force|keepnet] ‘-f’ replaced by keyword ‘force’.
iohyve add <name> <size> [pool] chyves <guest> disk add [<size>] Size is now optional and the default guest properties are referenced when not supplied. [pool] is no longer an option, guest resources must be contained under one dataset.
- chyves <guest> disk disk{n} {description|notes} <annotation> Adds an annotation to a disk which is displayed in chyves info -dn or chyves info -a.
iohyve remove [-f] <name> <diskN> chyves <guest> disk delete disk{n} Because the former syntax was confusing.
iohyve disks <name> chyves <guest> disk list  
iohyve resize <name> <diskN> <size> chyves <guest> disk resize disk{n} <new-size> Because the former syntax was confusing.
iohyve get <name> <prop> chyves <guest> get {<property>|all}  
iohyve getall <name>   Included under chyves <guest> get all
iohyve destroy <name> chyves <guest>|MG|all reclaim This reclaims the VMM resources allocated to a guest that was likely just running. This change is to avoid confusion with the ZFS keyword destroy and is more accurate to the action taken.
iohyve rename <name> <newname> chyves <guest> rename <new-guest-name>  
iohyve set <name> <property=value> … chyves <guest>|MG|global|defaults|all set [<prop1>=<value>]…  
iohyve snap <name>@<snap> chyves <guest> snapshot [<@snapshotname>]  
- chyves <guest> snapshot list  
- chyves <guest> snapshot delete <@snapshotname>  
iohyve roll <name>@<snap> chyves <guest> snapshot rollback [<@snapshotname>]  
iohyve start <name> [-s | -a] chyves <guest>|MG|all start [<iso>] The persist guest property has been deprecated so the iohyve -s and -a flags are irrelevant.
ISO images can be specified to start with a guest.
Guest now behaves like a VM would be expected to. From within a guest a ‘reboot’ will reboot and a ‘shutdown’ or ‘poweroff’ will shutdown. VMM resources are reclaimed on shutdowns. Hung bhyveload and grub-bhyve processes are killed when starting a guest.
iohyve install <name> <ISO> - Included under start command.
iohyve load <name> <path/to/bootdisk> - Included under start command.
iohyve boot <name> [runmode] [pcidevices] - Included under start command.
iohyve uefi <name> <ISO> - Included under start command.
iohyve stop <name> chyves <guest>|MG|all stop [force]  
iohyve forcekill <name> - Included under stop force command.
iohyve scram - Included under chyves all stop [force] command.
- chyves <guest>|MG|all upgrade Upgrades a guest to the current running chyves_guest_version version.
iohyve rmpci [-f] <name> <pcidev:N> chyves <guest>|MG|all unset <property>  
iohyve help chyves help  
iohyve info [-vsdl] chyves info [-zbprvstcdnakl|-h] Many more flags and defaults can be set with global property default_info_flags.
iohyve isolist chyves iso list  
iohyve cpiso <path> chyves iso import {http-URL|ftp-URL|local-path}  
iohyve renameiso <ISO> <newname> chyves iso rename <iso> <new-iso-name>  
iohyve rmiso <ISO> chyves iso delete <iso>  
iohyve list [-l] chyves list Shares backend with chyves info and can have the output changed by set the global property default_list_flags, these values are initially set to mimic historic output.
iohyve taplist chyves list bridges Now gives a network overview by displaying a hierarchal diagram of bridges, taps, physical, and vlan interfaces.
- chyves list clones Displays a hierarchal diagram of clone dependancies.
- chyves list defaults List default values used to create new guests.
iohyve disks <name> chyves list disks List all disk for guests. See chyves <guest> disk list for specific guest listing.
- chyves list global [<pool>|primary] Show global properties for all pools, a specific pool, or the primary pool.
- chyves list pools Shows all the pools, their role, and version number.
- chyves list processes [<guest>] Shows processes running for guest.
- chyves list properties Shows all properties currently set on system.
- chyves list <property> Shows the values of a particular property for all guests.
iohyve snaplist chyves list snapshots [<guest>] Only top level snapshot are displayed when <guest> parameter is omitted, this excludes snapshots for clones from being listed as those snapshots are only taken against the ZFS volumes containing the disks.
iohyve activetaps chyves list tap active  
- chyves network <guest> add {tap|tap{n}|vale{n}} Adds a tap or VALE interface to a guest. When a tap number is not given, the next tap is found and used. Tap interfaces are attached to the bridge set in default guest properties.
- chyves network <guest> add {tap|tap{n}} bridge{n} Adds a tap interface to a guest and associates that tap interface to a non-default bridge.
- chyves network <guest> remove {tap{n}|vale{n}} Removes a tap or VALE interface from a guest and dis-associates that tap with it’s bridge.
iohyve setup [net=iface] chyves network bridge{n} attach {vlan-iface{n}|physical-iface{n}} This has a similar function to the iohyve command but has the ability to specify the bridge. Network interface initially configured with chyves network bridge0 default <iface-name> when bridge0 is the desired default bridge. See NETWORK section in man page for details.
- chyves network bridge{n} {default|private} Makes a bridge the default or private. Default bridge is the bridge used to associate new tap interfaces to. A private bridge does not have any external interfaces attached.
- chyves network bridge{n} {join|unjoin} tap{n} Associates a tap interface to a bridge.
- chyves network bridge{n} migrate bridge{n} Migrates a current bridge to another bridge.
iohyve version chyves version Shows the running chyves version, dataset version, and guest version.
download
Get Started

Download:

From above links.

Install:

Unzip the repo: - unzip chyves-master.zip

Change into the directory: - cd chyves-master

Install: - make install

Update: - chyves upgrade

Remove: - make deinstall

Configure:

Install chyves datasets on '<pool>'. - chyves dataset <pool> install

Associate a physical or VLAN interface with a bridge. - chyves network bridge0 attach <iface> - ‘bridge0’ is used by default for historic reasons. However the default bridge can be changed with chyves network bridge{n} default.

Use:

See EXAMPLE section in man page.

See demonstrations above.

See USAGE.md on GitHub.

thank you

Thank you!

Thank you to all the developers, contributors, and general bad asses out there. Without their work and inspiration, this project would not exist.

Here are just a few:

  • Neel Natu

  • Peter Grehan

  • Allan Jude

  • Michael Dexter

  • Leon Dang

  • Tycho Nightingale

  • Matt Churchyard

  • Trent Thompson

  • Jim Thompson

  • Brandon Schneider

  • Peter Toth

  • Andrew Davis

  • Chris Alexander

  • Ryan Tomayko